The Longhorn Knowledge Base

Troubleshooting: Unable to access an NFS backup target

May 27, 2022

Applicable versions

All Longhorn versions.

Prerequisite

Longhorn supports only NFS versions 4.0, 4.1, and 4.2. Users should ensure that both their backup server and client support one of the supported versions.

How to determine which NFS version is …

Troubleshooting: Pod with `volumeMode: Block` is stuck in terminating

April 6, 2022

Applicable versions

All Longhorn versions.

Symptoms

User has a pod that uses a PVC with volumeMode: Block provisioned by Longhorn CSI driver. After an unexpected crash of the Longhorn volume (due to network, CPU pressure, hardware problem, …

Troubleshooting: Instance manager pods are restarted every hour

February 25, 2022

Applicable versions

v1.0.1 or newer

Background

Each Longhorn volume has one engine and one or more replicas (see more detail about Longhorn architecture at here). When a Longhorn volume is attached, Longhorn launches a process for each engine/replica …

Troubleshooting: Open-iSCSI on RHEL based systems

February 22, 2022

Applicable versions

All Longhorn versions.

Symptons

The iscsi.service systemd service may add about 2-3 minutes to the boot up time of a node if the node is restarted with longhorn volumes attached to it.

Background

Longhorn uses open-iscsi to create …

Troubleshooting: Upgrading volume engine is stuck in deadlock

January 3, 2022

Applicable versions

This happens when users upgrade Longhorn from version <= v1.1.1 to a newer version.

Symptoms

Upgrading Longhorn system includes 2 steps: first upgrade Longhorn manager to the latest version, then upgrade the Longhorn engine to …

Tip: Set Longhorn To Only Use Storage On A Specific Set Of Nodes

November 15, 2021

Applicable versions

All Longhorn versions.

Background

Let’s say you have a cluster of 5 nodes (node-1, node-2, …, node-5). You have some fast disks on node-1, node-2, and node-3 so you want Longhorn to use storage on those nodes only. …

Troubleshooting: Some old instance manager pods are still running after upgrade

November 9, 2021

Applicable versions

Longhorn >= v0.8.0.

Symptoms

Some old instance manager pods are still running after upgrade.

Details

This behavior is an expected behavior rather than a bug. In the following paragraphs, we will explain why.

Let us first take a …

Troubleshooting: Volume cannot be cleaned up after the node of the workload pod is down and recovered

November 8, 2021

Applicable versions

All Longhorn versions.

Symptoms

Volume cannot be cleaned up after the node of the workload pod is down and recovered.

Solution

The root cause is a race condition in the pod cleanup process of Kubernetes. It is fixed since …

Troubleshooting: DNS Resolution Failed

October 26, 2021

Applicable versions

All Longhorn versions.

Symptoms

The longhorn-driver-deployer or longhorn-csi-plugin or longhorn-ui Pods unable to access the longhorn manager backend http://longhorn-backend:9500/v1.

Reason

The CoreDNS of the Kubernetes cluster is …

Troubleshooting: Generate pprof runtime profiling data

October 18, 2021

Applicable versions

Longhorn >= v1.1.2.

Symptoms

Not able to investigate the longhorn-manager performance bottlenecks from the external state of the longhorn processes.

Solution

To invesigate the longhorn-manager performance bottlenecks, the …

Troubleshooting: Pod stuck in creating state when Longhorn volumes filesystem is corrupted

August 19, 2021

Applicable versions

All Longhorn versions.

Symptoms

The pod stays in container Creating with errors in the log.

  Warning  FailedMount             30s (x7 over 63s)  kubelet                  MountVolume.SetUp failed for volume …

Troubleshooting: None-standard Kubelet directory

June 18, 2021

Applicable versions

All Longhorn versions.

Symptoms

When the Kubernetes cluster is using a non-standard Kubelet directory, longhorn-csi-plugin is unable to start.

ip-172-30-0-73:/home/ec2-user # kubectl -n longhorn-system get pod
NAME …

Troubleshooting: Longhorn default settings do not persist

April 29, 2021

Applicable versions

Longhorn version < v1.3.0.

Symptoms

  • When upgrading Longhorn system via helm or Rancher App, the modified Longhorn default settings doesn’t persist.

  • When modifying the default settings via kubectl -n longhorn-system …

Troubleshooting: Recurring job does not create new jobs after detaching and attaching volume

April 21, 2021

Applicable versions

All Longhorn versions.

Symptoms

Recurring job does not create new jobs when the volume is attached after being detached for a long time.

According to Kubernetes CronJob limitations:

For every CronJob, the CronJob Controller …

Troubleshooting: Use Traefik 2.x as ingress controller

April 20, 2021

Applicable versions

All Longhorn versions with Traefik 2.x as ingress controller.

Note that, the CORS problem would not happen with Traefik 1.x as ingress controller.

Symptoms

The Longhorn GUI frontend API requests sometimes failed to reach …

Troubleshooting: Create Support Bundle with cURL

April 12, 2021

Applicable versions

All Longhorn versions.

Symptoms

Not able to create the support bundle with a web browser.

Solution

  1. Expose Longhorn backend service. Below is an example using NodePort, you can also export via load balancer if one is set up. …

Troubleshooting: Longhorn RWX shared mount ownership is shown as nobody in consumer Pod

March 31, 2021

Applicable versions

Longhorn versions = v1.1.0

Symptoms

When Pod mounts with RWX volume, the Pod share mount directory and all of the ownership of its recurring contents are shown as nobody, but in the share-manager is shown as root. …

Troubleshooting: `MountVolume.SetUp failed for volume` due to multipathd on the node

March 16, 2021

Applicable versions

All Longhorn versions.

Symptoms

The pod with the volume is not starting and encounters error messages in longhorn-csi-plugin:

time="2020-04-16T08:49:27Z" level=info msg="GRPC request: …

Troubleshooting: Longhorn-UI: Error during WebSocket handshake: Unexpected response code: 200 #2265

March 9, 2021

Applicable versions

Existing Longhorn versions < v1.1.0 upgrade to Longhorn >= v1.1.0.

Symptoms

After upgrading Longhorn to version >= v1.1.0, the following is encountered:

Ingress messages: …

Troubleshooting: Longhorn volumes take a long time to finish mounting

February 26, 2021

Applicable versions

All Longhorn versions.

Symptoms

When starting a workload pod that uses Longhorn volumes, the Longhorn UI shows that the Longhorn volumes are attached quickly, but it takes a long time for the volumes to finish mounting and for the …

Troubleshooting: `volume readonly or I/O error`

January 8, 2021

Applicable versions

All Longhorn versions.

Symptoms

When an application writes data to existing files or creates files in the mount point of a Longhorn volume, the following message is shown:

/ # cd data
/data # echo test > test
sh: can't …

Troubleshooting: `volume pvc-xxx not scheduled`

October 22, 2020

Applicable versions

All Longhorn versions.

Symptoms

When creating a Pod with Longhorn Volume as PVC, the Pod cannot start.

When checking for error message using kubectl describe <pod>, the following message is shown:

Warning  FailedAttachVolume …

© 2019-2022 Longhorn Authors | Documentation Distributed under CC-BY-4.0


© 2022 The Linux Foundation. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our Trademark Usage page.