Troubleshooting: Volume cannot be cleaned up after the node of the workload pod is down and recovered

| November 8, 2021

Applicable versions

All Longhorn versions.

Symptoms

Volume cannot be cleaned up after the node of the workload pod is down and recovered.

Solution

The root cause is a race condition in the pod cleanup process of Kubernetes. It is fixed since Kubernetes 1.22.0+ according to this commit.

Longhorn issue #3080

Back to Knowledge Base

Recent articles

Graceful Longhorn Node Eviction Before Cluster API (CAPI) Node Replacement
Manual Recovery of Nodes with Insufficient Space
Troubleshooting: iSCSId has no route to instance manager pod
Troubleshooting: Migratable RWX volume stuck in detaching/attaching loop
Restoring Data from an Orphaned Replica Directory
Troubleshooting: Handling Persistent Replica Failures via Node or Disk Isolation
Troubleshooting: Backing Image Manager CR naming collisions
Troubleshooting: Concurrent I/O Stuck On A RWX Volume
Troubleshooting: Backing Image Download Stuck After Node Disconnection
Backup store lock conflict error message
Troubleshooting: Mount Failure with XFS Filesystem
Troubleshooting: Encrypted RWX Volume Fails to Perform Live Expansion
Troubleshooting: Storage Network CSI Plugin Restart Triggers Unintended Restart of RWX Migratable Volume Workload
Troubleshooting: Migratable RWX volume migration stuck
Troubleshooting: Failed Replicas with Backing Images Remain On Evicted Nodes
Troubleshooting: Orphan Engine Or Replica Instance
Troubleshooting: Recurring Job Pod stuck in pending state
Troubleshooting: Resolving Backing Image Unavailability Issue
Troubleshooting: Longhorn Manager Crashes Due To Backing Image Eviction
Troubleshooting: Backing Image Creation Is Stuck Or Has Failed
Troubleshooting: Two active engines during volume live migration
Troubleshooting: Longhorn Manager Stuck in CrashLoopBackOff State Due to Inaccessible Webhook
Troubleshooting: Instance Manager Pods Are Restarted
Troubleshooting: NoExecute taint prevents workloads from terminating
Troubleshooting: Orphan ISCSI Session Error
Failure to Attach Volumes After Upgrade to Longhorn v1.5.x
Kubernetes resource revision frequency expectations
SELinux and Longhorn
Troubleshooting: RWX Volume Fails to Be Attached Caused by `Protocol not supported`
Troubleshooting: fstrim doesn't work on old kernel
Troubleshooting: Failed RWX mount due to connection timeout
Space consumption guideline
Troubleshooting: Unexpected expansion leads to degradation or attach failure
Troubleshooting: Failure to delete orphaned Pod volume directory
Troubleshooting: Volume attachment fails due to SELinux denials in Fedora downstream distributions
Troubleshooting: Volumes Stuck in Attach/Detach Loop When Using Longhorn on OKD
Troubleshooting: Velero restores Longhorn PersistentVolumeClaim stuck in the Pending state when using the Velero CSI Plugin version before v0.4.0
Analysis: Potential Data/Filesystem Corruption
Instruction: How To Migrate Longhorn Chart Installed In Old Rancher UI To The Chart In New Rancher UI
Troubleshooting: Unable to access an NFS backup target
Troubleshooting: Pod with `volumeMode: Block` is stuck in terminating
Troubleshooting: Instance manager pods are restarted every hour
Troubleshooting: Open-iSCSI on RHEL based systems
Troubleshooting: Upgrading volume engine is stuck in deadlock
Tip: Set Longhorn To Only Use Storage On A Specific Set Of Nodes
Troubleshooting: Some old instance manager pods are still running after upgrade
Troubleshooting: Volume cannot be cleaned up after the node of the workload pod is down and recovered
Troubleshooting: DNS Resolution Failed
Troubleshooting: Generate pprof runtime profiling data
Troubleshooting: Pod stuck in creating state when Longhorn volumes filesystem is corrupted
Troubleshooting: None-standard Kubelet directory
Troubleshooting: Longhorn default settings do not persist
Troubleshooting: Recurring job does not create new jobs after detaching and attaching volume
Troubleshooting: Use Traefik 2.x as ingress controller
Troubleshooting: Create Support Bundle with cURL
Troubleshooting: Longhorn RWX shared mount ownership is shown as nobody in consumer Pod
Troubleshooting: `MountVolume.SetUp failed for volume` due to multipathd on the node
Troubleshooting: Longhorn-UI: Error during WebSocket handshake: Unexpected response code: 200 #2265
Troubleshooting: Longhorn volumes take a long time to finish mounting
Troubleshooting: `volume readonly or I/O error`
Troubleshooting: `volume pvc-xxx not scheduled`

© 2019-2026 Longhorn a Series of LF Projects, LLC. Documentation Distributed under CC-BY-4.0.


For website terms of use, trademark policy and other project policies please see lfprojects.org/policies.