Recover volume after unexpected detachment


  1. Now Longhorn can automatically reattach then remount volumes if unexpected detachment happens. e.g., Kubernetes upgrade, Docker reboot.
  2. After reattachment and remount complete, users may need to manually restart the related workload containers for the volume restoration if the following recommended setup is not applied.


Longhorn will reattach the volume if the volume engine fails unexpectedly.


  • Longhorn will detect and remount filesystem for the volume after the reattachment.
  • But the auto remount does not work for xfs filesystem.
    • Since mounting one more layers with xfs filesystem is not allowed and will trigger the error XFS (sdb): Filesystem has duplicate UUID <filesystem UUID> - can't mount.
    • Users need to manually unmount then mount the xfs filesystem on the host. The device path on host for the attached volume is /dev/longhorn/<volume name>

In order to recover unexpectedly detached volumes automatically, users can set restartPolicy to Always then add livenessProbe for the workloads using Longhorn volumes. Then those workloads will be restarted automatically after reattachment and remount.

Here is one example for the setup:

apiVersion: v1
kind: PersistentVolumeClaim
  name: longhorn-volv-pvc
    - ReadWriteOnce
  storageClassName: longhorn
      storage: 2Gi
apiVersion: v1
kind: Pod
  name: volume-test
  namespace: default
  restartPolicy: Always
  - name: volume-test
    image: nginx:stable-alpine
    imagePullPolicy: IfNotPresent
        - ls
        - /data/lost+found
      initialDelaySeconds: 5
      periodSeconds: 5
    - name: volv
      mountPath: /data
    - containerPort: 80
  - name: volv
      claimName: longhorn-volv-pvc
  • The directory used in the livenessProbe will be <volumeMount.mountPath>/lost+found
  • Don’t set a short interval for livenessProbe.periodSeconds, e.g., 1s. The liveness command is CPU consuming.

Manually restart workload containers

This solution is applied only if:

  1. The Longhorn volume is reattached and remounted automatically.
  2. The above setup is not included when the related workload is launched.


  1. Figure out on which node the related workload’s containers are running
kubectl -n <namespace of your workload> get pods <workload's pod name> -o wide
  1. Connect to the node. e.g., ssh
  2. Figure out the containers belonging to the workload
docker ps

By checking the columns COMMAND and NAMES of the output, you can find the corresponding container

  1. Restart the container
docker restart <the container ID of the workload>


Typically the volume mount propagation is not Bidirectional. It means the Longhorn remount operation won’t be propagated to the workload containers if the containers are not restarted.

© 2019-2024 Longhorn Authors | Documentation Distributed under CC-BY-4.0

© 2024 The Linux Foundation. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our Trademark Usage page.