Troubleshooting: Unable to access an NFS backup target

| May 27, 2022

Applicable versions

All Longhorn versions.

Prerequisite

Longhorn supports only NFS versions 4.0, 4.1, and 4.2. Users should ensure that both their backup server and client support one of the supported versions.

How to determine which NFS version is used

  • Server side:

Check which versions of NFS are currently enabled.

  # cat /proc/fs/nfsd/versions
  • Client side:

What versions the NFS mount is configured to support:

  # nfsstat -m

Symptom 1

The Backup page pops up with a “No such file or directory” error, for example,

, error exit status 32: vers=4.2: Failed to execute: mount [-t nfs4 -o nfsvers=4.2 -o actimeo=1 192.168.121.170:/opt/nfs-server /var/lib/longhorn-backupstore-mounts/192_168_121_170/opt/nfs-server], output mount.nfs4: mounting 192.168.121.170:/opt/nfs-server failed, reason given by server: No such file or directory
, error exit status 32: Cannot mount using NFSv4

Possible Reason 1

The backup target is not correct or exported directory does not exist on NFS server.

Solution

Correct the backup target or create the directory on NFS server.

Possible Reason 2

If the configuration file /etc/exports is as the below example:

/opt/nfs-server 192.168.121.0/24(rw,sync,no_subtree_check,crossmnt,fsid=0)

For NFSv4, the fsid=0 or fsid=root option means the exported directory is the root of all exported filesystems. The example here is /opt/nfs-server.

If users try to use the absolute path of the exported directory to list backups, that mounting error will happen.

Solution

  1. Mount with the absolute path of the exported directory (ex: /opt/nfs-server) on the client side without the option fsid=0 or fsid=root on the server side.

  2. Mount with the path “/” on the client side still with the option fsid=0 or fsid=root on the server side.

Symptom 2

The Backup page pops up with a permission denied error, for example,

error running create backup command: failed to create backup to nfs://192.168.121.170:/opt/nfs-server for volume test-for-backup: rpc error: code = Unknown desc = mkdir /var/lib/backupstore/192_168_121_170/opt/nfs-server/backup/longhorn/backupstore: permission denied" , error exit status 1

Reason

The exported directory is not accessible by non-root users due to the root_squash option being used.

Solution

  1. Use the option no_root_squash instead of root_squash in the exported directory

  2. Execute chmod o+w [exported directory path] or change the owner of the directory to nobody.

Back to Knowledge Base

Recent articles

Troubleshooting: NoExecute taint prevents workloads from terminating
Troubleshooting: Orphan ISCSI Session Error
Failure to Attach Volumes After Upgrade to Longhorn v1.5.x
Kubernetes resource revision frequency expectations
SELinux and Longhorn
Troubleshooting: RWX Volume Fails to Be Attached Caused by `Protocol not supported`
Troubleshooting: fstrim doesn't work on old kernel
Troubleshooting: Failed RWX mount due to connection timeout
Space consumption guideline
Troubleshooting: Unexpected expansion leads to degradation or attach failure
Troubleshooting: Failure to delete orphaned Pod volume directory
Troubleshooting: Volume attachment fails due to SELinux denials in Fedora downstream distributions
Troubleshooting: Volumes Stuck in Attach/Detach Loop When Using Longhorn on OKD
Troubleshooting: Velero restores Longhorn PersistentVolumeClaim stuck in the Pending state when using the Velero CSI Plugin version before v0.4.0
Analysis: Potential Data/Filesystem Corruption
Instruction: How To Migrate Longhorn Chart Installed In Old Rancher UI To The Chart In New Rancher UI
Troubleshooting: Unable to access an NFS backup target
Troubleshooting: Pod with `volumeMode: Block` is stuck in terminating
Troubleshooting: Instance manager pods are restarted every hour
Troubleshooting: Open-iSCSI on RHEL based systems
Troubleshooting: Upgrading volume engine is stuck in deadlock
Tip: Set Longhorn To Only Use Storage On A Specific Set Of Nodes
Troubleshooting: Some old instance manager pods are still running after upgrade
Troubleshooting: Volume cannot be cleaned up after the node of the workload pod is down and recovered
Troubleshooting: DNS Resolution Failed
Troubleshooting: Generate pprof runtime profiling data
Troubleshooting: Pod stuck in creating state when Longhorn volumes filesystem is corrupted
Troubleshooting: None-standard Kubelet directory
Troubleshooting: Longhorn default settings do not persist
Troubleshooting: Recurring job does not create new jobs after detaching and attaching volume
Troubleshooting: Use Traefik 2.x as ingress controller
Troubleshooting: Create Support Bundle with cURL
Troubleshooting: Longhorn RWX shared mount ownership is shown as nobody in consumer Pod
Troubleshooting: `MountVolume.SetUp failed for volume` due to multipathd on the node
Troubleshooting: Longhorn-UI: Error during WebSocket handshake: Unexpected response code: 200 #2265
Troubleshooting: Longhorn volumes take a long time to finish mounting
Troubleshooting: `volume readonly or I/O error`
Troubleshooting: `volume pvc-xxx not scheduled`

© 2019-2024 Longhorn Authors | Documentation Distributed under CC-BY-4.0


© 2024 The Linux Foundation. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our Trademark Usage page.