Troubleshooting: Longhorn volumes take a long time to finish mounting
Phan Le | February 26, 2021
All Longhorn versions.
When starting a workload pod that uses Longhorn volumes, the Longhorn UI shows that the Longhorn volumes are attached quickly, but it takes a long time for the volumes to finish mounting and for the workload to be able to start.
This issue only happens when the Longhorn volumes have many files/directories and
securityContext.fsGroup is set in the workload pod as shown below:
spec: securityContext: runAsUser: 1000 runAsGroup: 3000 fsGroup: 2000
By default, when seeing
fsGroup field, each time a volume is mounted, Kubernetes recursively calls
chmod() on all the files and directories inside the volume.
This happens even if group ownership of the volume already matches the requested
and can be pretty expensive for larger volumes with lots of small files, which causes pod startup to take a long time.
There is no workaround for this problem in Kubernetes version v1.19.x and before.
Since version v1.20.x, Kubernetes introduces a new beta feature: the field
spec: securityContext: runAsUser: 1000 runAsGroup: 3000 fsGroup: 2000 fsGroupChangePolicy: "OnRootMismatch"
fsGroupChangePolicy is set to
OnRootMismatch, if the root of the volume already has the correct permissions,
the recursive permission and ownership change will be skipped.
It means that if users don’t change the
pod.spec.securityContext.fsGroup between pod’s startups,
K8s will only have to check the permissions and ownership of the root and the mounting process will be much faster compared to always recursively changing the volumes’ ownership and permissions.
Recent articlesTroubleshooting: Volumes Stuck in Attach/Detach Loop When Using Longhorn on OKD
© 2019-2023 Longhorn Authors | Documentation Distributed under CC-BY-4.0
© 2023 The Linux Foundation. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our Trademark Usage page.