Troubleshooting: None-standard Kubelet directory

| June 18, 2021

Applicable versions

All Longhorn versions.

Symptoms

When the Kubernetes cluster is using a non-standard Kubelet directory, longhorn-csi-plugin is unable to start.

ip-172-30-0-73:/home/ec2-user # kubectl -n longhorn-system get pod
NAME                                        READY   STATUS              RESTARTS   AGE
longhorn-ui-5b864949c4-4sgws                1/1     Running             0          7m35s
longhorn-manager-tx469                      1/1     Running             0          7m35s
longhorn-driver-deployer-5444f75b8f-kgq5v   1/1     Running             0          7m35s
longhorn-csi-plugin-s4fg7                   0/2     ContainerCreating   0          6m59s
instance-manager-r-d185a1e9                 1/1     Running             0          7m10s
instance-manager-e-b5e69e2d                 1/1     Running             0          7m10s
csi-attacher-7d975797bc-qpfrv               1/1     Running             0          7m
csi-snapshotter-7dbfc7ddc6-nqqtg            1/1     Running             0          6m59s
csi-attacher-7d975797bc-td6tw               1/1     Running             0          7m
csi-resizer-868d779475-v6jvv                1/1     Running             0          7m
csi-resizer-868d779475-2bbs2                1/1     Running             0          7m
csi-provisioner-5c6845945f-46qnb            1/1     Running             0          7m
csi-resizer-868d779475-n5vjn                1/1     Running             0          7m
csi-provisioner-5c6845945f-fjnrq            1/1     Running             0          7m
csi-snapshotter-7dbfc7ddc6-mhfpl            1/1     Running             0          6m59s
csi-provisioner-5c6845945f-4lx5c            1/1     Running             0          7m
csi-attacher-7d975797bc-flldq               1/1     Running             0          7m
csi-snapshotter-7dbfc7ddc6-cms2v            1/1     Running             0          6m59s
engine-image-ei-611d1496-dlqcs              1/1     Running             0          7m10s

Reason

Caused by Longhorn cannot detect where is the root dir setup for Kubelet.

Solution

Longhorn installed via longhorn.yaml

Uncomment and edit:

        #- name: KUBELET_ROOT_DIR
        #  value: /var/lib/rancher/k3s/agent/kubelet

Longhorn installed via Rancher - App:

Click Customize Default Settings to set Kubelet Root Directory

Back to Knowledge Base

Recent articles

Troubleshooting: NoExecute taint prevents workloads from terminating
Troubleshooting: Orphan ISCSI Session Error
Failure to Attach Volumes After Upgrade to Longhorn v1.5.x
Kubernetes resource revision frequency expectations
SELinux and Longhorn
Troubleshooting: RWX Volume Fails to Be Attached Caused by `Protocol not supported`
Troubleshooting: fstrim doesn't work on old kernel
Troubleshooting: Failed RWX mount due to connection timeout
Space consumption guideline
Troubleshooting: Unexpected expansion leads to degradation or attach failure
Troubleshooting: Failure to delete orphaned Pod volume directory
Troubleshooting: Volume attachment fails due to SELinux denials in Fedora downstream distributions
Troubleshooting: Volumes Stuck in Attach/Detach Loop When Using Longhorn on OKD
Troubleshooting: Velero restores Longhorn PersistentVolumeClaim stuck in the Pending state when using the Velero CSI Plugin version before v0.4.0
Analysis: Potential Data/Filesystem Corruption
Instruction: How To Migrate Longhorn Chart Installed In Old Rancher UI To The Chart In New Rancher UI
Troubleshooting: Unable to access an NFS backup target
Troubleshooting: Pod with `volumeMode: Block` is stuck in terminating
Troubleshooting: Instance manager pods are restarted every hour
Troubleshooting: Open-iSCSI on RHEL based systems
Troubleshooting: Upgrading volume engine is stuck in deadlock
Tip: Set Longhorn To Only Use Storage On A Specific Set Of Nodes
Troubleshooting: Some old instance manager pods are still running after upgrade
Troubleshooting: Volume cannot be cleaned up after the node of the workload pod is down and recovered
Troubleshooting: DNS Resolution Failed
Troubleshooting: Generate pprof runtime profiling data
Troubleshooting: Pod stuck in creating state when Longhorn volumes filesystem is corrupted
Troubleshooting: None-standard Kubelet directory
Troubleshooting: Longhorn default settings do not persist
Troubleshooting: Recurring job does not create new jobs after detaching and attaching volume
Troubleshooting: Use Traefik 2.x as ingress controller
Troubleshooting: Create Support Bundle with cURL
Troubleshooting: Longhorn RWX shared mount ownership is shown as nobody in consumer Pod
Troubleshooting: `MountVolume.SetUp failed for volume` due to multipathd on the node
Troubleshooting: Longhorn-UI: Error during WebSocket handshake: Unexpected response code: 200 #2265
Troubleshooting: Longhorn volumes take a long time to finish mounting
Troubleshooting: `volume readonly or I/O error`
Troubleshooting: `volume pvc-xxx not scheduled`

© 2019-2024 Longhorn Authors | Documentation Distributed under CC-BY-4.0


© 2024 The Linux Foundation. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our Trademark Usage page.