You can generate a support bundle file for offline troubleshooting. See Support Bundle for detail.
Check if the volume plugin directory has been set correctly. This is automatically detected unless user explicitly set it.
By default, Kubernetes uses
/usr/libexec/kubernetes/kubelet-plugins/volume/exec/, as stated in the official document.
Some vendors choose to change the directory for various reasons. For example, GKE uses
The correct directory can be found by running
ps aux|grep kubelet on the host and check the
--volume-plugin-dir parameter. If there is none, the default
/usr/libexec/kubernetes/kubelet-plugins/volume/exec/ will be used.
There are a few components in Longhorn: Manager, Engine, Driver and UI. By default, all of those components run as pods in the
longhorn-system namespace in the Kubernetes cluster.
Most of the logs are included in the Support Bundle. You can click the Generate Support Bundle link at the bottom of the UI to download a zip file that contains Longhorn-related configuration and logs.
One exception is the
dmesg, which needs to be retrieved from each node by the user.
Make use of the Longhorn UI is a good start for the troubleshooting. For example, if Kubernetes cannot mount one volume correctly, after stop the workload, try to attach and mount that volume manually on one node and access the content to check if volume is intact.
Also, the event logs in the UI dashboard provides some information of probably issues. Check for the event logs in
You can get the logs from the Longhorn Manager and Engines to help with troubleshooting. The most useful logs are the ones from
longhorn-manager-xxx, and the logs inside Longhorn instance managers, e.g.
Since normally there are multiple Longhorn Managers running at the same time, we recommend using kubetail, which is a great tool to keep track of the logs of multiple pods. To track the manager logs in real time, you can use:
kubetail longhorn-manager -n longhorn-system
For the CSI driver, check the logs for
csi-provisioner-0, as well as containers in
The FlexVolume driver is deprecated as of Longhorn v0.8.0 and should no longer be used.
First check where the driver has been installed on the node. Check the log of
longhorn-driver-deployer-xxxx for that information.
Then check the kubelet logs. The FlexVolume driver itself doesn’t run inside the container. It would run along with the kubelet process.
If kubelet is running natively on the node, you can use the following command to get the logs:
journalctl -u kubelet
Or if kubelet is running as a container (e.g. in RKE), use the following command instead:
docker logs kubelet
For even more detailed logs of Longhorn FlexVolume, run the following command on the node or inside the container (if kubelet is running as a container, e.g. in RKE):
© 2019-2023 Longhorn Authors | Documentation Distributed under CC-BY-4.0
© 2023 The Linux Foundation. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our Trademark Usage page.