Troubleshooting: DNS Resolution Failed

JenTing Hsiao | October 26, 2021

Applicable versions

All Longhorn versions.

Symptoms

The longhorn-driver-deployer or longhorn-csi-plugin or longhorn-ui Pods unable to access the longhorn manager backend http://longhorn-backend:9500/v1.

Reason

The CoreDNS of the Kubernetes cluster is unable to resolve the longhorn-backend service, causing the DNS resolution to fail.

Solution

  1. Check the longhorn-backend Service is available.
    kubectl get service longhorn-backend -n longhorn-system
    
  2. Make sure the longhorn-manager Pod(s) is/are up and running.
    kubectl get pod -l app=longhorn-manager -n longhorn-system
    
  3. Make sure the CoreDNS Pod(s) is/are up and running.
    kubectl get pod -n kube-system
    
  4. SSH into one of the longhorn-manager Pod, and check the nslookup test result, to make sure the DNS resolution result.
    kubectl exec -it <longhorn-manager-pod-name> -- nslookup longhorn-backend
    
  5. If you’re using k3s or rke2, add
    K3S_RESOLV_CONF=/etc/resolv.conf
    RKE2_RESOLV_CONF=/etc/resolv.conf
    

    to k3s or rke2 env files on all nodes, then restart the k3s or rke2 server/agent service.

  6. If you’re not using k3s or rke2, check the Kubernetes DNS Resolution Known Issues to see how to set kubelet’s --resolv-conf flag to point to the correct resolv.conf.
Back to Knowledge Base

Recent articles

Instruction: How To Migrate Longhorn Chart Installed In Old Rancher UI To The Chart In New Rancher UI
Troubleshooting: Unable to access an NFS backup target
Troubleshooting: Pod with `volumeMode: Block` is stuck in terminating
Troubleshooting: Instance manager pods are restarted every hour
Troubleshooting: Open-iSCSI on RHEL based systems
Troubleshooting: Upgrading volume engine is stuck in deadlock
Tip: Set Longhorn To Only Use Storage On A Specific Set Of Nodes
Troubleshooting: Some old instance manager pods are still running after upgrade
Troubleshooting: Volume cannot be cleaned up after the node of the workload pod is down and recovered
Troubleshooting: DNS Resolution Failed
Troubleshooting: Generate pprof runtime profiling data
Troubleshooting: Pod stuck in creating state when Longhorn volumes filesystem is corrupted
Troubleshooting: None-standard Kubelet directory
Troubleshooting: Longhorn default settings do not persist
Troubleshooting: Recurring job does not create new jobs after detaching and attaching volume
Troubleshooting: Use Traefik 2.x as ingress controller
Troubleshooting: Create Support Bundle with cURL
Troubleshooting: Longhorn RWX shared mount ownership is shown as nobody in consumer Pod
Troubleshooting: `MountVolume.SetUp failed for volume` due to multipathd on the node
Troubleshooting: Longhorn-UI: Error during WebSocket handshake: Unexpected response code: 200 #2265
Troubleshooting: Longhorn volumes take a long time to finish mounting
Troubleshooting: `volume readonly or I/O error`
Troubleshooting: `volume pvc-xxx not scheduled`

© 2019-2022 Longhorn Authors | Documentation Distributed under CC-BY-4.0


© 2022 The Linux Foundation. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our Trademark Usage page.