Troubleshooting: DNS Resolution Failed

JenTing Hsiao | October 26, 2021

Applicable versions

All Longhorn versions.

Symptoms

The longhorn-driver-deployer or longhorn-csi-plugin or longhorn-ui Pods unable to access the longhorn manager backend http://longhorn-backend:9500/v1.

Reason

The CoreDNS of the Kubernetes cluster is unable to resolve the longhorn-backend service, causing the DNS resolution to fail.

Solution

  1. Check the longhorn-backend Service is available.
    kubectl get service longhorn-backend -n longhorn-system
    
  2. Make sure the longhorn-manager Pod(s) is/are up and running.
    kubectl get pod -l app=longhorn-manager -n longhorn-system
    
  3. Make sure the CoreDNS Pod(s) is/are up and running.
    kubectl get pod -n kube-system
    
  4. SSH into one of the longhorn-manager Pod, and check the nslookup test result, to make sure the DNS resolution result.
    kubectl exec -it <longhorn-manager-pod-name> -- nslookup longhorn-backend
    
  5. If you’re using k3s or rke2, add
    K3S_RESOLV_CONF=/etc/resolv.conf
    RKE2_RESOLV_CONF=/etc/resolv.conf
    

    to k3s or rke2 env files on all nodes, then restart the k3s or rke2 server/agent service.

  6. If you’re not using k3s or rke2, check the Kubernetes DNS Resolution Known Issues to see how to set kubelet’s --resolv-conf flag to point to the correct resolv.conf.
Back to Knowledge Base

© 2019-2021 Longhorn Authors | Documentation Distributed under CC-BY-4.0


© 2021 The Linux Foundation. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our Trademark Usage page.