Troubleshooting: `volume pvc-xxx not scheduled`

Sheng Yang | October 22, 2020

Applicable versions

All Longhorn versions.

Symptoms

When creating a Pod with Longhorn Volume as PVC, the Pod cannot start.

When checking for error message using kubectl describe <pod>, the following message is shown:

Warning  FailedAttachVolume  4m29s (x3 over 5m33s)  attachdetach-controller     AttachVolume.Attach failed for volume "pvc-xxx" : rpc error: code = Internal desc = Bad response statusCode [500]. Status [500 Internal Server Error]. Body: [message=unable to attach volume pvc-xxx to node-xxx: volume pvc-xxx not scheduled, code=Server Error, detail=] from [http://longhorn-backend:9500/v1/volumes/pvc-xxx?action=attach]

Noticed the message returned by Longhorn in the error above:

unable to attach volume pvc-xxx to node-xxx: volume pvc-xxx not scheduled

Details

This is caused by Longhorn cannot find enough spaces on different nodes to store the data for the volume, which result in the volume scheduling failure.

Most common reason

For Longhorn v1.0.x, the default Longhorn installation has following settings:

  1. Node Level Soft Anti-affinity: false.
  2. The default StorageClass longhorn's Replica count is set to 3.

That means Longhorn will always try to allocate enough space on three different nodes for three replicas.

If this requirement cannot be satisfied, e.g. due to there are less than 3 nodes in the cluster, the volume scheduling will fail.

Solution

If this is the case, you can:

  1. either set Node Level Soft Anti-affinity to true.
  2. or, create a new StorageClass with replica count set to 1 or 2.
  3. or, adding more nodes to your cluster.

Other reasons

See scheduling section in the Longhorn doc for a detail description of the scheduling policy.

Starting Longhorn v1.1.0, we will introduce a new setting Allow Volume Creation With Degraded Availability(true by default) to help with the use case on smaller cluster.

See https://github.com/longhorn/longhorn/issues/1701 for details.

Back to Knowledge Base

Recent articles

Instruction: How To Migrate Longhorn Chart Installed In Old Rancher UI To The Chart In New Rancher UI
Troubleshooting: Unable to access an NFS backup target
Troubleshooting: Pod with `volumeMode: Block` is stuck in terminating
Troubleshooting: Instance manager pods are restarted every hour
Troubleshooting: Open-iSCSI on RHEL based systems
Troubleshooting: Upgrading volume engine is stuck in deadlock
Tip: Set Longhorn To Only Use Storage On A Specific Set Of Nodes
Troubleshooting: Some old instance manager pods are still running after upgrade
Troubleshooting: Volume cannot be cleaned up after the node of the workload pod is down and recovered
Troubleshooting: DNS Resolution Failed
Troubleshooting: Generate pprof runtime profiling data
Troubleshooting: Pod stuck in creating state when Longhorn volumes filesystem is corrupted
Troubleshooting: None-standard Kubelet directory
Troubleshooting: Longhorn default settings do not persist
Troubleshooting: Recurring job does not create new jobs after detaching and attaching volume
Troubleshooting: Use Traefik 2.x as ingress controller
Troubleshooting: Create Support Bundle with cURL
Troubleshooting: Longhorn RWX shared mount ownership is shown as nobody in consumer Pod
Troubleshooting: `MountVolume.SetUp failed for volume` due to multipathd on the node
Troubleshooting: Longhorn-UI: Error during WebSocket handshake: Unexpected response code: 200 #2265
Troubleshooting: Longhorn volumes take a long time to finish mounting
Troubleshooting: `volume readonly or I/O error`
Troubleshooting: `volume pvc-xxx not scheduled`

© 2019-2022 Longhorn Authors | Documentation Distributed under CC-BY-4.0


© 2022 The Linux Foundation. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our Trademark Usage page.