Best Practices
We recommend the following setup for deploying Longhorn in production.
Longhorn supports the following architectures:
Note: CentOS Linux has been removed from the verified OS list below, as it has been discontinued in favor of CentOS Stream [ref], a rolling-release Linux distribution. Our focus for verifying RHEL-based downstream open source distributions will be enterprise-grade, such as Rocky and Oracle Linux.
The following Linux OS distributions and versions have been verified during the v1.8.0 release testing. However, this does not imply that Longhorn exclusively supports these distributions. Essentially, Longhorn should function well on any certified Kubernetes cluster running on Linux nodes with a wide range of general-purpose operating systems, as well as verified container-optimized operating systems like SLE Micro.
No. | OS | Versions |
---|---|---|
1. | Ubuntu | 24.04 |
2. | SSUSE Linux Enterprise Server | 15 SP6 |
3. | SUSE Linux Enterprise Micro | 6.0 |
4. | Red Hat Enterprise Linux | 9.3 |
5. | Oracle Linux | 9.4 |
6. | Rocky Linux | 9.3 |
7. | Talos Linux | 1.7 |
8. | Container-Optimized OS (GKE) | 113 |
Longhorn relies heavily on kernel functionality and performs better on some kernel versions. The following activities, in particular, benefit from usage of specific kernel versions.
v5.8
or later. See Issue
#2507 for details.5.17
or later to ensure that a volume crash during a filesystem freeze cannot lock up a node.5.19
or later to ensureThe list below contains known broken kernel versions that users should avoid using:
No. | Version | Distro | Additional Context |
---|---|---|---|
1. | 6.5.6 | Vanilla kernel | Related to this bug https://longhorn.io/kb/troubleshooting-rwx-volume-fails-to-attached-caused-by-protocol-not-supported/ |
2. | 5.15.0-94 | Ubuntu | Related to this bug https://longhorn.io/kb/troubleshooting-rwx-volume-fails-to-attached-caused-by-protocol-not-supported/ |
3. | 6.5.0-21 | Ubuntu | Related to this bug https://longhorn.io/kb/troubleshooting-rwx-volume-fails-to-attached-caused-by-protocol-not-supported/ |
4. | 6.5.0-1014-aws | Ubuntu | Related to this bug https://longhorn.io/kb/troubleshooting-rwx-volume-fails-to-attached-caused-by-protocol-not-supported/ |
We recommend running your Kubernetes cluster on one of the following versions. These versions are the active supported versions prior to the Longhorn release, and have been tested with Longhorn v1.8.0.
Release | Released | End-of-life |
---|---|---|
1.30 | 17 Apr 2024 | 28 Jun 2025 |
1.28 | 15 Aug 2023 | 28 Oct 2024 |
1.27 | 11 Apr 2023 | 28 Jun 2024 |
Referenced to https://endoflife.date/kubernetes.
We recommend the following setup for nodes and disks.
It’s recommended to dedicate a disk for Longhorn storage for production, instead of using the root disk.
If you need to use the root disk, use the default minimal available storage percentage
setup which is 25%, and set overprovisioning percentage
to 100% to minimize the chance of DiskPressure.
If you’re using a dedicated disk for Longhorn, you can lower the setting minimal available storage percentage
to 10%.
For the Over-provisioning percentage, it depends on how much space your volume uses on average. For example, if your workload only uses half of the available volume size, you can set the Over-provisioning percentage to 200
, which means Longhorn will consider the disk to have twice the schedulable size as its full size minus the reserved space.
Since Longhorn doesn’t currently support sharding between the different disks, we recommend using LVM to aggregate all the disks for Longhorn into a single partition, so it can be easily extended in the future.
Any extra disks must be written in the /etc/fstab
file to allow automatic mounting after the machine reboots.
Don’t use a symbolic link for the extra disks. Use mount --bind
instead of ln -s
and make sure it’s in the fstab
file. For details, see the section about multiple disk support.
To use a directory other than the default /var/lib/longhorn
for storage, the Default Data Path
setting can be changed before installing the system. For details on changing pre-installation settings, refer to this section.
The Default node/disk configuration feature can be used to customize the default disk after installation. Customizing the default configurations for disks and nodes is useful for scaling the cluster because it eliminates the need to configure Longhorn manually for each new node if the node contains more than one disk, or if the disk configuration is different for new nodes. Remember to enable Create default disk only on labeled node
if applicable.
Before configuring workloads, ensure that you have set up the following basic requirements for optimal volume performance.
longhorn-critical
.The following sections outline other recommendations for production environments.
Storage network: Use a dedicated storage network to improve IO performance and stability.
Longhorn disk: Use a dedicated disk for Longhorn storage instead of using the root disk.
Replica count: Set the default replica count to “2” to achieve data availability with better disk space usage or less impact to system performance. This practice is especially beneficial to data-intensive applications.
Storage tag: Use storage tags to define storage tiering for data-intensive applications. For example, only high-performance disks can be used for storing performance-sensitive data.
Data locality: Use best-effort
as the default data locality of Longhorn StorageClasses.
For applications that support data replication (for example, a distributed database), you can use the strict-local
option to ensure that only one replica is created for each volume. This practice prevents the extra disk space usage and IO performance overhead associated with volume replication.
For data-intensive applications, you can use pod scheduling functions such as node selector or taint toleration. These functions allow you to schedule the workload to a specific storage-tagged node together with one replica.
Recurring snapshots: Periodically clean up system-generated snapshots and retain only the number of snapshots that makes sense for your implementation.
For applications with replication capability, periodically delete all types of snapshots.
Recurring filesystem trim: Periodically trim the filesystem inside volumes to reclaim disk space.
Snapshot space management: Configure global and volume-specific settings to prevent unexpected disk space exhaustion.
Recurring backups: Create recurring backup jobs for mission-critical application volumes.
System backup: Create periodic system backups.
If you’re using ext4
as the filesystem of the volume, we recommend adding a liveness check to workloads to help automatically recover from a network-caused interruption, a node reboot, or a Docker restart. See this section for details.
Using Longhorn’s built-in backup feature is highly recommended. You can save backups to an object store such as S3 or to an NFS server. Saving to an object store is preferable because it generally offers better reliability. Another advantage is that you do not need to mount and unmount the target, which can complicate failover and upgrades.
For each volume, schedule at least one recurring backup. If you must run Longhorn in production without a backupstore, then schedule at least one recurring snapshot for each volume.
Longhorn system will create snapshots automatically when rebuilding a replica. Recurring snapshots or backups can also automatically clean up the system-generated snapshot.
We recommend setting the CPU request for Longhorn instance manager pods.
The Guaranteed Instance Manager CPU
setting allows you to reserve a percentage of the total allocatable CPU resources on each node for each instance manager pod when the V1 Data Engine is enabled. The default value is 12.
You can also set a specific milli CPU value for instance manager pods on a particular node by updating the node’s Instance Manager CPU Request
field.
Note: This field will overwrite the above setting for the specified node.
Refer to Guaranteed Instance Manager CPU for more details.
The Guaranteed Instance Manager CPU for V2 Data Engine
setting allows you to reserve a specific number of millicpus on each node for each instance manager pod when the V2 Data Engine is enabled. By default, the Storage Performance Development Kit (SPDK) target daemon within each instance manager pod uses 1 CPU core. Configuring a minimum CPU usage value is essential for maintaining engine and replica stability, especially during periods of high node workload. The default value is 1250.
We don’t recommend modifying the default StorageClass named longhorn
, since the change of parameters might cause issues during an upgrade later. If you want to change the parameters set in the StorageClass, you can create a new StorageClass by referring to the StorageClass examples.
Recommend:
false
This setting should be set to false
in production environment to ensure the best availability of the volume. Otherwise, one node down event may bring down more than one replicas of a volume.
Recommend:
false
This setting should be set to false
in production environment to ensure every volume have the best availability when created. Because with the setting set to true
, the volume creation won’t error out even there is only enough room to schedule one replica. So there is a risk that the cluster is running out of the spaces but the user won’t be made aware immediately.
Recommend:
least-effort
For production environments, we recommend setting Replica Auto-Balance to least-effort
. This setting ensures that at least one replica is placed on a different node in each zone, providing extra high availability (HA).
In certain edge cases, you might consider using the best-effort
, which continuously attempts to evenly distribute replicas across nodes and zones. However, this setting can lead to frequent rebuilds if the cluster is unstable.
For most users, having multiple replicas without Replica Auto-Balance setting is sufficient to achieve basic HA, especially if you prefer to avoid excessive rebuilds and resource usage.
© 2019-2024 Longhorn Authors | Documentation Distributed under CC-BY-4.0
© 2024 The Linux Foundation. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our Trademark Usage page.