In this section, you’ll have a better understanding of concepts related to volume size.
In the example, we will explain how volume
actual size get changed after a bunch of IO and snapshot related operations.
The illustration presents the file organization of one replica. The volume head and snapshots are actually sparse files, which we mentioned above.
sizeis 12 Gi and the
actual sizeis almost 0.
actual sizeis 260 Mi and is not exactly 0.
actual sizeis increased by 4 Gi because of the allocated blocks in the replica for the 4 Gi data. Meanwhile,
dfcommand in the filesystem also shows the 4 Gi used space. See Figure 2 of the illustration.
Delete the 4 Gi data. Then,
df command shows that the used space of the filesystem is nearly 0, but the
actual size is unchanged.
Users can see the volume
actual sizeis not shrunk after deleting the 4 Gi data. Longhorn is a block-level storage system. Therefore, the deletion in the filesystem only marks the blocks that belong to the deleted file as unused. Currently, Longhorn does not support TRIM/UNMAP operations, so the
discardmount option or
fstrimin the filesystem layer cannot reclaim the unused blocks. In consequence, the actual size of Longhorn volumes cannot be shrunk in this case.
Then, rewrite the 4 Gi data (data#1), and the
df command in the filesystem shows 4 Gi used space again. However, the
actual size is increased by 4 Gi and becomes 8.25Gi. See Figure 3(a) of the illustration.
After deletion, filesystem may or maynot reuse the recently freed blocks from recently deleted files according to the filesystem design and please refer to Block allocation strategies of various filesystems. If the volume nominal
sizeis 12 Gi, the
actual sizein the end would range from 4 Gi to 8 Gi since the filesystem may or maynot reuse the freed blocks. On the other hand, if the volume nominal
sizeis 6 Gi, the
actual sizeat the end would range from 4 Gi to 6 Gi, because the filesystem has to reuse the freed blocks in the 2nd round of writing. See Figure 3(b) of the illustration.
Thus, allocating an appropriate nominal
sizefor a volume that holds heavy writing tasks according to the IO pattern would make disk space usage more efficient.
actual sizeremains 8.25 Gi.
Delete data#1 from the mount point.
actual sizeis still 8.25 Gi.
Write 8 Gi data (data#2) in the volume mount, then take one more snapshot (snapshot#2). See Figure 5 of the illustration.
actual sizeis 16.2 Gi, which is greater than the volume nominal
The volume head holds the latest data of the volume only, while each snapshot may store historical data as well as active data, which consumes at most size space. Therefore, the volume
actual size, which is the size sum of the volume head and all snapshots, is possibly bigger than the size specified by users.
Even if users will not take snapshots for volumes, there are operations like rebuilding, expansion, or backing up that would lead to system (hidden) snapshot creation. As a result, volume
actual sizebeing larger than size is unavoidable under some use cases.
Here we summarize the important things related to disk space usage we have in the example:
Unused blocks are not released
Longhorn does not support TRIM/UNMAP operations. Hence deleting files from filesystems will not lead to volume actual size decreasing/shrinking.
Allocated blocks but unused are not reused
Deleting then writing new files would lead to the actual size keeps increasing. Since the filesystem may not reuse the recently freed blocks from recently deleted files. Thus, allocating an appropriate nominal size for a volume that holds heavy writing tasks according to the IO pattern would make disk space usage more efficient.
By deleting snapshots, the overlapping part of the used blocks might be eliminated regardless of whether the blocks are recently released blocks by the filesystem or still contain historical data.
Reserve enough free space in disks as buffers in case of the actual size of existing volumes keep growing up.
A general estimation for the maximum space consumption of a volume is
(N + 1) x head/snapshot average actual size
Nis the total number of snapshots the volume contains (including the volume head), and the extra
1is for the temporary space that may be required by snapshot deletion.
Storage Over Provisioning Percentageto be smaller than 100% to avoid disk space exhaustion.
There is one snapshot recurring job with retention number is
N. Then the formula can be extended to:
(M + N + 1 + 1 + 1 + 1) x head/snapshot average actual size
Mis the snapshots created by users manually. Recurring jobs are not responsible for removing this kind of snapshot. They can be deleted by users only.
Nis the snapshot recurring job retain number.
1means the volume head.
1means the extra snapshot created by the recurring job. Since the recurring job always creates a new snapshot then deletes the oldest snapshot when the current snapshots created by itself exceeds the retention number. Before the deletion starts, there is one extra snapshot that can take extra disk space.
1is the system snapshot. If the rebuilding is triggered or the expansion is issued, Longhorn will create a system snapshot before starting the operations. And this system snapshot may not be able to get cleaned up immediately.
1is for the temporary space that may be required by snapshot deletion/purge.
Users don’t want snapshot at all. Neither (manually created) snapshot nor recurring job will be launched. Assume setting Automatically Cleanup System Generated Snapshot is enabled, then formula would become:
(1 + 1 + 1) x head/snapshot average actual size
1means the volume head.
1is the second system snapshot mentioned in the worst case.
1is for the temporary space that may be required by the 2 system snapshot purge/coalescing.
Do not retain too many snapshots for the volumes.
Cleaning up snapshots will help reclaim disk space. There are two ways to clean up snapshots:
Also, notice that the extra space, up to volume nominal
size, is required during snapshot cleanup and merge.
An appropriate the volume nominal
size according to the workloads.
© 2019-2023 Longhorn Authors | Documentation Distributed under CC-BY-4.0
© 2023 The Linux Foundation. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our Trademark Usage page.