Volume Size
In this section, you’ll have a better understanding of concepts related to volume size.
Size
:ext4
).Actual Size
In the example, we will explain how volume size
and actual size
get changed after a bunch of IO and snapshot related operations.
The illustration presents the file organization of one replica. The volume head and snapshots are actually sparse files, which we mentioned above.
size
is 12 Gi and the actual size
is almost 0.actual size
is 260 Mi and is not exactly 0.actual size
is increased by 4 Gi because of the allocated blocks in the replica for the 4 Gi data. Meanwhile, df
command in the filesystem also shows the 4 Gi used space. See Figure 2 of the illustration.Delete the 4 Gi data. Then, df
command shows that the used space of the filesystem is nearly 0, but the actual size
is unchanged.
Users can see by default the volume
actual size
is not shrunk after deleting the 4 Gi data. Longhorn is a block-level storage system. Therefore, the deletion in the filesystem only marks the blocks that belong to the deleted file as unused. Currently, Longhorn will not apply TRIM/UNMAP operations automatically/periodically. if you want to do filesystem trim, please check this doc for details.
Then, rewrite the 4 Gi data (data#1), and the df
command in the filesystem shows 4 Gi used space again. However, the actual size
is increased by 4 Gi and becomes 8.25Gi. See Figure 3(a) of the illustration.
After deletion, filesystem may or maynot reuse the recently freed blocks from recently deleted files according to the filesystem design and please refer to Block allocation strategies of various filesystems. If the volume nominal
size
is 12 Gi, theactual size
in the end would range from 4 Gi to 8 Gi since the filesystem may or maynot reuse the freed blocks. On the other hand, if the volume nominalsize
is 6 Gi, theactual size
at the end would range from 4 Gi to 6 Gi, because the filesystem has to reuse the freed blocks in the 2nd round of writing. See Figure 3(b) of the illustration.Thus, allocating an appropriate nominal
size
for a volume that holds heavy writing tasks according to the IO pattern would make disk space usage more efficient.
actual size
remains 8.25 Gi.Delete data#1 from the mount point.
actual size
is still 8.25 Gi.Write 8 Gi data (data#2) in the volume mount, then take one more snapshot (snapshot#2). See Figure 5 of the illustration.
actual size
is 16.2 Gi, which is greater than the volume nominal size
.The volume head holds the latest data of the volume only, while each snapshot may store historical data as well as active data, which consumes at most size space. Therefore, the volume
actual size
, which is the size sum of the volume head and all snapshots, is possibly bigger than the size specified by users.Even if users will not take snapshots for volumes, there are operations like rebuilding, expansion, or backing up that would lead to system (hidden) snapshot creation. As a result, volume
actual size
being larger than size is unavoidable under some use cases.
Here we summarize the important things related to disk space usage we have in the example:
Unused blocks are not released
Longhorn will not issue TRIM/UNMAP operations automatically. Hence deleting files from filesystems will not lead to volume actual size decreasing/shrinking. You may need to check the doc and handle it by yourself if needed.
Allocated blocks but unused are not reused
Deleting then writing new files would lead to the actual size keeps increasing. Since the filesystem may not reuse the recently freed blocks from recently deleted files. Thus, allocating an appropriate nominal size for a volume that holds heavy writing tasks according to the IO pattern would make disk space usage more efficient.
By deleting snapshots, the overlapping part of the used blocks might be eliminated regardless of whether the blocks are recently released blocks by the filesystem or still contain historical data.
Reserve enough free space in disks as buffers in case of the actual size of existing volumes keep growing up.
A general estimation for the maximum space consumption of a volume is
(N + 1) x head/snapshot average actual size
N
is the total number of snapshots the volume contains (including the volume head), and the extra 1
is for the temporary space that may be required by snapshot deletion.Storage Over Provisioning Percentage
to be smaller than 100% to avoid disk space exhaustion.There is one snapshot recurring job with retention number is N
. Then the formula can be extended to:
(M + N + 1 + 1 + 1 + 1) x head/snapshot average actual size
M
is the snapshots created by users manually. Recurring jobs are not responsible for removing this kind of snapshot. They can be deleted by users only.N
is the snapshot recurring job retain number.1
means the volume head.1
means the extra snapshot created by the recurring job. Since the recurring job always creates a new snapshot then deletes the oldest snapshot when the current snapshots created by itself exceeds the retention number. Before the deletion starts, there is one extra snapshot that can take extra disk space.1
is the system snapshot. If the rebuilding is triggered or the expansion is issued, Longhorn will create a system snapshot before starting the operations. And this system snapshot may not be able to get cleaned up immediately.1
is for the temporary space that may be required by snapshot deletion/purge.Users don’t want snapshot at all. Neither (manually created) snapshot nor recurring job will be launched. Assume setting Automatically Cleanup System Generated Snapshot is enabled, then formula would become:
(1 + 1 + 1) x head/snapshot average actual size
1
means the volume head.1
is the second system snapshot mentioned in the worst case.1
is for the temporary space that may be required by the 2 system snapshot purge/coalescing.Do not retain too many snapshots for the volumes.
Cleaning up snapshots will help reclaim disk space. There are two ways to clean up snapshots:
Also, notice that the extra space, up to volume nominal size
, is required during snapshot cleanup and merge.
An appropriate the volume nominal size
according to the workloads.
© 2019-2024 Longhorn Authors | Documentation Distributed under CC-BY-4.0
© 2024 The Linux Foundation. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our Trademark Usage page.