Recurring Snapshots and Backups
From the Longhorn UI, the volume can refer to recurring snapshots and backups as independent jobs or as recurring job groups.
To create a recurring job, you can go to the Recurring Job
page in Longhorn and Create Recurring Job
or in the volume detail view in Longhorn.
You can configure,
backup
, backup-force-create
, snapshot
, snapshot-force-create
, snapshot-cleanup
, snapshot-delete
or filesystem-trim
full-backup-interval
: Number of incremental backups that must be completed before Longhorn performs a full backup. This integer parameter is applied only to the backup. Notice that if the value is 0, Longhorn performs a incremental backup every time. For more information, see Periodic Full Backup and Create a Backup.Recurring jobs can be set up using the Longhorn UI, kubectl
, or by using a Longhorn RecurringJob
.
To add a recurring job to a volume, you will go to the volume detail view in Longhorn. Then you can set Recurring Jobs Schedule
.
Then Longhorn will automatically create snapshots or backups for the volume at the recurring job scheduled time, as long as the volume is attached to a node. If you want to set up recurring snapshots and backups even when the volumes are detached, see the section Allow Recurring Job While Volume Is Detached
You can set recurring jobs on a Longhorn Volume, Kubernetes Persistent Volume Claim (PVC), or Kubernetes StorageClass.
Note: When the PVC has recurring job labels, they will override all recurring job labels of the associated Volume.
For more information on how snapshots and backups work, refer to the concepts section.
Note: To avoid the problem that recurring jobs may overwrite the old backups/snapshots with identical backups and empty snapshots when the volume doesn’t have new data for a long time, Longhorn does the following:
- Recurring backup job only takes a new backup when the volume has new data since the last backup.
- Recurring snapshot job only takes a new snapshot when the volume has new data in the volume head (the live data).
Recurring snapshots and backups can be configured from the Recurring Job
page or the volume detail page.
You can also configure the recurring job by directly interacting with the Longhorn RecurringJob custom resource.
apiVersion: longhorn.io/v1beta1
kind: RecurringJob
metadata:
name: snapshot-1
namespace: longhorn-system
spec:
cron: "* * * * *"
task: "snapshot"
groups:
- default
- group1
retain: 1
concurrency: 2
labels:
label/1: a
label/2: b
The following parameters should be specified for each recurring job selector:
name
: Name of the recurring job. Do not use duplicate names. And the length of name
should be no more than 40 characters.
task
: Type of the job. Longhorn supports the following:
backup
: periodically create snapshots then do backups after cleaning up outdated snapshots
backup-force-create
: periodically create snapshots then do backups
snapshot
: periodically create snapshots after cleaning up outdated snapshots
snapshot-force-create
: periodically create snapshots
snapshot-cleanup
: periodically purge removable snapshots and system snapshots
Note: retain value has no effect for this task, Longhorn automatically mutates the
retain
value to 0.
snapshot-delete
: periodically remove and purge all kinds of snapshots that exceed the retention count.
Note: The
retain
value is independent of each recurring job.Using a volume with 2 recurring jobs as an example:
snapshot
with retain value set to 5snapshot-delete
: with retain value set to 2Eventually, there will be 2 snapshots retained after a complete
snapshot-delete
task execution.
filesystem-trim
: periodically trim filesystem to reclaim disk space
cron
: Cron expression. It tells the execution time of the job.
retain
: How many snapshots/backups Longhorn will retain for each volume job. It should be no less than 1.
concurrency
: The number of jobs to run concurrently. It should be no less than 1.
Optional parameters can be specified:
groups
: Any groups that the job should belong to. Having default
in groups will automatically schedule this recurring job to any volume with no recurring job.
labels
: Any labels that should be applied to the backup or snapshot.
Default recurring jobs can be set by tick the checkbox default
using UI or adding default
to the recurring job groups
.
Longhorn will automatically add a volume to the default
group when the volume has no recurring job.
Longhorn automatically removes Volume and PVC recurring job labels when a corresponding RecurringJob custom resource is deleted. However, if a recurring job label is added without an existing RecurringJob custom resource, Longhorn does not perform the cleanup process for that label.
The recurring job can be assigned on the volume detail page. To navigate to the volume detail page, click Volume then click the name of the volume.
kubectl
commandAdd recurring job group:
kubectl -n longhorn-system label volume/<VOLUME-NAME> recurring-job-group.longhorn.io/<RECURRING-JOB-GROUP-NAME>=enabled
# Example:
# kubectl -n longhorn-system label volume/pvc-8b9cd514-4572-4eb2-836a-ed311e804d2f recurring-job-group.longhorn.io/default=enabled
Add recurring job:
kubectl -n longhorn-system label volume/<VOLUME-NAME> recurring-job.longhorn.io/<RECURRING-JOB-NAME>=enabled
# Example:
# kubectl -n longhorn-system label volume/pvc-8b9cd514-4572-4eb2-836a-ed311e804d2f recurring-job.longhorn.io/backup=enabled
Remove recurring job:
kubectl -n longhorn-system label volume/<VOLUME-NAME> <RECURRING-JOB-LABEL>-
# Example:
# kubectl -n longhorn-system label volume/pvc-8b9cd514-4572-4eb2-836a-ed311e804d2f recurring-job.longhorn.io/backup-
kubectl
commandBy default, applying a recurring job to a Persistent Volume Claim (PVC) does not have any effect. You can enable or disable this feature using the recurring job source label.
Once the PVC is labeled as the source, any recurring job labels added or removed from the PVC will be periodically synchronized by Longhorn to the associated Volume.
kubectl -n <NAMESPACE> label pvc/<PVC-NAME> recurring-job.longhorn.io/source=enabled
# Example:
# kubectl -n default label pvc/sample recurring-job.longhorn.io/source=enabled
Add recurring job group:
kubectl -n <NAMESPACE> label pvc/<PVC-NAME> recurring-job-group.longhorn.io/<RECURRING-JOB-GROUP-NAME>=enabled
# Example:
# kubectl -n default label pvc/sample recurring-job-group.longhorn.io/default=enabled
Add recurring job:
kubectl -n <NAMESPACE> label pvc/<PVC-NAME> recurring-job.longhorn.io/<RECURRING-JOB-NAME>=enabled
# Example:
# kubectl -n default label pvc/sample recurring-job.longhorn.io/backup=enabled
Remove recurring job:
kubectl -n <NAMESPACE> label pvc/<PVC-NAME> <RECURRING-JOB-LABEL>-
# Example:
# kubectl -n default label pvc/sample recurring-job.longhorn.io/backup-
Recurring job assignment can be configured in the recurringJobSelector
parameters in a StorageClass.
Any future volumes created using this StorageClass will have those recurring jobs automatically assigned.
The recurringJobSelector
field should follow JSON format:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: longhorn
provisioner: driver.longhorn.io
parameters:
numberOfReplicas: "3"
staleReplicaTimeout: "30"
fromBackup: ""
recurringJobSelector: '[
{
"name":"snap",
"isGroup":true
},
{
"name":"backup",
"isGroup":false
}
]'
The following parameters should be specified for each recurring job selector:
name
: Name of an existing recurring job or an existing recurring job group.
isGroup
: is the name that belongs to a recurring job or recurring job group, either true
or false
.
Longhorn provides the setting allow-recurring-job-while-volume-detached
that allows you to do recurring backup even when a volume is detached.
You can find the setting in Longhorn UI.
When the setting is enabled, Longhorn will automatically attach the volume and take a snapshot/backup when it is time to do a recurring snapshot/backup.
Note that during the time the volume was attached automatically, the volume is not ready for the workload. Workload will have to wait until the recurring job finishes.
Longhorn performs delta backups by default, which means that only data that was changed since the last backup is uploaded. However, when a data block in the backupstore becomes corrupted, Longhorn does not replace that data block with a healthy one during subsequent backup operations. Corrupted data blocks in the backupstore may cause restoration operations to fail.
When a non-zero full-backup-interval
parameter is set, Longhorn performs a full backup every full-backup-interval
incremental backups. During a full backup, Longhorn uploads all data blocks in the volume. Data blocks that exist in the backupstore, including corrupted ones, are overwritten.
Important: Performing a full backup might take longer and generate higher network throughput and costs than the default incremental backup.
© 2019-2024 Longhorn Authors | Documentation Distributed under CC-BY-4.0
© 2024 The Linux Foundation. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our Trademark Usage page.