📏 How to deal with very large backups

Borg itself is great for efficiently de-duplicating data across successive backup archives, even when dealing with very large repositories. But you may find that while borgmatic's default actions of create, prune, compact, and check works well on small repositories, it's not so great on larger ones. That's because running the default pruning, compact, and consistency checks take a long time on large repositories.

See the actions documentation for details on customizing the actions that borgmatic runs.

Consistency check configuration

Another way of dealing with large backups is to customize your consistency checks. By default, if you omit consistency checks from configuration, borgmatic runs full-repository checks and per-archive checks within each repository on a monthly basis.

But if you find that archive checks are too slow and/or you'd like to customize the check frequency, see the consistency checks documentation for details.

Pre-backup safety validation

Before running a backup during the create action, borgmatic automatically runs through some pre-backup safety validation to ensure that:

1. You haven't accidentally excluded borgmatic's runtime directory from the backup, which would break things like database dumps, filesystem snapshots, etc.
2. Special files are auto-excluded to prevent Borg from hanging.

This validation does have a cost: performance. On a large filesystem, it can take a while to run. So if you are absolutely sure that you aren't excluding borgmatic's runtime directory, and you also aren't including any special files that might cause Borg to hang, you can disable the pre-backup validation as follows:

unsafe_skip_path_validation_before_create: true

However, this is indeed unsafe and could lead to hangs or data being left out of backups. Use this option at your own risk.

Troubleshooting

Broken pipe with remote repository

When running borgmatic on a large remote repository, you may receive errors like the following, particularly while "borg check" is validating backups for consistency:

    Write failed: Broken pipe
    borg: Error: Connection closed by remote host

This error can be caused by an ssh timeout, which you can rectify by adding the following to the ~/.ssh/config file on the client:

    Host *
        ServerAliveInterval 120

This should make the client keep the connection alive while validating backups.