Tarballs

If you backup tarballs as part of your source directories, Borg can't effectively deduplicate their contents because of the way that a tarball's files are packed together (and often compressed). This isn't a big deal for small tarballs, but it can really hurt for large ones like virtual machine exports.

New in version 2.1.11 Beta feature borgmatic supports configuring tarballs as data sources for backups, unpacking their contained files directly into a Borg archive so that deduplication works properly.

To take advantage of this feature, use the tarballs: hook. For example:

tarballs:
    - name: virtual-machine
      path: /var/lib/vz/dump/vzdump-vm.tar

And to restore such a tarball to its original path:

borgmatic restore --archive latest --data-source virtual-machine

During backup, borgmatic streams each tarball to Borg, unpacking its files directly into the created Borg archive. Under the hood, this doesn't use Borg's native import-tar, because that's limited to a single tarball per archive—and no additional source directories.

Upon restore, borgmatic exports the tarball from the archive, rewriting it on the fly to inject its original contained file permissions, ownership, symlinks, etc. The resulting tarball ends up at its original path (or restore_path).

Both create and restore operate without using any temporary files for tarball data.

Limitations

• Within an archive, tarball contents are not stored at their original path. Instead, they are stored within borgmatic/tarballs/ (with Borg 1.4+) so that the restore action can find them.
• Due to the way that borgmatic streams tarballs to Borg via named pipes, per-file information (permissions, ownership, symlink targets, etc.) is stored separately from file contents in a per-tarball manifest file within borgmatic/tarballs/. This is consumed by the restore action.
• During restore, borgmatic attempts to match the compression (if any) of the original tarball based on its file extension. If it can't determine the compression method, it errors.
• If a tarball that you've configured under tarballs: also appears in your source directories, it will get backed up twice—once tarred and once untarred. You can of course add an exclude if you'd like a tarball to only get backed up once.
• When this hook is enabled, borgmatic instructs Borg to consume special files (via --read-special) to support tarball streaming—regardless of the value of your read_special configuration option.
• Tarballs backed up and restored with this data source hook are not bit-identical to their corresponding original tarballs, even though their file contents and metadata should be the same.
• During the restore action, borgmatic removes any existing tarball already present at the restore path before replacing it with the restored tarball.
• Currently, this hook only supports the following compression types on tarballs: gzip, bzip2, xz, and no compression.

Full configuration

Here's an example configuration with all of the available options for this feature in the most recent version of borgmatic. If you're using an older version, some of these options may not work, and you should instead generate a sample configuration file specific to your borgmatic version.

# List of one or more tarballs to include in backups. borgmatic adds
# such a tarball to your source directories at runtime and streams it
# directly to Borg as the individual files it contains—thereby
# allowing Borg to effectively deduplicate your file data across
# archives. In your configured source directories, you can omit or
# exclude any tarballs that you specify here.
tarballs:
    # This is used to tag the tarball with a name. It is not
    # the path to the tarball itself. The name "all" has no
    # special meaning for tarballs.
    - name: source_code

      # Path to the tarball file to backup. If relative, it is
      # relative to the current working directory.
      path: /var/lib/vz/dump/vzdump-vm.tar

      # Label to identify the tarball in the backup.
      label: my_backup_label

      # Path to the tarball file to restore to. Defaults to the
      # "path" option.
      restore_path: /var/lib/vz/dump/vzdump-vm.tar