Optimised Backups on NixOS with restic & fd — felschr's blog

Optimised Backups on NixOS with restic & fd

I'd like to present a simple restic backup configuration I've been using that allows excluding a lot of unnecessary files via ignore patterns & by respecting .gitignore, thus shrinking backup sizes.

The Problem

Backups are important but not all files need to be backed up. Excluding paths from our backups can be very useful when we know the data is reproducible. E.g., if the target system runs podman containers that are all declared via NixOS options we don't really need most of the data in /var/lib/containers. When using volumes, though, /var/lib/containers/storage/volumes should be backed up. Other examples of files to exclude would be: caches, temporary files, artifacts in development directories. We might also decide that some data, while not reproducible, just isn't important to back up.

Restic

For my personal needs I've decided to use restic as my backup solution. I won't go into the details why I chose it over other solutions, but some of the reasons I went with restic are:

existing NixOS module
many storage options (e.g. Backblaze)
incremental backups
deduplication
strong encryption
zstd compression

In my experience restic is easy to use, fast & has some powerful options.

Finding the right approach

While the NixOS options for restic allow specifying paths to include via services.restic.backups.<name>.paths there is no native option to add exclusion patterns. So I've looked at the other options and found services.restic.backups.<name>.dynamicFilesFrom, which expects a script that produces a list of paths to back up.
This is great. My initial ideas was to generate a list of all files we want to back up, but this would become huge and even worse, restic's parent-snapshot detection will fail when paths used for --files-from change (#2246).

So, instead I need to do the opposite: have few backup paths and exlude a list of files to ignore. Looking further into restic's documentation, I found the option --exclude-file which does exactly that. We can use this option in NixOS via services.restic.backups.<name>.extraBackupArgs

Now that we found a way that allows us to exclude bloat from our backups, let's look how that might look in practise next.

Full solution

This is a simplified version of what I'm using in my desktop configuration. For a complete list check out my actual config. Hopefully some of these ignore patterns will fit your use case.

services.restic.backups.full = {
  initialize = true;
  repository = "SOME-REPO";
  timeConfig.OnCalendar = "daily";
  paths = [ "/etc/nixos" "/var/lib/" "/home" ];
  extraBackupArgs = let
    ignorePatterns = [
      "/var/lib/systemd"
      "/var/lib/containers"
      "/var/lib/flatpak"
      "/home/*/.local/share/Trash"
      "/home/*/.cache"
      "/home/*/Downloads"
      "/home/*/.npm"
      "/home/*/Games"
      "/home/*/.local/share/containers"
      "/home/felschr/dev" # backup ~/dev-backup instead
      ".cache"
      ".tmp"
      ".log"
      ".Trash"
    ];
    ignoreFile = builtins.toFile "ignore"
      (foldl (a: b: a + "\n" + b) "" ignorePatterns);
  in [ "--exclude-file=${ignoreFile}" ];
  pruneOpts = [
    "--keep-daily 7"
    "--keep-weekly 4"
    "--keep-monthly 3"
    "--keep-yearly 1"
  ];
};

Respecting `.gitignore`

As you might have noticed above, I've excluded my ~/dev directory from backups. Since my development folder is huge and has lots of dependencies, build artifacts, etc., I've decided the best way to include it in my backups is to exclude everything that's mentioned in .gitignore files.

For this, I've decided to tackle this by generating ~/dev-backup in the pre-start script of restic's systemd service. In there I use rsync to clone ~/dev to ~/dev-backup. With the --filter option rsync supports exclude files like .gitignore natively. By also providing the --link-dest option, we tell rsync to create links to the original files instead of duplicating them. This way we don't have any storage overhead for this setup.

The resulting configuration looks like this:

systemd.services."restic-backups-full" = {
  preStart = ''
    rm -rf /home/felschr/dev-backup
    ${pkgs.rsync}/bin/rsync \
      -a --delete \
      --filter=':- .gitignore' \
      --link-dest=/home/felschr/dev \
      /home/felschr/dev/ /home/felschr/dev-backup
  '';
};

I hope you found this small guide useful and I could spare you some of the pitfalls I ran into before.

My full restic config can be found here: https://gitlab.com/felschr/nixos-config/tree/main/services/restic

UPDATE 2023-01-07

The article has been adjusted to fix an issue in the original approach. For more details of the changes check out the git history of this article.