Felix Schröter
felschr's dev blog

felschr's dev blog

Optimised Backups on NixOS with restic & fd

Optimised Backups on NixOS with restic & fd

Felix Schröter's photo
Felix Schröter
·May 28, 2022·

3 min read

UPDATE 2022-07-04

After using this approach for a while I noticed a couple of downsides. The first most obvious is that all paths created by the dynamicFilesFrom clutter the logs. More problematic is that restic's incremental backups only work properly when backup paths don't change (#2246). This causes restic to rescan all files on every run, and the prune job won't clean up all past backups. Thus, my current recommendation is using paths and excluding files by using --exclude-file in extraBackupArgs. I'll update this post later with full instructions & a way to still backups source directories while respecting .gitignore files.

Introduction

I'd just like to present a simple restic backup configuration I've been using that allows excluding a lot of unnecessary files via ignore patterns & by respecting .gitignore, thus shrinking backup sizes.

While the NixOS options for restic allow specifying paths to include via services.restic.backups.<name>.paths there is no native option to add exclusion patterns. However, we can use services.restic.backups.<name>.dynamicFilesFrom which expects a script that produces a list of paths to back up. This allows us to use other tools to create more granular inclusions.

Path matching

I've opted to use fd in my configuration to generate the inclusion list. I've previously tried using rg --files but it doesn't match empty directories which can become an issue with services that expect certain paths to exist (PostgreSQL is one such example).

By default, fd ignores hidden files & respects various ignore files (.gitignore, .ignore & .fdignore). We definitely want to include hidden files in backups, so we'll use fd's --hidden argument. Respecting ignore files on the other hand can be very useful to exclude caches & artefacts which can easily be reproduced. If you don't want this behaviour, it can be disabled with --no-ignore.

Excluding

Excluding paths from our backups can be very useful when we know the data is reproducible. E.g., if the target system runs podman containers that are all declared via NixOS options we don't really need most of the data in /var/lib/containers. When using volumes, though, /var/lib/containers/storage/volumes should be backed up.

Full solution

This is a simplified version of what I'm using in my desktop configuration. Hopefully some of these ignore patterns will fit for your use case.

Note the use of sed to escape [ & ], restic would otherwise complain about these paths.

services.restic.backups.full = let
  paths = [ "/etc/nixos" "/var/lib/" "/home" ];
  ignorePatterns = [
    "/var/lib/systemd"
    "/var/lib/containers"
    "/var/lib/lxcfs"
    "/var/lib/docker"
    "/var/lib/flatpak"
    "/home/*/.local/share/Trash"
    "/home/*/.cache"
    "/home/*/Downloads"
    "/home/*/.npm"
    "/home/*/Games"
    "/home/*/.steam"
    "/home/*/.local/share/containers"
    "/home/*/.local/share/Steam"
    "/home/*/.local/share/lutris"
    "**/.git"
  ];
in {
  initialize = true;
  repository = "SOME-REPO";
  timeConfig.OnCalendar = "daily";
  dynamicFilesFrom = let
    paths_ = foldl (a: b: "${a} ${b}") "" paths;
    ignoreFile = builtins.toFile "ignore"
      (foldl (a: b: a + "\n" + b) "" ignorePatterns);
  in ''
    ${pkgs.fd}/bin/fd \
      --hidden \
      --ignore-file ${ignoreFile} \
      . ${paths_} \
      | sed "s/\[/\\\[/" | sed "s/\]/\\\]/"
  '';
};

My full restic config can be found here: gitlab.com/felschr/nixos-config/tree/main/s..

 
Share this