Network backups (restic)

The darkone.service.restic module backs up fleet hosts with restic 🡕: each host (client) pushes its data to a repository, either local or remote via a zone REST server.

Principles

Two roles, one module:

• REST server: a host that declares the restic service in config.yaml. It stores all hosts’ repositories and listens on the internal network.
• Client: any host with enable = true that declares targets (backup destinations).

Two passwords not to be confused

• Repository passphrase restic-password-<zone>: encrypts the content of the repository. Without it, no restoration is possible.
• REST credential restic/<host>/rest-password: authenticates the host with the REST server. One per host.

Both are generated by just configure-admin-host (see below).

Repository layout: one repository per category, named after it.

<root>/<host>/system  # category "system" (/ minus exclusions)
<root>/<host>/nfs     # category "nfs"    (/srv/nfs/<...>)
<root>/<host>/medias  # category "medias" (/srv/medias/<...>)

The repository path is exactly two levels deep

rest-server only serves repositories located two components under its data directory: <host>/medias responds, <host>/srv/medias returns 404 — and restic then displays the misleading message repository does not exist even though the data is intact.

The subpath is therefore the category name, never the source directory. Since this layout is identical for local and REST, a repository can move from one to the other without being renamed.

Repositories created before this layout

Repositories written when the subpath copied the source directory are located under <host>/srv/nfs or <host>/srv/medias. Rename them on the server before the next run: with initialize set to true, restic would otherwise create an empty repository at the new path and silently start a full backup — several TB over a slow link.

mv <data>/<host>/srv/medias <data>/<host>/medias
rmdir <data>/<host>/srv

Renaming within the same filesystem is instant, moves no data, and preserves history and incrementality.

Setting up a backup server

1. In etc/config.yaml, add the restic service to the server host:

   services:
     restic:

2. Plug in the backup disk and mount it on /mnt/backup in the host configuration (usr/machines/<host>/), with the nofail option:

   fileSystems."/mnt/backup" = {
     device = "/dev/disk/by-uuid/<uuid>";
     fsType = "ext4";
     options = [ "nofail" ];
   };
   
   # The REST server only starts once the disk is mounted.
   systemd.services.restic-rest-server = {
     requires = [ "mnt-backup.mount" ];
     after = [ "mnt-backup.mount" ];
   };

3. The repository is stored under serverDataDir (default /mnt/backup/restic). The firewall opens the REST port on the internal interface automatically.

4. If other zone clients need to access it via the tailnet, add listenAll: by default the server only listens on the host’s LAN address, invisible from the VPN.

   darkone.service.restic = {
     enable = true;
     enableServer = true;
     listenAll = true;
   };

Server = client too

A server can back up its own data by pointing a target to its local disk (root = "/mnt/backup/restic").

restic-rest-server.service inactive: it's normal

The REST server is socket-activated. restic-rest-server.service therefore remains inactive until a client connects — after every boot, and as long as a disk using x-systemd.automount stays unused (the requires on the mount then stops the daemon so it can be unmounted). It is restic-rest-server.socket, always listening, that carries the ServiceDown monitoring; a crashed daemon is still covered by SystemdUnitFailed.

Adding a client

Declare one or more targets in the host configuration:

darkone.service.restic = {
  enable = true;
  targets = [
    {
      # Remote repository via the zone's REST server.
      root = "rest:http://restic.home.example.net:8888";
      zone = "home";
      categories = [ "system" "nfs" ];
    }
  ];
};

target fields:

Field	Role
name	Identifier (default main), suffix for jobs/units.
root	Local path or rest:// URL.
zone	Zone that selects the restic-password-<zone> passphrase.
categories	Data to back up: system, nfs, medias.

Multiple destinations

Add a second target (different name, different root/zone) to replicate the backup, for example to the REST server of another zone.

Off-site backup

A second target pointing to another zone’s REST server provides the “1” in the 3-2-1 rule. Two precautions specific to off-site backups.

Address the server by its tailnet IP, never by its zone name:

{
  name = "offsite";
  root = "rest:http://100.64.0.20:8888";
  zone = "home";
  categories = [ "system" ];
}

A name like restic.<remote-zone>.example.net is only resolved by that zone’s gateway. If the remote site goes down, the resolver goes down with it and the failure appears as a DNS resolution error, which misleads diagnostics. The tailnet IP removes this indirection, along with the dependency on the remote site’s subnet route. It requires listenAll on the server side (see above).

An absent off-site server does not fail the unit

Each rest: target carries a reachability probe (ExecCondition). Server unreachable: the unit is skipped, not failed — no SystemdUnitFailed alert at night, no quarter-hour of restic retries, and no empty repository created against a dead address. The missed backup is still reported by ResticBackupStale (36 h) then ResticBackupCritical (7 days), which monitor the date of the last success. The journal then shows Skipped due to ExecCondition=.

Generate passwords

The generation of internal secrets is idempotent: it creates what is missing, never overwrites.

just configure-admin-host

It generates, in usr/secrets/secrets.yaml:

• a restic-password-<zone> passphrase per zone;
• a restic/<host>/rest-password credential per host in the fleet.

New host

Run just configure-admin-host again after adding a host, then redeploy the servers so they pick up its new REST account.

Running a manual backup

The module creates one systemd service per target × category pair, named restic-backups-<category>-<name> (the <name> comes from the target’s name field, main by default). Examples: restic-backups-system-main, restic-backups-nfs-main.

1. List the backup services present on the host:

   systemctl list-units 'restic-backups-*'

2. Trigger an immediate backup without waiting for the timer:

   sudo systemctl start restic-backups-system-main.service

3. Follow its progress in real time:

   journalctl -fu restic-backups-system-main.service

Simulate without writing anything

Setting darkone.service.restic.enableDryRun = true adds --dry-run to all jobs: the backup runs in dry mode, no snapshot is created.

Repository isolation

The REST server runs with private repositories (privateRepos = true): each host only accesses the repository whose path starts with its own hostname. The module automatically aligns the REST account (<hôte>) with the repository prefix (<racine>/<hôte>/...), so isolation is permanent → no configuration needed. An attempt to access another host’s repository returns a 403 error.

Restore

On the host in question, the secrets are already in place. List and restore:

# Adapter la zone du dépôt (et la catégorie : system, nfs, medias).
ZONE=home

# Variables d'accès REST (générées par sops) et passphrase de dépôt.
set -a; . /run/secrets/rendered/restic-rest-env; set +a
export RESTIC_PASSWORD_FILE=/run/secrets/restic-password-$ZONE
export RESTIC_REPOSITORY=rest:http://restic.$ZONE.example.net:8888/$(hostname)/system

restic snapshots
restic restore latest --target /restore

Adapter zone and category

Set ZONE to the repository zone and replace system in RESTIC_REPOSITORY with the category (nfs, medias) to restore.

Verifying

1. On a server, one account per fleet host in the htpasswd file:

   sudo cut -d: -f1 /run/restic-rest/htpasswd

2. On a client, run a manual backup (see above) and check that it completes without errors.

3. Isolation: from a client, accessing another host’s repository must be denied (401/403 error).