Migrating a gateway's hardware

Replace a gateway computer without losing its configuration or data. The difference from a fresh install boils down to one sentence : the machine is new, the state is not.

That is where almost all the pitfalls on this page come from.

What carries over on its own, what does not

Repris automatiquement	À reprendre à la main
DNS and DHCP for the zone	Service data
Service registry, portal, reverse proxy	Backup repository
Monitoring targets, deployment tags	VPN identity
sops secret recipients	Owners of restored files
Deployment SSH keys	system.stateVersion

Secrets are not tied to the hostname : the sops rules file only knows the administration key and the infrastructure key. Renaming changes nothing here.

Surveying the new hardware

Nothing is modified at this stage : you only collect what will need to be written into the configuration.

1. Boot from the USB drive (just build-iso image), USB Wi-Fi card plugged into the port that will be its final port.

2. Record interfaces, MAC addresses and disks

   ip -o link
   lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE

   The permanent MAC of a Wi-Fi card differs from its current MAC, which is often random. The permanent one matters, visible in the permaddr field.

3. Physically identify which port will be the WAN and which ones the LAN, by unplugging and replugging. Label the ports.

Freeze the USB Wi-Fi card name

A USB card is named after the hub chain that carries it (wlp0s20f0u5u1u3) : moving it to another port renames it, and the configuration can no longer find it. Integrated cards, on the other hand, keep stable names (eno0, wlp4s0) tied to the physical port.

Freeze the only USB card, based on its permanent MAC :

usr/machines/gw/configuration.nix

systemd.network.links."10-wl-ext" = {
  matchConfig.PermanentMACAddress = "00:00:5e:00:53:01";
  linkConfig.Name = "wl-ext";
};

Avoid lan0 (already the bridge) and the eth/wlan prefixes (kernel namespace). NixOS copies the .link files into the initrd, so the names are identical at disk unlock time.

Preparing the configuration

Wi-Fi interfaces do not need to appear in gateway.lan.interfaces : hostapd places its interface in the bridge by itself. If you list one anyway, the framework removes it from the networkd bridge — otherwise networkd would try to attach it before hostapd has switched it to AP mode, fail in a loop, then give up managing the link.

etc/config.yaml

zones:
  maison:
    gateway:
      wan:
        interface: "eno0"
      lan:
        interfaces: ["eno1", "eno2", "eno3"]

hosts:
  - hostname: "gw"
    zone: "maison:1.1"
    profile: "gateway"
    mac: "00:00:5e:00:53:00"
    disko:
      profile: "luks-btrfs-1-disk"
      devices:
        main: "/dev/nvme0n1"

Renaming the host along the way is the right time to do it, since the machine is reinstalled anyway. An alias carrying the new name must then be deleted : the generator refuses a name that is both an alias and a hostname.

On the machine folder side, git mv usr/machines/<old> usr/machines/gw, then delete install/disko.nix (deleting the old disk) and keep install/state.nix. just generate recreates install/disko.nix from the disko block above.

The stateVersion is the one of the OLD machine

The reflex is to delete install/state.nix to start from a fresh installation. That is wrong here : the option fixes the defaults related to persistent data, not the purchase date of the hardware. And these data are those of the old machine.

The PostgreSQL major version depends directly on it. A stateVersion that is too recent selects a more recent major, which creates an empty cluster next to the restored cluster. Symptom : the service starts perfectly and responds, but no longer knows any account.

ls /var/lib/postgresql     # two majors side by side = alert

A missing mount can lock the machine

A mount without nofail that fails makes local-fs.target fail, which switches the boot to emergency mode : no more multi-user.target, hence no SSH. Combined with the absence of a root password and the bootloader editor disabled, the machine becomes inaccessible, console included.

Any mount whose source may be missing at boot (external disk, link on a directory created afterwards) must carry nofail.

Install

just install gw nixos <ip>

The command formats, installs and reboots. On an encrypted disk, the installation passphrase is requested during partitioning : note it down, just luks will need it.

No console access before just configure

On a freshly installed machine, sops has no key : the infrastructure key is only pushed by just configure, and the SSH host key is not a secret recipient. Declared passwords are therefore not decrypted, and no account is usable at the keyboard.

The only way in is the deployment SSH key. If the network does not answer, there is no fallback : plan physical access to the disk before running a remote installation.

Prefer the separate steps to just full-install : on an encrypted disk, waiting for the machine to come back assumes a boot without intervention, which entering the passphrase forbids.

Switch over

Preparing the new gateway behind the old one does not work

The idea seems natural : plug the new hardware into the existing network to configure it before the switch. It fails if both sides are not on disjoint subnets.

The lan0 bridge carries the zone address in /16 and therefore claims the whole address plan. An address obtained on the WAN side within that same /16 becomes unreachable : replies are routed to a bridge with no cable. The symptom is misleading, the machine answers ARP and gets a DHCP lease, yet every port looks filtered.

With a simple modem and no intermediate LAN, the condition cannot be met : switch directly. The old gateway stays re-pluggable within minutes, that is the real safety net.

1. Back up, on the old gateway

   systemctl stop forgejo postgresql
   systemctl start restic-backups-system-main.service

   Do not stop the zone’s DNS resolver : the backup target is referred to by name, stopping it makes it unfindable.

2. Rename the repository, on the backup server

   mv /mnt/backup/restic/<ancien> /mnt/backup/restic/gw

   A repository’s path contains the hostname. Renaming keeps the whole history, whereas a new name would start from an empty repository.

3. Swap the hardware: shut down, move the data disks, identical wiring, boot and unlock the disk.

4. Configure and deploy

   just configure gw
   just apply-verbose gw

   just configure pushes the infrastructure key : this is the step that unlocks sops, and therefore the emergency console access.

5. Republish the backup accounts on the server that hosts them

   just apply-local

The new name still points to the old machine

Until the switch has happened, the new gateway’s name resolves to the zone address, hence to the machine still in service. A deployment launched at that moment would overwrite its configuration.

Check before any deployment :

sudo -u nix ssh -G gw | awk '$1 == "hostname" { print $2 }'

Restoring data

The backup server isolates repositories by host: the authenticated user must match the first path segment. The renamed repository therefore remains inaccessible until the server republishes the new account.

restic-system-main snapshots

This check is the real control: it must list the history of the old machine. An empty repository signals successful authentication but on the wrong path; do not restore anything in that case.

1. Stop the services whose state is about to be replaced.

2. Empty the state directories before restoring.

3. Restore, one mode at a time

   restic-system-main restore latest --target / \
     --include /var/lib/forgejo --include /var/lib/postgresql

4. Restore ownership, by name.

5. Restart the services and verify.

Restoring merges, it does not replace

Between installation and restoration, services have been running on the new machine and writing their own state. Restoration writes over existing data without deleting it, and the two generations get mixed together.

On storage with sequentially numbered files, this is corruption. The time-series database refuses to start because segments are no longer consecutive. Timestamps cleanly separate the two generations.

Hence the rule: empty the state directory before restoring, or stop the affected services on first boot.

Digital identities cannot be transposed

Restoration restores numeric UIDs and GIDs, not names. System accounts without a fixed identifier are numbered in creation order, which differs between machines. A restored directory ends up assigned to an unrelated account, or to none at all.

The error message never mentions permissions in a readable way:

Failed at step CHDIR ... Permission denied

Restore by name, then scan for what remains:

chown -R forgejo:forgejo /var/lib/forgejo
find /var/lib/forgejo \( -nouser -o -nogroup \)

Preserving VPN identity

Restoring the VPN client’s state directory, then restarting its service, makes the new gateway resume the old node’s identity, and thus its mesh address. Nothing to change in etc/config.yaml.

Do this only once the old machine is permanently shut down, never with both running. Otherwise, just tailnet-enroll gw enrolls it as a new machine (the old node, offline, is replaced): record its new address, see Registering a gateway.

Checking before concluding

Contrôle	Attendu
systemctl --failed	aucune unité
ls /var/lib/postgresql	une seule majeure
Comptes et dépôts applicatifs	ceux d’avant la migration
restic-system-main snapshots	historique complet
Adresse sur le maillage VPN	inchangée
Points d’accès Wi-Fi	présents, clients servis
Redémarrage à froid	la machine remonte seule

Hardware keys do not enroll by themselves

On an encrypted disk, the enrollment service runs at boot, before the secrets can be decrypted. It does nothing and exits with success, so it is never replayed. just luks only reruns it if it had to boot a key slot.

Result : no key enrolled, and a manual unlock at every boot. Count the tokens after the first deployment, then rerun the two services if the count is zero.

cryptsetup luksDump /dev/disk/by-partlabel/disk-main-luks | grep -c systemd-fido2

See Chiffrement des disques and YubiKey.

The disko block of etc/config.yaml is only read at installation : once the migration is complete, it is frozen in install/disko.nix and can be removed. Details of the recipes in Justfile and of the configuration keys in Le fichier config.yaml.