Migrating a gateway's hardware

Replace a gateway computer without losing its configuration or data. The difference from a fresh install boils down to one sentence : the machine is new, the state is not.

That is where almost all the pitfalls on this page come from.

What carries over on its own, what does not

Repris automatiquement	À reprendre à la main
DNS and DHCP for the zone	Service data
Service registry, portal, reverse proxy	Backup repository
Monitoring targets, deployment tags	VPN identity
sops secret recipients	Owners of restored files
Deployment SSH keys	system.stateVersion

Secrets are not tied to the hostname : the sops rules file only knows the administration key and the infrastructure key. Renaming changes nothing here.

Record the new hardware

Nothing is modified at this stage : you collect what will need to be written into the configuration.

1. Boot from the USB drive (just build-iso image), USB Wi-Fi card plugged into the port that will be its final port.

2. Record interfaces, MAC addresses and disks

   ip -o link
   lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE

   A Wi-Fi card’s permanent MAC differs from its current, often random, MAC. The permanent one is what matters, visible in the permaddr field.

3. Physically identify which port will be the WAN and which the LAN, by unplugging and replugging. Label the sockets.

Freezing the USB Wi-Fi card name

A USB card is named after the hub chain that carries it (wlp0s20f0u5u1u3) : moving it to another port renames it, and the configuration no longer finds it. Built-in cards, for their part, keep stable names (eno0, wlp4s0) tied to the physical socket.

Freeze the only USB card, using its permanent MAC :

usr/machines/gw/default.nix

systemd.network.links."10-wl-ext" = {
  matchConfig.PermanentMACAddress = "00:00:5e:00:53:01";
  linkConfig.Name = "wl-ext";
};

Avoid lan0 (already the bridge) and the eth/wlan prefixes (kernel namespace). NixOS copies the .link files into the initrd, so the names are identical at disk unlock.

Prepare the configuration

The Wi-Fi interfaces do not need to appear in gateway.lan.interfaces : hostapd places its own interface in the bridge.

etc/config.yaml

zones:
  maison:
    gateway:
      wan:
        interface: "eno0"
      lan:
        interfaces: ["eno1", "eno2", "eno3"]

hosts:
  - hostname: "gw"
    zone: "maison:1.1"
    profile: "gateway"
    mac: "00:00:5e:00:53:00"
    disko:
      profile: "luks-btrfs-1-disk"
      devices:
        main: "/dev/nvme0n1"

Renaming the host along the way is the right time, since the machine is being reinstalled anyway. An alias carrying the new name must then be removed : the generator refuses a name being both an alias and a hostname.

On the machine folder side, git mv usr/machines/<ancien> usr/machines/gw, then import ./generated-configuration.nix and ./disko.nix in default.nix.

The stateVersion is that of the OLD machine

The reflex is to take the value from the fresh install template. That is misguided here : the option sets the defaults tied to persistent data, not the hardware’s purchase date. Yet that data is the old machine’s.

The PostgreSQL major version depends directly on it. A too-recent stateVersion selects a more recent major version, which creates an empty cluster next to the restored one. Symptom : the service starts perfectly and answers, but no longer knows any account.

ls /var/lib/postgresql     # two majors side by side = alert

A missing mount can lock the machine

A mount without nofail that fails makes local-fs.target fail, which drops the boot into emergency mode : no more multi-user.target, therefore no more SSH. Combined with no root password and a disabled bootloader editor, the machine becomes unreachable, console included.

Any mount whose source can be missing at boot (external disk, link to a directory created afterwards) must carry nofail.

Install

just install gw nixos <ip>

The command formats, installs and reboots. On an encrypted disk, the installation passphrase is requested during partitioning : note it down, just luks will need it.

No console access before just configure

On a freshly installed machine, sops has no key : the infrastructure key is only pushed by just configure, and the SSH host key is not a secret recipient. Declared passwords are therefore not decrypted, and no account is usable at the keyboard.

The only way in is the deployment SSH key. If the network does not answer, there is no fallback : plan physical access to the disk before running a remote installation.

Prefer the separate steps to just full-install : on an encrypted disk, waiting for the machine to come back assumes a boot without intervention, which entering the passphrase forbids.

Switch over

Preparing the new gateway behind the old one does not work

The idea seems natural : plug the new hardware into the existing network to configure it before the switch. It fails if both sides are not on disjoint subnets.

The lan0 bridge carries the zone address in /16 and therefore claims the whole address plan. An address obtained on the WAN side within that same /16 becomes unreachable : replies are routed to a bridge with no cable. The symptom is misleading, the machine answers ARP and gets a DHCP lease, yet every port looks filtered.

With a simple modem and no intermediate LAN, the condition cannot be met : switch directly. The old gateway stays re-pluggable within minutes, that is the real safety net.

1. Back up, on the old gateway

   systemctl stop forgejo postgresql
   systemctl start restic-backups-system-main.service

   Do not stop the zone’s DNS resolver : the backup target is referred to by name, stopping it makes it unfindable.

2. Rename the repository, on the backup server

   mv /mnt/backup/restic/<ancien> /mnt/backup/restic/gw

   A repository’s path contains the hostname. Renaming keeps the whole history, whereas a new name would start from an empty repository.

3. Swap the hardware: shut down, move the data disks, identical wiring, boot and unlock the disk.

4. Configure and deploy

   just configure gw
   just apply-verbose gw

   just configure pushes the infrastructure key : this is the step that unlocks sops, and therefore the emergency console access.

5. Republish the backup accounts on the server that hosts them

   just apply-local

The new name still points to the old machine

Until the switch has happened, the new gateway’s name resolves to the zone address, hence to the machine still in service. A deployment launched at that moment would overwrite its configuration.

Check before any deployment :

sudo -u nix ssh -G gw | awk '$1 == "hostname" { print $2 }'

Restore the data

The backup server isolates repositories per host : the authenticated user must equal the path’s first segment. The renamed repository therefore stays inaccessible until the server has republished the new account.

restic-system-main snapshots

This check is the real control : it must list the old machine’s history. An empty repository signals a successful authentication but against the wrong path, restore nothing in that case.

1. Stop the services whose state is going to be replaced.

2. Empty the state directories before restoring.

3. Restore, one mode at a time

   restic-system-main restore latest --target / \
     --include /var/lib/forgejo --include /var/lib/postgresql

4. Restore the owners, by name.

5. Restart the services and check.

Restore merges, it does not replace

Between installation and restore, the services ran on the new machine and wrote their own state. The restore writes over without deleting what already exists, and the two generations mix together.

On storage with sequentially numbered files, that is corruption. The time series database refuses to start, the segments no longer being consecutive. Timestamps separate the two generations unambiguously.

Hence the rule : empty the state directory before restoring it, or stop the services concerned from the very first boot.

Digital identities do not carry over

Restore reinstates numeric UIDs and GIDs, not names. Yet system accounts without a fixed identifier are numbered in order of creation, which differs from one machine to another. A restored directory ends up assigned to an unrelated account, or even to none.

The error message never mentions permissions in a readable way :

Failed at step CHDIR ... Permission denied

Restore by name, then sweep what remains :

chown -R forgejo:forgejo /var/lib/forgejo
find /var/lib/forgejo \( -nouser -o -nogroup \)

Keeping the VPN identity

Restoring the VPN client’s state directory, then restarting its service, makes the new gateway take over the old one’s node identity, and therefore its address on the mesh. Nothing to change in etc/config.yaml.

Only do this once the old machine is definitively powered off, never with both in service. Otherwise, follow Enregistrer une passerelle and report the new address.

Verify before concluding

Contrôle	Attendu
systemctl --failed	no failed units
ls /var/lib/postgresql	a single major version
Application accounts and repositories	the ones from before the migration
restic-system-main snapshots	full history
Address on the VPN mesh	unchanged
Wi-Fi access points	present, clients served
Cold reboot	the machine comes back on its own

Hardware keys do not enroll by themselves

On an encrypted disk, the enrollment service runs at boot, before secrets can be decrypted. It does nothing and exits successfully, so it is never replayed. just luks only reruns it if it had to boot a key slot.

Result : no enrolled key, and a manual unlock at every boot. Count the tokens after the first deployment, then restart both services if the count is zero.

cryptsetup luksDump /dev/disk/by-partlabel/disk-main-luks | grep -c systemd-fido2

See Chiffrement des disques and YubiKey.

The disko block in etc/config.yaml is only read at install time, comment it out once the migration is complete. Recipes detail in Justfile and configuration keys in Le fichier config.yaml.