Maintenance and Troubleshooting

The administrator’s daily routine: keeping the fleet clean, intervening on a host, rolling back after a bad configuration, reading logs.

Acting on a host

All these commands target a host (or a pattern / list) via colmena:

Command	Effect
just enter <host>	Open an SSH shell (alias e)
just reboot <host>	Reboot
just halt <host>	Shut down
just gc <host> [keep]	Purge generations + clean the store
just fix-boot <host>	Reinstall the boot loader

Multiple hosts at once

The target accepts a pattern or a list: just gc '*', just reboot a,b.

The keep argument of just gc limits the purge:

• just gc <host>: keeps only the current generation.
• just gc <host> 3: keeps the last 3 generations, including the current one.
• just gc <host> 3d: removes generations older than 3 days.

Cleaning and formatting code

Before committing or deploying, normalize the repository:

just clean      # fix + check + generate + format

• just check: code + unit tests + flake checks (before deployment).

Rollback

Each deployment creates a NixOS generation. In case of issues:

1. At boot : choose a previous generation from the boot menu.

2. Live rollback : revert to the previous generation then make it active:

   just enter <host>
   sudo nixos-rebuild switch --rollback

3. Through code : git revert the faulty configuration, then redeploy:

   just apply <host>

GC warning

just gc removes old generations : you cannot revert to them at boot anymore. Verify the host is stable before cleaning.

Reading logs

just enter <host>
journalctl -u <service> -n 100 --no-pager     # logs d'un service
journalctl -p err -b                          # erreurs depuis le démarrage
systemctl status <service>                    # état d'une unité

Guided diagnostics

For centralized monitoring (metrics + Caddy logs), see Monitoring. For identity replication health: just idm-fix-replica <hcs> <gateways>.

Services failing after a gateway reboot

Symptom: systemctl --failed lists, right after boot, units that start back up without a hitch once manually restarted (NFS mounts, OIDC provisioning, anything that resolves a name).

Cause: on a gateway, the system resolver is AdGuard Home (nameserver 127.0.0.1). Its unit is Type=simple, so systemd declares it active as soon as the process starts — about twenty seconds before it actually listens on port 53. Anything resolving a name during that window fails.

The framework puts up a barrier for this: dns-ready.service queries the local resolver until it gets a response, and blocks nss-lookup.target until then.

systemctl status dns-ready              # when names got resolved
journalctl -b -u adguardhome | head -30 # when the DNS proxy finally listens

A unit that resolves a name at boot must therefore order itself after this target — after = [ "nss-lookup.target" ] for a service, the mount option x-systemd.after=nss-lookup.target for a mount point. On non-gateway hosts, the target is reached immediately: no added delay.

A service that depends on another host

Local ordering says nothing about the availability of a remote machine (the IdM, an NFS server, a host still booting). For these dependencies, the only reliable answer is to retry: Restart + RestartSec on the unit, rather than a definitive failure on every reboot.

Expired TLS certificates on a gateway

A gateway does not request its own certificates: the HCS obtains them, and the sync-caddy-certs unit copies them every ten minutes from the HCS to the local Caddy storage.

systemctl status sync-caddy-certs   # dernier passage et son résultat
journalctl -u sync-caddy-certs -n 50
systemctl list-timers sync-caddy-certs

The transfer goes through two directories:

Path	Role
/var/lib/caddy-cert-sync	Staging area (StateDirectory of the unit)
/var/lib/caddy/storage	Actual Caddy storage being served

The unit runs as root, pulls from the HCS using the nix user’s SSH key, then publishes to the Caddy storage. It first waits for the tailnet to be actually mounted (up to five minutes), then retries the transfer three times before giving up. A systemctl status showing failed therefore remains the signal to watch, and a repeated failure indicates an expired certificate.

Why this wait

On first boot of a gateway, tailscaled may be running while the tailnet is not yet reachable: the WAN link — and the router behind it — can sometimes take several minutes, and the daemon then enters an exponential backoff before retrying its connection to the control plane. The timer used to fire straight through this gap and leave the unit in failed until the next run. Ordering behind tailscaled.service is not enough: it only proves the daemon has started, hence the explicit wait on BackendState and Self.Online.

Empty staging area

Publication is refused if the staging area contains no file (message empty staging dir, refusing to publish). Without this safeguard, the sync would delete the certificates in place and the ACME account key. If this message appears, the issue is upstream: SSH access to the HCS, tailnet, or empty Caddy storage on the HCS side.

Impact on exposed services

The HCS verifies the certificate of the gateway it relays to. A gateway whose sync never succeeded therefore returns 502 on its exposed services, instead of relaying to a certificate that cannot be verified. First step: systemctl start sync-caddy-certs on the gateway.

Misaligned gdm-greeter UIDs

Symptom, on every GNOME workstation startup:

warning: not applying UID change of user 'gdm-greeter-2' (60580 -> 60579)

nixpkgs renumbered its gdm-greeter* accounts; NixOS never renumbers an existing account. Workstations installed before keep the old UIDs, and two greeters can end up sharing the same one.

just fix-gdm-greeters poste-01,poste-02         # diagnostic, writes nothing
just fix-gdm-greeters poste-01,poste-02 apply

apply stops GDM, recreates the misaligned accounts with the declared UIDs, reassigns their files, then restarts GDM. A host with an open graphical session is left aside: restart after logging out. Backup of /etc/passwd and /etc/shadow in /var/lib/nixos/backup-gdm-greeters-<date>/.