Maintenance and Troubleshooting

The administrator’s daily routine: keeping the fleet clean, intervening on a host, rolling back after a bad configuration, reading logs.

Acting on a host

All these commands target a host (or a pattern / list) via colmena:

Command	Effect
just enter <host>	Open an SSH shell (alias e)
just reboot <host>	Reboot
just halt <host>	Shutdown
just gc <host>	Purge generations + clean the store
just fix-boot <host>	Reinstall the boot loader

Multiple hosts at once

The target accepts a pattern or list: just gc '*', just reboot a,b.

Clean and format code

Before committing or deploying, normalize the repository:

just clean      # fix + check + generate + format

• just check-all : unit tests + flake checking (before deployment).
• just gc <host> : reclaim disk space (deletes old generations).

Rollback

Each deployment creates a NixOS generation. In case of issues:

1. At boot : choose a previous generation from the boot menu.

2. Live rollback : revert to the previous generation then make it active:

   just enter <host>
   sudo nixos-rebuild switch --rollback

3. Through code : git revert the faulty configuration, then redeploy:

   just apply <host>

GC warning

just gc removes old generations : you cannot revert to them at boot anymore. Verify the host is stable before cleaning.

Reading logs

just enter <host>
journalctl -u <service> -n 100 --no-pager     # logs d'un service
journalctl -p err -b                          # erreurs depuis le démarrage
systemctl status <service>                    # état d'une unité

Guided diagnostics

For centralized monitoring (metrics + Caddy logs), see Monitoring. For identity replication health: just idm-fix-replica <hcs> <gateways>.

Services failing after a gateway reboot

Symptom: systemctl --failed lists, right after boot, units that start back up without a hitch once manually restarted (NFS mounts, OIDC provisioning, anything that resolves a name).

Cause: on a gateway, the system resolver is AdGuard Home (nameserver 127.0.0.1). Its unit is Type=simple, so systemd declares it active as soon as the process starts — about twenty seconds before it actually listens on port 53. Anything resolving a name during that window fails.

The framework puts up a barrier for this: dns-ready.service queries the local resolver until it gets a response, and blocks nss-lookup.target until then.

systemctl status dns-ready              # when names got resolved
journalctl -b -u adguardhome | head -30 # when the DNS proxy finally listens

A unit that resolves a name at boot must therefore order itself after this target — after = [ "nss-lookup.target" ] for a service, the mount option x-systemd.after=nss-lookup.target for a mount point. On non-gateway hosts, the target is reached immediately: no added delay.

A service that depends on another host

Local ordering says nothing about the availability of a remote machine (the IdM, an NFS server, a host still booting). For these dependencies, the only reliable answer is to retry: Restart + RestartSec on the unit, rather than a definitive failure on every reboot.

Expired TLS certificates on a gateway

A gateway does not request its own certificates: the HCS obtains them, and the sync-caddy-certs unit copies them every ten minutes from the HCS to the local Caddy storage.

systemctl status sync-caddy-certs   # last run and its result
journalctl -u sync-caddy-certs -n 50
systemctl list-timers sync-caddy-certs

The transfer goes through two directories:

Path	Role
/var/lib/caddy-cert-sync	Staging area (StateDirectory of the unit)
/var/lib/caddy/storage	Actual Caddy storage being served

The unit runs as root, pulls from the HCS using the SSH key of the nix user, then publishes to the Caddy storage. It fails outright on the first error: a systemctl status showing failed is therefore the signal to watch for, and repeated failures indicate an upcoming certificate expiration.

Empty staging area

Publishing is refused if the staging area contains no files (empty staging dir, refusing to publish). Without this safeguard, synchronization would erase the certificates in place and the ACME account key. If this message appears, the issue is upstream: SSH access to the HCS, tailnet, or empty Caddy storage on the HCS side.