Deployment and updates

The fleet is deployed with colmena 🡕: a single command builds and applies the configuration on one or more hosts, remotely, from the admin workstation.

Apply a configuration

just apply <target> [action]    # alias : a

• target = hostname, pattern ('*'), list (a,b) or colmena tag (@server).
• action = switch (default), boot, test or build.

Command	Purpose
just apply <target>	Build + activate on the target(s)
just apply-local	Apply on the current machine (alias al)
just apply-verbose <target>	Same as apply in trace mode (alias av)

Build on target

apply builds on the target host (--build-on-target): the admin workstation doesn’t need the power or architecture of the deployed machines.

Deploy safely

Build confidence through action: each step is less risky than the next.

just apply <host> build      # downloads + compiles: 100% safe, nothing is activated
just apply <host> test       # activates without switching: no generation, no boot
just apply <host>            # switch: activates and creates a new generation

Progress from core to edge: deploy nodes in this order, to never cut off access to a node by the one preceding it.

Why this order

A remote host’s connectivity depends on its gateway, which depends on the VPN coordinated by the HCS. Deploying a link before its support can make subsequent hosts unreachable.

Update

The system is pinned by flakes. Updating = refresh the inputs, then redeploy.

just update-flake     # updates dnf/ + root, commits the locks
just apply '*'        # deploys the update

Always test first

Before a large deployment, validate locally: just check-flake (or just check-all), then just apply <pilot-host> on one test host.

Binary cache

Two services combine to avoid recompiling or re-downloading derivations on each deployment.

Service	Role
harmonia	Serves the local /nix/store, signed, directly to hosts (LAN, or VPN if global).
nix-cache	Per-zone nginx proxy that caches the public cache.nixos.org cache on the gateway.

A host queries these sources in the following priority order. The first fetch from the Internet is thus shared across the whole zone.

Priority	Source	Scope
20	zone harmonia	LAN, what the zone has compiled
35	zone nix-cache proxy	LAN, shared mirror of the public cache
40	cache.nixos.org directly	safety net if the gateway is down
45	global harmonia	last resort, over the VPN

How the order is guaranteed

Nix does not follow the order of the substituters list; it sorts caches according to the priority each one announces in its nix-cache-info. But these announced values are equal: all harmonia services declare 30, and the proxy relaying cache.nixos.org as-is declares 40, like direct access to the public cache. DNF therefore sets the priority in the substituter URL (?priority=35), which overrides the announced value and makes the table order explicit.

Two intended consequences. The zone proxy goes before direct access to the public cache, otherwise zone sharing would never happen. And a global harmonia goes after this public cache: harmonia serves a whole /nix/store without being able to tell what it compiled from the rest, so a well-stocked host would capture all the traffic the public cache is meant to serve. As a last resort, it is only consulted for what cache.nixos.org does not have, that is, its own builds.

A host does not use its own harmonia

This service serves its own /nix/store, so it can only provide paths already present locally. On that host, anything missing necessarily comes from another source: that is normal, its harmonia only benefits the other machines in the zone.

Roaming host (multi-zone portable)

A laptop that moves between zones should not keep substituters frozen to its declared zone: unreachable elsewhere, they cost timeouts on every request. Enable roaming mode:

# usr/machines/<hôte>/default.nix
darkone.service.nix-cache.roaming = true;

Its substituters become neutral names (harmonia.dnf.internal, nix-cache.dnf.internal) that each zone’s DNS resolves to its own caches → the cache follows the machine. Outside any zone, immediate NXDOMAIN response → direct fallback to cache.nixos.org, with no delay. Since signing keys are shared across the whole network, nothing else to configure.

Activation

harmonia and nix-cache are services to enable on a network host (nix-cache on the zone gateway). The harmonia signing key is managed by sops (just configure-admin-host).

Why a single-upstream proxy

nix-cache relays only one upstream (cache.nixos.org): it caches narinfo and nar as-is, signatures intact. No URL rewriting and no database, hence no HTTP 500 “invalid nar hash” error from the old ncps relay.

Build farm

Fetching a package and building it are two distinct needs. The binary cache above covers the first. For the second, a powerful network host can take over the long compilations of an administration host.

Declare it with the build-farm feature:

etc/config.yaml

    features: ["build-farm"]

Each administration host then adds it to its nix.buildMachines, connecting over SSH with the nix deployment account. Nothing else to configure: the key and the authorization are already in place across the whole fleet.

Only long compilations leave

The delegation is deliberately narrow: the farm is only offered derivations declaring requiredSystemFeatures = [ "big-parallel" ], i.e. the kernel, browsers, llvm, rustc… Everything else is built locally. This is decisive when the farm sits behind a slow link, since only the requested result makes the return trip.

Adjust with darkone.admin.nix.remoteBuilders: maxJobs, speedFactor, and mandatoryFeatures (empty list = everything goes to the farm).

A farm is not a priority cache

Do not promote a remote farm to a priority global harmonia to share its results: harmonia serves the whole /nix/store and would capture the traffic the public cache is meant to serve, across the slowest link on the network. That is the role of level 45 in the table above: last resort, never the main source.

Fallback if the farm is unreachable

The administration host keeps big-parallel in its own system-features. When the farm does not respond, Nix simply resumes the build locally instead of failing. Do not remove this feature to force delegation.