Prometheus alerts

Maintenance map of the alert code: what goes where, and how to extend it without breaking anything. The admin-facing operation is described in Monitoring & Alerts.

Pure logic vs wiring

The code is split in two to keep rule generation testable:

• dnf/lib/alerts.nix : pure functions that produce Prometheus rules from the topology. Tested in dnf/tests/unit/lib/alerts_test.nix.
• dnf/modules/service/prometheus.nix : impure wiring (Alertmanager, routing by severity, sops, Matrix bot, vhost, blackbox probes).

Any non-trivial logic goes into alerts.nix ; the module merely plugs it in.

Helpers (alerts.nix)

Exposed via dnfLib (see dnf/lib/default.nix):

Helper	Role
serviceUnits	DNF service → systemd unit (e.g. idm → kanidm.service)
nodeClass	Class of a node (critical / non-critical / disabled): alert-* features then profile
nodeAlertEligible	Selection: a node is monitored only if it is a network/server node, carries alert-non-critical, or hosts a mapped service
severityForClass	Class → severity (critical or warning)
hostExpectedUnits	Expected units of a host (based on its enabled services)
mkNodeRuleGroups	NodeDown, ServiceDown, SystemdUnitFailed (+ reach label local/wan)
mkResourceRuleGroups	Disk, RAM, load, inodes, OOM, read-only FS, disk prediction, clock, conntrack
mkNetworkRuleGroups	Blackbox probes (gateway, tailnet, DNS, ZoneInternetDown)
mkHttpRuleGroups	HTTP probes: ServiceEndpointDown + TLS certificate expiration
mkResticRuleGroups	Backup freshness (ResticBackupStale/Critical)
mkSmartctlRuleGroups	SMART disk health (DiskSmartFailing, DiskTemperatureHigh)
mkPostfixRuleGroups	Postfix relay (PostfixRelayUnhealthy, PostfixDeferredQueueHigh)
mkSynapseRuleGroups	Synapse (SynapseRestarting, SynapseHighErrorRate)
mkMaintenanceRuleGroups	Maintenance flag (silence during rebuild)
mkTailscaleRuleGroups	Tailnet self-healing (TailscaleUnhealthy, TailscaleFlapping)
mergeRuleGroups	Merges fragments into a single document
mkAlertRuleGroups	Shortcut: nodes + resources + restic + SMART + tailscale
mkSilenceRoutes	Alertmanager routes to the null receiver (known, accepted alerts)

The host label, or why alerts are named

Each scrape target carries a host label set to the hostname, assigned by mkHostTargets in prometheus.nix (one static_config per host rather than a single list of targets). Two effects :

• the Matrix bot displays DiskSpaceLow at srv-backup instead of the raw instance <ip>:<port> ;
• a silence route can target a single machine.

Setting it at scrape time propagates it for free to all alerts from these jobs, including the generic resource rules, which know nothing of the topology at evaluation time.

Aggregations erase labels

A by (...) clause removes every non-grouped label. ResticBackupStale (max by (instance, job, host)) and SynapseHighErrorRate (sum by (instance, host)) therefore quote host explicitly. Forgetting it in a new aggregated rule makes the notification fall back to the raw instance.

The bot does not display summary

The Matrix bot renders the title then the description annotation, and only falls back to summary if description is missing : all DNF rules define one, so summary never reaches it. Information that must appear in the notification has to go into description.

Monitored node selection

Monitoring is opt-out for infrastructure, opt-in for the rest. nodeAlertEligible retains a node only if it is of class critical (gateway/hcs/server profile or alert-critical feature), if it carries alert-non-critical explicitly, or if it hosts at least one service mapped in serviceUnits. A bare laptop/desktop therefore generates no node alerts.

Internet outage vs remote host down

A zone’s Prometheus connects to hosts in other zones (e.g. the public HCS) only over the WAN. When internet goes down, these targets become unreachable and would trigger a false “host down” alert. To prevent this:

• each node rule carries a reach label (local same zone, wan cross-zone);
• an ICMP blackbox probe to external IPs feeds ZoneInternetDown (fired only if all external targets fail);
• an Alertmanager inhibition rule (equal: [zone]) silences reach="wan" alerts of the zone while ZoneInternetDown is active — the real cause is notified, not the symptoms.

The trap : a single rule document

Never forget

NixOS concatenates all entries of services.prometheus.rules (a list of strings) into a single file. Multiple groups: documents without --- separator → only the first one is read. Stacking a second entry silently breaks following groups.

So we merge everything via mergeRuleGroups, then emit only one entry :

dnf/modules/service/prometheus.nix

services.prometheus.rules = [
  (builtins.toJSON (dnfLib.mergeRuleGroups (
    [ (dnfLib.mkAlertRuleGroups { inherit nodes; /* … */ }) ]
    ++ lib.optional alerting.silenceOnRebuild (dnfLib.mkMaintenanceRuleGroups { /* … */ })
    ++ lib.optional alerting.network.enable (dnfLib.mkNetworkRuleGroups { /* … */ })
  )))
];

Where to add what

I want to…	I touch…
monitor a new service	serviceUnits in alerts.nix
add a new rule family	a mkXRuleGroups + add it to the module’s mergeRuleGroups
wire up a metrics exporter	exporter in its module (gated monitoring-node, bind preferredIp) + scrape job + mkXRuleGroups gated in prometheus.nix
change a default threshold	defaultThresholds in alerts.nix
change a node’s class	alert-*[:zone] feature (no code)
force/exclude a node from monitoring	alert-non-critical / alert-disabled feature
monitor a URL + its TLS certificate	alerting.network.httpProbeUrls in the config
tune internet outage detection	alerting.network.internetProbeTargets
silence a known, accepted alert	alerting.silences in a consumer module (no code)
silence a broken systemd unit across the whole fleet	ignoredUnits in dnf/config/alerts.nix
add a new destination	the alertmanager block of prometheus.nix (receiver + route)

DNF convention

A lib/ helper = a test. Any function added to alerts.nix comes with a case in dnf/tests/unit/lib/alerts_test.nix (nix-unit).

Alertmanager route order

continue is false by default : the first matching child route wins. The routes of mkSilenceRoutes must therefore be inserted before the routes by severity, otherwise they are never reached and the alert notifies anyway. The null receiver is shared with the maintenance route, hence its guard silenceOnRebuild || silences != [] : referencing a missing receiver prevents Alertmanager from starting.

Textfile metric : permissions

A .prom file from the textfile collector must be readable by node-exporter (non-root user). mktemp creates with 0600 : set a chmod 0644 before the atomic mv, otherwise the file is ignored (node_textfile_scrape_error 1) and the metric is never exposed, so the alert stays blind.