Prometheus alerts

Maintenance map of the alert code: what goes where, and how to extend it without breaking anything. The admin-facing operation is described in Monitoring & Alerts.

Pure logic vs wiring

The code is split in two to keep rule generation testable:

• dnf/lib/alerts.nix : pure functions that produce Prometheus rules from the topology. Tested in dnf/tests/unit/lib/alerts_test.nix.
• dnf/modules/service/prometheus.nix : impure wiring (Alertmanager, routing by severity, sops, Matrix bot, vhost, blackbox probes).

Any non-trivial logic goes into alerts.nix ; the module merely plugs it in.

The helpers (alerts.nix)

Exposed via dnfLib (see dnf/lib/default.nix) :

Helper	Role
serviceUnits	DNF service → systemd unit (e.g. idm → kanidm.service)
nodeClass	Node class (critical / non-critical / disabled): alert-* features then profile
nodeAlertEligible	Selection: a node is only monitored if it is a network/server node, carries alert-non-critical, or hosts a mapped service
severityForClass	Class → severity (critical or warning)
hostExpectedUnits	A host’s expected units (from its enabled services)
mkNodeRuleGroups	NodeDown, ServiceDown, SystemdUnitFailed (+ local/WAN reach label)
mkResourceRuleGroups	Disk, RAM, load, inodes, OOM, read-only FS, disk prediction, clock, conntrack
mkNetworkRuleGroups	Blackbox probes (gateway, tailnet, DNS, ZoneInternetDown)
mkHttpRuleGroups	HTTP probes: ServiceEndpointDown + TLS certificate expiry
mkResticRuleGroups	Backup freshness (ResticBackupStale/Critical)
mkSmartctlRuleGroups	SMART disk health (DiskSmartFailing, DiskTemperatureHigh)
mkPostfixRuleGroups	Postfix relay (PostfixRelayUnhealthy, PostfixDeferredQueueHigh)
mkSynapseRuleGroups	Synapse (SynapseRestarting, SynapseHighErrorRate)
mkMaintenanceRuleGroups	Maintenance flag (silence during rebuild)
mkTailscaleRuleGroups	Tailnet self-repair (TailscaleUnhealthy, TailscaleFlapping)
mkHeadscaleRuleGroups	Tailnet audit on the HCS (HeadscaleNodeDrift, HeadscaleUnexpectedNode, HeadscalePolicyInvalid, HeadscaleOidcFallback, HeadscaleNodeExpiring, HeadscaleAuditStale)
mergeRuleGroups	Merges fragments into a single document
mkAlertRuleGroups	Shortcut: nodes + resources + restic + SMART + tailscale + headscale
mkSilenceRoutes	Alertmanager routes to the null receiver (known, accepted alerts)

The host label, or why alerts are named

Every scrape target carries a host label set to the hostname, set by mkHostTargets in prometheus.nix (one static_config per host rather than a single target list). Two effects:

• the Matrix bot shows DiskSpaceLow at srv-backup instead of the raw instance <ip>:<port>;
• a silence route can target a single machine.

Setting it at scrape time propagates it for free to all alerts from these jobs, including the generic resource rules, which know nothing of the topology at evaluation time.

Blackbox probes have no host (the target is an IP). PeerGatewayDown therefore sets it in the rule, with the name of the targeted gateway. Without it, a zone’s rules would differ only by their expr → promtool flags them as duplicates.

Aggregations erase labels

A by (...) clause drops every label not grouped. ResticBackupStale (max by (instance, backup, host)), SynthesisHighErrorRate — actually SynapseHighErrorRate (sum by (instance, host)) and the FS rules (byDevice) therefore cite host explicitly. Forgetting it in a new aggregated rule makes the notification fall back to the raw instance.

An exported job label is renamed at scrape time

The scrape job sets its own job label. An exported metric’s label (textfile collector included) therefore becomes exported_job. The restic metric carries the job name in backup : grouping on job would merge all of a host’s jobs, and a failing job would stay hidden as long as another one succeeds.

A job that never succeeded has no series

dnf_restic_last_success_timestamp only exists after a first success. The restic module therefore also exports dnf_restic_declared_timestamp, the first declaration date of each job. The rules take the success, or the declaration as a fallback : two aggregates by (instance, backup, host), the left side wins. Without this list, a node keeps the old behaviour.

FS rules: one alert per partition

node_filesystem_* is emitted per mount. A partition mounted several times (btrfs subvolumes, bind mounts) therefore triggered each FS rule as many times as it has mounts, for a single full disk.

mkResourceRuleGroups aggregates on byDevice (instance, job, host, device, fstype) : min for space and inodes, max for FilesystemReadOnly. Both members of DiskWillFillSoon’s and are aggregated — a raw member still carries mountpoint and matches nothing on the other side.

mountpoint disappears with the aggregation. PromQL cannot concatenate label values: the list of mounts is therefore resolved by the query function of the Go template engine that Prometheus applies to annotations (deviceMounts). Cost : one instant query per active alert and per evaluation, none while the rule stays silent.

FilesystemReadOnly: a dedicated metric

node_filesystem_readonly merges the mount’s ro (intended) with the superblock’s one (failure : btrfs forced read-only, ext4 errors=remount-ro). The rule therefore reads dnf_filesystem_forced_readonly :

• Source: service/monitoring/fs-readonly.nix, on every node, every minute.
• Content: rw mounts on a ro superblock, read from /proc/1/mountinfo.
• Location: textfile on tmpfs (textfileRuntimeDir), rewritten even when the disk has failed.
• Test: tests/scenarios/scripts/fs-forced-readonly.nix pins the parser.

The bot does not display summary

The Matrix bot renders the title then the description annotation, and falls back to summary only if description is missing : all DNF rules define one, so summary never reaches it. Information to show in the notification must go into description.

Monitored node selection

Monitoring is opt-out for infrastructure, opt-in for the rest. nodeAlertEligible retains a node only if it is of class critical (gateway/hcs/server profile or alert-critical feature), if it carries alert-non-critical explicitly, or if it hosts at least one service mapped in serviceUnits. A bare laptop/desktop therefore generates no node alerts.

Internet outage vs remote host down

A zone’s Prometheus connects to hosts in other zones (e.g. the public HCS) only over the WAN. When internet goes down, these targets become unreachable and would trigger a false “host down” alert. To prevent this:

• each node rule carries a reach label (local same zone, wan cross-zone);
• an ICMP blackbox probe to external IPs feeds ZoneInternetDown (fired only if all external targets fail);
• an Alertmanager inhibition rule (equal: [zone]) silences reach="wan" alerts of the zone while ZoneInternetDown is active — the real cause is notified, not the symptoms.

The trap : a single rule document

Never forget

NixOS concatenates all entries of services.prometheus.rules (a list of strings) into a single file. Multiple groups: documents without --- separator → only the first one is read. Stacking a second entry silently breaks following groups.

So we merge everything via mergeRuleGroups, then emit only one entry :

dnf/modules/service/prometheus.nix

services.prometheus.rules = [
  (builtins.toJSON (dnfLib.mergeRuleGroups (
    [ (dnfLib.mkAlertRuleGroups { inherit nodes; /* … */ }) ]
    ++ lib.optional alerting.silenceOnRebuild (dnfLib.mkMaintenanceRuleGroups { /* … */ })
    ++ lib.optional alerting.network.enable (dnfLib.mkNetworkRuleGroups { /* … */ })
  )))
];

Where to add what

I want to…	I touch…
monitor a new service	serviceUnits in alerts.nix
a new rule family	a mkXRuleGroups + add it to the module’s mergeRuleGroups
wire up a metrics exporter	exporter in its module (gated monitoring-node, bind preferredIp) + scrape job + gated mkXRuleGroups in prometheus.nix
change a default threshold	defaultThresholds in alerts.nix
stop alerting on an ro FS mounted on purpose	readOnlyByDesignFstypes (thresholds) in alerts.nix
monitor (or not) the space of a network mount	remoteFstypes (thresholds) in alerts.nix
change a node’s class	feature alert-*[:zone] (no code)
force/exclude a node from monitoring	feature alert-non-critical / alert-disabled
monitor a URL + its TLS certificate	alerting.network.httpProbeUrls in the config
tune internet outage detection	alerting.network.internetProbeTargets
silence a known and accepted alert	alerting.silences in a consumer module (no code)
silence a broken systemd unit across the whole fleet	ignoredUnits in dnf/config/alerts.nix
a new destination	the alertmanager block of prometheus.nix (receiver + route)

Exclude rather than silence

A permanent, intended condition must not be silenced : alerting.silences cuts the notification but lets the alert fire, so it keeps the UIs red, forever. It must exit the rule’s selector. Two threshold lists do this, in mkResourceRuleGroups :

• remoteFstypes — removed from all FS rules. A client mount reflects the server’s space : monitoring it turns a full export into the same alert on every machine that mounts it, while the server already alerts on its own, with the real partition.
• readOnlyByDesignFstypes — removed from FilesystemReadOnly only (optical media or images : their driver may set the superblock ro under a rw mount, same signature as a failure).

DNF convention

A helper in lib/ = a test. Any function added to alerts.nix is paired with a case in dnf/tests/unit/lib/alerts_test.nix (nix-unit).

Alertmanager route order

continue defaults to false: the first matching child route wins. Routes from mkSilenceRoutes must therefore be inserted before the severity routes, otherwise they are never reached and the alert notifies anyway. The null receiver is shared with the maintenance route,ence hence its guard silenceOnRebuild || silences != [] : referencing a missing receiver prevents Alertmanager from starting.

textfile metric: permissions

A .prom file from the textfile collector must be readable by node-exporter (a non-root user). mktemp creates with 0600: set a chmod 0644 before the atomic mv, otherwise the file is ignored (node_textfile_scrape_error 1) and the metric is never exposed, so the alert stays blind.