Alerts

The alerting system delivers Prometheus alerts to Matrix rooms (and via email for incidents). A one-time provisioning of the bot is enough: everything is declarative afterwards. The overall architecture is described in Monitoring & Alerting.

Prerequisites

Prerequisite	Why
prometheus + monitoring on the collector	carries Alertmanager and the rules
matrix service reachable	hosts the bot and channels
SSH access to the Matrix host (nix)	creates the bot account with dnf-mas
Admin sops key present	writes the bot secrets

Install the bot

A single command, idempotent (safe to run repeatedly): it creates the bot account, channels, and writes the identity to var/generated/matrix.nix.

1. Declare the administrators (local parties) in the configuration:

   etc/config.yaml

   network:
     matrix:
       admins: ["alice", "bob"]

2. Run the provisioning:

   just configure-alert-bot

   The command, leveraging the admin sops key:

   • creates the @alertbot:domain.tld account (dnf-mas manage register-user on the Matrix host);
   • creates or resolves the #alert-warnings and #alert-incidents channels, adds the bot to them and invites the admins;
   • writes bot and channel IDs to var/generated/matrix.nix;
   • stores the password, access token and webhook secret in sops.

   Admin API not public

   If /_synapse/admin is not exposed, the script automatically opens an SSH tunnel to the Matrix host (via the nix deployment user).

   Caddy anti-bot filter

   The Matrix vhost rejects (403) a User-Agent of type curl. The script therefore sends a neutral agent: do not reproduce the calls with raw curl.

3. Deploy the collector:

   just generate
   just apply <collector>

   Alerting activates automatically as soon as the channels exist (var/generated/matrix.nix is populated). To disable it anyway: darkone.service.prometheus.alerting.enable = false;.

Change the bot name

The bot is named alertbot by default. For a different name: ALERT_BOT=supervision just configure-alert-bot.

Created secrets

Provisioning stores three secrets in usr/secrets/secrets.yaml:

Sops key	Role
alertmanager-matrix-password	bot account password (reconnection)
alertmanager-matrix-token	bot access token (message sending)
alertmanager-webhook-secret	authenticates Alertmanager to the Matrix bridge

Sending a message to a room

Once the bot is in place, any tool in the project can write to the alert rooms. The message body, in markdown, arrives on standard input:

just send-msg warnings < rapport.md
echo "**Sauvegarde nocturne** : 2 dépôts en retard" | just send-msg incidents

The recipe reads the rooms from var/generated/matrix.nix and the token from sops, under the deployment identity nix. The caller therefore does not need to know either of them.

Markdown is converted to HTML before sending; otherwise Matrix clients would display it as-is: headings, lists, bold, code and links. No italics: a lone * is a host pattern (gw-*).

Exit code	Meaning
0	Message sent
10	Nothing configured here: missing room, token or program
11	The Matrix server refused the message

Who uses it

fleet-update --send-report sends its report through this recipe (see Déploiement). Exit code 10 is enough for it to know that retrying is pointless.

Testing an alert

1. On a supervised node, stop a monitored service:

   sudo systemctl stop <unit>

2. Within a minute, a ServiceDown (or SystemdUnitFailed) message appears in #alert-warnings, #alert-incidents and an email if the node is critical. The alert link points to https://prometheus.<zone>.domain.tld.

3. Restart the service: the alert resolves (resolution notification). For SystemdUnitFailed, resolution only comes after 30 minutes without failure (why).

   sudo systemctl start <unit>

Silencing during a rebuild

The deployment flow sets a maintenance flag to avoid triggering false alerts during a nixos-rebuild. Manually:

dnf-maintenance on    # inhibits node alerts
dnf-maintenance off   # re-enables them

Silencing a known alert

Some alerts are accurate but not actionable: a backup drive intentionally filled permanently triggers DiskSpaceLow. silences prevents them from notifying.

usr/modules/alerts.nix

{
  darkone.service.prometheus.alerting.silences = [
    {
      alert = "DiskSpaceLow";
      host = "srv-backup";
      matchers.device = "/dev/sdb";
      reason = ''
        Disque de sauvegarde volontairement rempli, seuil structurellement
        franchi. Retirer si la rétention est réduite.
      '';
    }
  ];
}

Field	Role
alert	Name of the alert to silence (alertname)
host	Target host name (host label), null for the entire fleet
matchers	Additional labels in strict equality, combined with AND
reason	Required: why the alert is accepted

Each entry becomes an Alertmanager route to the null receiver. The alert continues to fire and remains visible in Prometheus, Alertmanager, and Grafana; it simply stops notifying.

A silence is surgical, not global

Only the cited alertname is affected: silencing DiskSpaceLow on a partition leaves DiskSpaceCritical and DiskWillFillSoon fully armed on that same partition. Actual fullness still triggers an alert.

Target a partition, not a mount point

Disk rules alert once per partition, regardless of the number of mount points it carries (btrfs subvolumes /, /nix, /home, /.swapfile: a single alert). These alerts therefore have no mountpoint label: a matchers.mountpoint would never match. Use matchers.device, read from the alert description or with df -h.

Where to declare these silences

The rules and routes for a zone are generated on the host running Prometheus, not on the machine in question. Since each host is evaluated independently, usr/machines/<host>/ alone is not enough. A consumer module (usr/modules/alerts.nix) is imported everywhere and groups into a single file what is intentionally ignored, across all zones.

Verify routing on the Prometheus host:

amtool config routes test --config.file=/etc/alertmanager.yml \
  alertname=DiskSpaceLow host=srv-backup device=/dev/sdb

To silence a systemic failure mode rather than an isolated case, see ignoredUnits (dnf/config/alerts.nix), which excludes a systemd unit from SystemdUnitFailed across the entire fleet.

Troubleshooting

Symptom	Likely cause	Fix
No messages in a room	bot not in the room or expired token	re-run just configure-alert-bot
Room creation 403	Caddy anti-bot filter	already handled (neutral agent) ; do not use raw curl
admin API not public	/_synapse/admin closed	let the automatic SSH tunnel work (Matrix host must be reachable)
Unresolved alert link	missing DNS or certificate for the prometheus vhost	deploy the gateway / HCS for the new subdomain
matrix.nix ignored at deploy	file not readable by nix	the script does a chmod 644 ; check var/generated/ permissions
No incident email	local Postfix relay inactive	alerting enables it by default ; check darkone.service.prometheus.alerting.email