Alerts

The alerting system delivers Prometheus alerts to Matrix rooms (and via email for incidents). A one-time provisioning of the bot is enough: everything is declarative afterwards. The overall architecture is described in Monitoring & Alerting.

Prerequisites

Prerequisite	Why
prometheus + monitoring on the collector	hosts Alertmanager and the rules
Reachable matrix service	hosts the bot and the rooms
Registration secret in sops (matrix-rss-password)	creates the bot account
Admin sops key present	writes the bot secrets

Installing the bot

A single idempotent command (safe to re-run): it creates the bot account, the rooms, and writes the identity into var/generated/matrix.nix.

1. Declare the human administrator (local part) in the configuration:

   etc/config.yaml

   network:
     matrix:
       admin: "alice"

2. Run provisioning:

   just configure-alert-bot

   The command, using the admin sops key:

   • creates the @alertbot:domain.tld account (nonce HMAC flow on the shared secret) ;
   • creates or resolves the #alert-warnings and #alert-incidents rooms, invites the bot and invites the admin ;
   • writes bot + room IDs into var/generated/matrix.nix ;
   • stores password, access token and webhook secret in sops.

   Non-public admin API

   If /_synapse/admin is not exposed, the script automatically opens an SSH tunnel to the Matrix host (using the nix deployment user).

   Caddy anti-bot filter

   The Matrix vhost rejects (403) a curl-like User-Agent. The script sends a neutral agent instead: do not reproduce API calls with raw curl.

3. Deploy the collector:

   just generate
   just apply <collector>

   Alerting activates automatically as soon as the rooms exist (var/generated/matrix.nix populated). To force-disable it: darkone.service.prometheus.alerting.enable = false;.

Changing the bot name

The bot is named alertbot by default. For a different name: ALERT_BOT=supervision just configure-alert-bot.

Created secrets

Provisioning stores three secrets in usr/secrets/secrets.yaml:

Sops key	Role
alertmanager-matrix-password	bot account password (reconnection)
alertmanager-matrix-token	bot access token (message sending)
alertmanager-webhook-secret	authenticates Alertmanager to the Matrix bridge

Testing an alert

1. On a supervised node, stop a monitored service:

   sudo systemctl stop <unité>

2. Within a minute, a ServiceDown (or SystemdUnitFailed) message appears in #alert-warnings, #alert-incidents and an email if the node is critical. The alert link points to https://prometheus.<zone>.domain.tld.

3. Restart the service: the alert resolves (resolution notification).

   sudo systemctl start <unité>

Silencing during a rebuild

The deployment flow sets a maintenance flag to avoid triggering false alerts during a nixos-rebuild. Manually:

dnf-maintenance on    # inhibits node alerts
dnf-maintenance off   # re-enables them

Silence a known alert

Some alerts are accurate but not actionable: a backup disk that is deliberately filled permanently crosses DiskSpaceLow. silences prevents them from notifying.

usr/modules/alerts.nix

{
  darkone.service.prometheus.alerting.silences = [
    {
      alert = "DiskSpaceLow";
      host = "srv-backup";
      matchers.mountpoint = "/mnt/backup";
      reason = ''
        Backup disk deliberately filled, threshold structurally crossed. Remove
        if retention is reduced.
      '';
    }
  ];
}

Field	Role
alert	Name of the alert to silence (alertname)
host	Host name to target (host label), null for the whole fleet
matchers	Additional labels in strict equality, combined with AND
reason	Required: why the alert is accepted

Each entry becomes an Alertmanager route to the null receiver. The alert still fires and stays visible in Prometheus, Alertmanager and Grafana, it simply stops notifying.

A silence is surgical, not global

Only the cited alertname is affected: silencing DiskSpaceLow on one mount point leaves DiskSpaceCritical and DiskWillFillSoon fully armed on that same mount point. A real fill-up always alerts.

Where to declare these silences

The rules and routes of a zone are generated on the host running Prometheus, not on the machine concerned. Since each host is evaluated independently, usr/machines/<host>/ is not enough. A consumer module (usr/modules/alerts.nix) is imported everywhere and groups in a single file what is deliberately ignored, across all zones.

Check the routing on the Prometheus host:

amtool config routes test --config.file=/etc/alertmanager.yml \
  alertname=DiskSpaceLow host=srv-backup mountpoint=/mnt/backup

To silence a systemic failure mode rather than a one-off case, see ignoredUnits (dnf/config/alerts.nix), which excludes a systemd unit from SystemdUnitFailed across the whole fleet.

Troubleshooting

Symptom	Likely cause	Fix
No messages in a room	bot not in the room or expired token	re-run just configure-alert-bot
Room creation 403	Caddy anti-bot filter	already handled (neutral agent) ; do not use raw curl
admin API not public	/_synapse/admin closed	let the automatic SSH tunnel work (Matrix host must be reachable)
Unresolved alert link	missing DNS or certificate for the prometheus vhost	deploy the gateway / HCS for the new subdomain
matrix.nix ignored at deploy	file not readable by nix	the script does a chmod 644 ; check var/generated/ permissions
No incident email	local Postfix relay inactive	alerting enables it by default ; check darkone.service.prometheus.alerting.email