Monitoring

Monitoring relies on Prometheus 🡕 (metrics), Grafana 🡕 (dashboards) and Loki 🡕 (logs). It is organized by zone: one host collects, the others expose their metrics.

Roles

Role	Who	What
Collector	Host with prometheus (+ monitoring, + loki)	Metrics & alerts, dashboards, logs
Monitored node	Hosts tagged monitoring-node	node-exporter (system metrics)
Log source	Hosts behind Caddy	Alloy pushes access logs

prometheus (metrics + alerts) and monitoring (Grafana) are two separate services on the same host; Grafana requires Prometheus. Details are in Monitoring & Alerts.

One collector per zone

Only one collector is allowed per zone. It monitors nodes whose monitoring-node tag points to its zone.

Enable

Monitoring is a service. On the collector:

etc/config.yaml

    services:
      prometheus:              # metrics + alerts (required by monitoring)
      monitoring:
        domain: "stats"        # → stats.<zone>.domain.tld

• Nodes to monitor receive the monitoring-node tag from their zone.
• loki follows monitoring: Caddy logs are collected automatically as soon as Caddy and monitoring are active.

Access

Grafana and Prometheus are protected by SSO Kanidm. The Grafana domain root redirects to a home dashboard.

Enable alerts

Matrix/email notifications require a one-time bot provisioning: see Alerts.

Useful settings

Option	Effect	Default
prometheus.retentionTime	Prometheus metrics retention	30d
monitoring.isNode	Makes the host a monitored node	follows the tag
monitoring.kioskTarget	Dashboard displayed at the root	node-exporter

Reference

Detailed options: monitoring module and the loki module for log collection.