VPN (Headscale / Tailscale)

The VPN meshes the zones, the coordination server and remote devices : Headscale 🡕 coordinates from the HCS, Tailscale 🡕 is the client on each node. This page gathers the operational workflows. Routing, DNS, MagicDNS, access policy and keys are explained in Tailnet operation.

Enable the VPN

Enabling coordination and declaring the HCS (profile: hcs) is enough :

etc/config.yaml

network:
  coordination:
    enable: true
    hostname: "hcs"
    domain: "headscale"

Each node’s role is derived from the topology :

Node	Identity	Role
HCS	tag:hcs	coordination, DNS pivot, global services, exit node (closed)
Gateway	tag:gw-<zone>	subnet router: publishes its zone’s /16
Admin workstation	tag:admin	SSH to all machines
Personal device	its owner	services, per its groups

An admin workstation is a host whose groups contains admin and whose host file enables the client :

usr/machines/portable/configuration.nix

darkone.service.tailscale.enable = true;
darkone.service.tailscale.autoPauseOnLan.enable = true;

autoPauseOnLan pauses the client when the laptop is plugged into a zone (NetworkManager required).

Enrolling a machine

Machines join the tailnet without a shared key : the HCS creates a single-use key for each of them, with their declared tags. just configure handles this, and so does just full-install. The recipe alone can be re-run at any time :

just tailnet-enroll gw

[ DNF ] TAILNET • gw not enrolled, backend NeedsLogin: sending a single-use key...
[ DNF ] TAILNET • gw: gw tag:gw-maison 100.64.0.8

It is idempotent :

Situation	Effect
Node enrolled, name and tags conform	nothing (already enrolled)
Different tags or name	corrected on the HCS, without reconnection
Never enrolled, deleted or reinstalled	new key, registration
Older offline node with the same name	deleted, the new one takes the name
Older online node with the same name	refused : two machines carry this name
Host without Tailscale client	ignored

The mechanism is detailed in Activation keys and enrollment.

Deploying the HCS after declaring the host

The HCS only creates a key for a host it knows. A host just added to etc/config.yaml first requires just apply hcs, otherwise : no declared tailnet identity.

Failure during an installation

HCS unreachable, just configure warns without stopping : re-run just tailnet-enroll <host> once the HCS is back.

New address after a reinstallation

Headscale assigns addresses in order and does not allow fixing them. A reinstalled machine receives a new tailnet address : for a gateway, report it in vpn.ipv4. To keep it, restore the client state rather than re-enrolling (see Migrating a gateway’s hardware).

Registering a gateway

A gateway enrolls like any machine ; two points are specific to it.

1. Check its zone’s route

   The gateway advertises its zone’s /16, which the policy approves by default for tag:gw-<zone>. On the HCS :

   sudo headscale nodes list-routes

2. Declare its tailnet address

   The HCS reaches the zone’s resolver through this address (DNS pivot), like the services that target it (backups, monitoring). Read it in the output of just tailnet-enroll, then report it :

   etc/config.yaml

   zones:
     maison:
       gateway:
         vpn:
           ipv4: "100.64.0.8"

3. Propagate the address

   just generate
   just apply hcs

   Also redeploy the hosts that target this address. HeadscaleNodeDrift reports a declared address that no longer matches.

Connecting a personal device

A smartphone or a personal computer connects with the Kanidm identity of its owner : neither key, nor manual validation.

Can connect	Condition in etc/config.yaml
Administrator	admin or nix-admin profile
Member of a zone	zone-<zone> group

These accounts form the Kanidm group tailnet, provisioned on its own ; Kanidm refuses the others. To open access, add the group to the user, regenerate, then just apply hcs. The user then follows Connect a device.

The device belongs to the person (alice@) and expires after 180 days : they then reconnect with their identifier.

sudo headscale nodes list                 # devices, owners, expiration
sudo headscale nodes expire -i <id>       # force a reconnection
sudo headscale nodes delete -i <id>       # remove the device

Connection page not found

Split DNS sends every domain.tld to the HCS through the tunnel, including headscale.domain.tld and idm.domain.tld. As long as the tunnel is not up, the browser cannot resolve them. Workaround : disable “Use Tailscale DNS” in the app while connecting, then enable it again.

Adjust the access policy

The policy is generated from the topology : its rules are detailed in Access policy. Options adjust it, on the HCS :

Option darkone.service.headscale.	Default	Effect
policy.adminDevices	{ }	personal devices allowed in SSH : name → tailnet IPv4
policy.exitNodeSources	[ ]	sources allowed to reach the Internet via the HCS
policy.extraHosts	{ }	additional aliases : name → CIDR
policy.extraAcls	[ ]	rules appended after the generated rules
policy.enforce	true	false : everything allowed, migration only
nodeExpiry	"180d"	expiration of personal devices (0 : never)

usr/machines/hcs/configuration.nix

darkone.service.headscale.policy = {

  # SSH depuis le téléphone d'Alice (adresse lue dans `headscale nodes list`)
  adminDevices.phone-alice = "100.64.0.9";

  # Exit node du HCS ouvert aux administrateurs
  exitNodeSources = [ "group:admins" ];

  # Imprimante réseau pour tous les appareils personnels
  extraHosts.printer = "10.0.4.10/32";
  extraAcls = [
    {
      action = "accept";
      src = [ "autogroup:member" ];
      dst = [ "printer:631" ];
    }
  ];
};

Then just apply hcs : the policy is checked at build time, and Headscale reloads it without restarting.

Declared device address

adminDevices targets an IPv4 address. A device re-registered as a new node changes address : HeadscaleNodeDrift reports it, report the new one.

Exit node

Opening exitNodeSources is not enough : the user picks the HCS as exit node in their application.

Administration

On the HCS (just enter hcs) :

Command	Usage
sudo headscale nodes list	nodes, owner, tags, addresses, expiration
sudo headscale nodes list-routes	advertised and approved routes
sudo headscale nodes rename -i <id> <name>	rename a node
sudo headscale nodes expire -i <id>	force a reconnection
sudo headscale nodes delete -i <id>	delete a node
sudo headscale users list	owners of personal devices
sudo headscale preauthkeys list	pre-authorized keys : tags, usage, expiration
sudo headscale policy get	served policy
sudo dnf-tailnet-enroll status <host>	declared identity of a host

Tags and routes are not set by hand : just tailnet-enroll sets the declared tags, the policy approves the routes. The other client settings (advertised routes, Tailscale SSH off, exit node) are realigned by tailscaled-autoconnect, at startup and at deployments that modify it.

JSON outputs

nodes list -o json and preauthkeys list -o json may contain pre-authorized keys : do not copy them into a ticket or a chat room.

Introspecting and troubleshooting

On a node

Command	Answers
tailscale status	peers, online or not, direct path or relay
tailscale ping <node>	actual path : via <ip>:41641 (direct) or via DERP
tailscale whois <ip>	node, tags or owner behind an address
tailscale dns status	MagicDNS, split DNS, search domains received
journalctl -u tailscaled-autoconnect	last registration or realignment of the settings

Testing a policy

Without a --bypass… option, headscale policy check compiles a file against the real nodes and runs its tests block, without touching the policy in place.

1. Start from the served policy :

   sudo headscale policy get > /tmp/policy.json

2. Add a tests block : a source, then the expected destinations, each a single host (address and port) :

   "tests": [
     { "src": "alice@", "accept": ["100.64.0.8:443"], "deny": ["100.64.0.8:22"] }
   ]

3. Check :

   sudo headscale policy check -f /tmp/policy.json

Common failures

Symptom	Check	Remedy
TailscaleUnhealthy, backend NeedsLogin	tailscale status on the node	just tailnet-enroll <host>
Zone unreachable from the tailnet	headscale nodes list-routes	gateway enrolled with its tag, route approved
Zone service refused to a device	user’s groups, tailscale whois <ip>	add zone-<zone>, just apply hcs
SSH refused on a tailnet address	tag:admin machine or adminDevices device	declare the device
Internal names unresolved outside the zone	tailscale dns status	enable “Use Tailscale DNS”
Slow traffic	tailscale ping shows via DERP	outbound UDP filtered, or strict NAT on both sides
no declared tailnet identity	host tagged, HCS deployed since its declaration	just apply hcs
named … is online, not deleting it	two machines carry the same name	shut down the older one or rename the host

Audit and alerts

Every 15 minutes, headscale-audit compares the real nodes against the declared topology, without modifying anything. The details of the discrepancies are in its journal :

journalctl -u headscale-audit -n 20

Alert	Reports	After
HeadscaleNodeDrift	HCS or gateway missing, wrong tags, address different from the declared one	30 min
HeadscaleUnexpectedNode	tagged node not declared, or device from an account outside the tailnet group	immediately
HeadscalePolicyInvalid	policy refused, or served policy different from the file	30 min
HeadscaleOidcFallback	Kanidm missing at Headscale startup : OIDC connections cut	immediately
HeadscaleNodeExpiring	personal device expiring within 7 days	1 h
HeadscaleAuditStale	audit without success for over an hour	immediately

All are in warning, on the Prometheus that supervises the HCS. A missing admin machine is not an anomaly ; if present, its tags are checked.

After HeadscaleOidcFallback

Kanidm is back, restart Headscale : sudo systemctl restart headscale.

Client self-repair

A tailscaled can silently detach from Headscale : the control connection is lost while the service remains “active”, and a gateway’s subnet becomes unreachable. A local watchdog detects and fixes this case, on each Tailscale client.

Aspect	Behavior
Probe	every 60 s : tailscale status (backend, online status)
Debounce	3 consecutive failed probes (~3 min) before acting
Fix	restarts tailscaled, which reconnects to the control plane
Anti-loop	a single fix per 10 min window
Unenrolled node	no restart (NeedsLogin) : just tailnet-enroll
Portable on pause	watchdog stopped, metric removed
Observability	dnf_tailscale_healthy, dnf_tailscale_selfheal_restarts_total (textfile node_exporter)

Two alerts accompany the mechanism :

• TailscaleUnhealthy : still detached after 10 min, despite the fix ;
• TailscaleFlapping : more than 3 fixes in one hour.

Enabled by default

To disable it on a host : darkone.service.tailscale.selfHeal.enable = false;.

Control plane bootstrap

On a gateway, dnsmasq forwards any domain.tld to the HCS resolver through the tailnet : the client would need the VPN to find the server that brings up the VPN. At boot, moreover, the local resolver is not listening yet.

Each Tailscale client therefore receives the HCS public address in its /etc/hosts, derived from etc/config.yaml (the HCS itself is excluded). dnsmasq answers this name from that file, never forwarding it :

grep headscale /etc/hosts     # 203.0.113.10 headscale.domain.tld

The client thus connects without any resolver, including when the local DNS is down, precisely when the self-repair must be able to act.

HCS address change

/etc/hosts takes precedence over DNS. If the VPS address changes, update etc/config.yaml then redeploy the Tailscale clients : as long as they carry the old entry, they do not reconnect, even with a correct public DNS.

Outside a zone

• Personal device : see Connect a personal device.
• Admin laptop : on the move, it joins all zones through the tailnet (just enter <hôte>). Plugged back into a zone, its client pauses and the local gateway takes over again.