access: tailnet ACL, Headscale login and admin tools behind Authelia (tracking PRs #246-#250) #251

Open
opened 2026-10-16 19:06:49 -04:00 by jfraeys · 3 comments
Owner

Tracking issue for the tailnet and admin-tool access work of 2026-10-16/17. Prompted by: how do I add another user's PC, and which tools should be behind Authelia?

Findings

  1. Any LLDAP user could enroll a tailnet device (the Headscale OIDC client used one_factor), and eight ACL rules used src: "*", so any enrolled device reached SSH on both hosts, the LLDAP admin page, Alertmanager (no login), Forgejo git-SSH and web port, and the CI cache. The onboarding runbook claimed web:443 only.
  2. Alertmanager's UI had no login (published on the tailnet address). Prometheus, Vaultwarden /admin and Grafana were already behind Authelia (verified live); Traefik's dashboard is unreachable; Loki is disabled and bound to 127.0.0.1 (no UI, stays unrouted).
  3. manage health showed Prometheus as stopped while it ran (Compose container names vs an exact-name match).
  4. The ACL cannot be exercised by CI: staging nodes join the live Headscale, so headscale policy check (syntax only; it accepts a nonexistent user) is the only automated check.

Pull requests (merge and deploy in this order)

# What Deploy
#246 ACL: group:admin (headscale_admin_users) + tagged infrastructure for the admin ports; everyone else web:443 only; Alertmanager rule removed --tags headscale --limit services
#247 Authelia policy headscale: only admin and client-<site>-admin/-employee can sign in --tags authelia --limit services
#250 Alertmanager behind Traefik + Authelia (alertmanager.<domain>, admin only), no published port --tags traefik,authelia,alertmanager --limit services
#249 manage health: Compose names recognised; Prometheus and Alertmanager monitored --tags manage_cli
#248 manage user roles/access/grant/revoke --tags manage_cli

Verify after the deploys

  • From the operator's Mac over the tailnet: nc -z 100.64.0.3 17170, 3000, 2222, 8089 open. If not: the other form of the user name in headscale_admin_users (user name vs the OIDC email) is the fix; public SSH still works; roll back by redeploying the previous commit.
  • python3 scripts/checks/check-exposure.py: all login checks OK (Alertmanager fails until #250 is deployed, which is the gap).
  • Sign in to Forgejo through SSO after the Authelia deploy.
  • manage user access jfraeys: Tailnet reach should say admin devices (before #246 it correctly says every device reaches the admin ports).

Follow-ups (not done)

  • Runners share tag:web with the public web host, so a CI job's node can reach the admin ports through the tag rule. Give runners their own tag (tag:ci: add to tagOwners, mint the runner key with it, set the ephemeral play's advertised tag) with only services:3000, :2222, :8089; then narrow tag:web to services:25.
  • tag:operator for the specific admin machines (a lost phone is currently an admin device).
  • A post-deploy smoke script for tailnet reach (what headscale policy check cannot prove).
  • Remove 6 stale offline Headscale nodes (ids 21, 47, 48, 49, 63, 64): headscale nodes delete, not manage vpn revoke (that only expires). Do it before the OVH name swap.
  • Decide the Headscale OIDC allowed_* filters if Authelia policy is ever not enough.
  • alertmanager_webhook_url defaults to http://localhost:9093/webhook in the hardening scripts, which was never reachable (the port was bound to the tailnet address); either point them at the container or remove.
Tracking issue for the tailnet and admin-tool access work of 2026-10-16/17. Prompted by: how do I add another user's PC, and which tools should be behind Authelia? ## Findings 1. **Any LLDAP user could enroll a tailnet device** (the Headscale OIDC client used `one_factor`), and **eight ACL rules used `src: "*"`**, so any enrolled device reached SSH on both hosts, the LLDAP admin page, Alertmanager (no login), Forgejo git-SSH and web port, and the CI cache. The onboarding runbook claimed `web:443` only. 2. **Alertmanager's UI had no login** (published on the tailnet address). Prometheus, Vaultwarden `/admin` and Grafana were already behind Authelia (verified live); Traefik's dashboard is unreachable; Loki is disabled and bound to 127.0.0.1 (no UI, stays unrouted). 3. `manage health` showed Prometheus as stopped while it ran (Compose container names vs an exact-name match). 4. The ACL cannot be exercised by CI: staging nodes join the live Headscale, so `headscale policy check` (syntax only; it accepts a nonexistent user) is the only automated check. ## Pull requests (merge and deploy in this order) | # | What | Deploy | |---|---|---| | #246 | ACL: `group:admin` (`headscale_admin_users`) + tagged infrastructure for the admin ports; everyone else `web:443` only; Alertmanager rule removed | `--tags headscale --limit services` | | #247 | Authelia policy `headscale`: only `admin` and `client-<site>-admin/-employee` can sign in | `--tags authelia --limit services` | | #250 | Alertmanager behind Traefik + Authelia (`alertmanager.<domain>`, admin only), no published port | `--tags traefik,authelia,alertmanager --limit services` | | #249 | `manage health`: Compose names recognised; Prometheus and Alertmanager monitored | `--tags manage_cli` | | #248 | `manage user roles/access/grant/revoke` | `--tags manage_cli` | ## Verify after the deploys - From the operator's Mac over the tailnet: `nc -z 100.64.0.3 17170`, `3000`, `2222`, `8089` open. If not: the other form of the user name in `headscale_admin_users` (user name vs the OIDC email) is the fix; public SSH still works; roll back by redeploying the previous commit. - `python3 scripts/checks/check-exposure.py`: all login checks OK (Alertmanager fails until #250 is deployed, which is the gap). - Sign in to Forgejo through SSO after the Authelia deploy. - `manage user access jfraeys`: Tailnet reach should say admin devices (before #246 it correctly says every device reaches the admin ports). ## Follow-ups (not done) - **Runners share `tag:web`** with the public web host, so a CI job's node can reach the admin ports through the tag rule. Give runners their own tag (`tag:ci`: add to `tagOwners`, mint the runner key with it, set the ephemeral play's advertised tag) with only `services:3000`, `:2222`, `:8089`; then narrow `tag:web` to `services:25`. - `tag:operator` for the specific admin machines (a lost phone is currently an admin device). - A post-deploy smoke script for tailnet reach (what `headscale policy check` cannot prove). - Remove 6 stale offline Headscale nodes (ids 21, 47, 48, 49, 63, 64): `headscale nodes delete`, not `manage vpn revoke` (that only expires). Do it before the OVH name swap. - Decide the Headscale OIDC `allowed_*` filters if Authelia policy is ever not enough. - `alertmanager_webhook_url` defaults to `http://localhost:9093/webhook` in the hardening scripts, which was never reachable (the port was bound to the tailnet address); either point them at the container or remove.
Author
Owner

Status 2026-10-17: ALL FIVE PRs ARE MERGED, PROMOTED AND DEPLOYED, and the checks from this issue pass.
Done: #246 (ACL: admin group + tagged infrastructure, everyone else web:443; Alertmanager rule removed), #247 (Authelia policy 'headscale'), #250 (Alertmanager behind Traefik + Authelia), #249 (manage health: Compose names, Prometheus and Alertmanager monitored), #248 (manage user roles/access/grant/revoke). Promotion #252 approved; prod == main.
Verified on production: Headscale loaded policy contains group:admin and no 9093 rule; Authelia has the headscale policy; Traefik has the alertmanager router and the container publishes no port; manage user_access.sh and the new health.sh are installed; scripts/checks/check-exposure.py reports all three login checks OK (prometheus, alertmanager, vault /admin -> Authelia 401 + login Location) and 'exposure matches expectations'; from the operator's Mac over the tailnet ports 22, 17170, 3000, 2222, 8089 are open and 9093 is closed; 'manage user access jfraeys' says role admin, Headscale login yes, tailnet reach admin devices; 'manage user roles' lists admin and the coaching client-admin / client groups (no members yet).
Still open (follow-ups in this issue's list): CI runners share tag:web with the public web host (plan: tag:ci); tag:operator for specific admin machines; tailnet reach smoke script; delete 6 stale Headscale nodes (21, 47, 48, 49, 63, 64) before the OVH name swap; the unreachable localhost:9093 webhook default in the hardening scripts. Small fix pending: the 'Tailnet reach' line in manage user access still lists Alertmanager among the admin-device ports. Not yet confirmed which form of the admin user (name or OIDC email) Headscale matches; both are listed, drop the unused one later.

Status 2026-10-17: ALL FIVE PRs ARE MERGED, PROMOTED AND DEPLOYED, and the checks from this issue pass. Done: #246 (ACL: admin group + tagged infrastructure, everyone else web:443; Alertmanager rule removed), #247 (Authelia policy 'headscale'), #250 (Alertmanager behind Traefik + Authelia), #249 (manage health: Compose names, Prometheus and Alertmanager monitored), #248 (manage user roles/access/grant/revoke). Promotion #252 approved; prod == main. Verified on production: Headscale loaded policy contains group:admin and no 9093 rule; Authelia has the headscale policy; Traefik has the alertmanager router and the container publishes no port; manage user_access.sh and the new health.sh are installed; scripts/checks/check-exposure.py reports all three login checks OK (prometheus, alertmanager, vault /admin -> Authelia 401 + login Location) and 'exposure matches expectations'; from the operator's Mac over the tailnet ports 22, 17170, 3000, 2222, 8089 are open and 9093 is closed; 'manage user access jfraeys' says role admin, Headscale login yes, tailnet reach admin devices; 'manage user roles' lists admin and the coaching client-admin / client groups (no members yet). Still open (follow-ups in this issue's list): CI runners share tag:web with the public web host (plan: tag:ci); tag:operator for specific admin machines; tailnet reach smoke script; delete 6 stale Headscale nodes (21, 47, 48, 49, 63, 64) before the OVH name swap; the unreachable localhost:9093 webhook default in the hardening scripts. Small fix pending: the 'Tailnet reach' line in manage user access still lists Alertmanager among the admin-device ports. Not yet confirmed which form of the admin user (name or OIDC email) Headscale matches; both are listed, drop the unused one later.
Author
Owner

Update: Headscale stale nodes 21, 47, 48, 49, 63, 64 were REVOKED (expired 2026-10-17 01:33 UTC, so they cannot connect) but not DELETED; their entries and names remain, on the live Headscale and on the OVH copy. Deleting is still needed before the OVH name swap: headscale nodes delete --identifier (manage vpn revoke only expires). Also done: firewall tag on ovh-services verified (#233 closed).

Update: Headscale stale nodes 21, 47, 48, 49, 63, 64 were REVOKED (expired 2026-10-17 01:33 UTC, so they cannot connect) but not DELETED; their entries and names remain, on the live Headscale and on the OVH copy. Deleting is still needed before the OVH name swap: headscale nodes delete --identifier <id> (manage vpn revoke only expires). Also done: firewall tag on ovh-services verified (#233 closed).
Author
Owner

Status 2026-10-17: the follow-up 'runners share tag:web' is in PR #274 (tag:ci: Forgejo :3000/:2222 and the cache :8089 only; the CI authkey is minted with tag:ci). Checked that coaching-site's refresh_demo SSH target is a public address, so it is unaffected. Rollout is two deploys in order (headscale ACL first, then the provisioner key). Still open here: narrowing tag:web (needs web's real needs verified) and tag:operator.

Status 2026-10-17: the follow-up 'runners share tag:web' is in PR #274 (tag:ci: Forgejo :3000/:2222 and the cache :8089 only; the CI authkey is minted with tag:ci). Checked that coaching-site's refresh_demo SSH target is a public address, so it is unaffected. Rollout is two deploys in order (headscale ACL first, then the provisioner key). Still open here: narrowing tag:web (needs web's real needs verified) and tag:operator.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
jfraeys/infra#251
No description provided.