infra

Overview

This repo manages two hosts deployed via modular Terraform configuration:

• web (jfraeys.com) - web server
• services (services.jfraeys.com) - services host

Terraform structure:

terraform/
├── main.tf              # Root orchestrator with moved blocks
├── modules/
│   ├── compute/         # Compute instances (Linode/OVH/etc)
│   └── dns/             # DNS records (Cloudflare)

The routing convention is service.server.jfraeys.com.

Examples:

• git.jfraeys.com -> services host (Forgejo)
• auth.jfraeys.com -> services host (Authelia)
• demo.jfraeys.com -> services host (shared demo/sandbox gateway, path-routed by app slug — see docs/runbooks/app-deployment-contract.md item 6)
• app.jfraeys.com -> services host, reserved for future permanent/prod apps — nothing uses this yet

Traefik runs on both servers and routes only the services running on that server.

Workflow

CI validates; you deploy. Nothing deploys production by itself, and merging the promotion PR deploys nothing.

PR ─▶ lint + secret scan ─▶ merge to main ─▶ staging-validate (CI) ─▶ promotion PR (main ─▶ prod)
                                             fresh nodes, full deploy,        │ review `scripts/release/plan-prod.sh`,
                                             checks, destroyed (~15 min)      │ merge it = the approval record
                                                                              ▼
                              check out the approved commit, run `scripts/release/promote-prod.sh`

1. A PR gets lint and secret scan.
2. Merging to main with changes under roles/, playbooks/, inventory/, terraform/, scripts/lib/ and similar starts staging-validate: two fresh nodes, full deploy, hardening, connectivity and port checks, then destroy. When it passes, the promote job opens or updates the promotion PR (main to prod) and records the commit it validated.
3. Review what production would change: scripts/release/plan-prod.sh (read-only).
4. Approve: ./infra approve merges the promotion PR after re-checking it (you type the PR number). This is the approval record.
5. Deploy exactly that commit: git checkout <commit named in the PR>, then ./infra deploy (= scripts/release/promote-prod.sh: verifies the approval, shows the Terraform plan, asks, then runs ./setup --workspace linode --confirm-prod). For an Ansible-only change use ./infra deploy --tags <tags> --limit services; it has the same gate.
6. After a firewall or Traefik deploy, check what production exposes: scripts/checks/check-exposure.py.

The gate (scripts/lib/setup/promotion.sh) refuses unless the latest merged promotion PR names the commit staging validated, merged exactly that commit, and your checkout is that commit.

Emergency bypass (not the approved commit): run ./setup --workspace linode --confirm-prod (or deploy-services.sh) and type deploy when asked. Every bypass is appended to ~/.local/state/infra/emergency-deploys.log.

To run staging yourself instead of waiting for CI: ./setup --workspace linode-staging (creates temporary VMs).

Quickstart

This repo is intended to be driven by setup:

./setup

For options:

./setup --help

Staging Validation (Required Before Production)

# Test changes in staging (creates temporary VMs)
./setup --workspace linode-staging

# Deploy to production (gated: see Workflow)
./setup --workspace linode --confirm-prod

What it does:

• Applies Terraform from terraform/
• Writes inventory/hosts.yml and inventory/host_vars/web.yml (gitignored)
• Runs playbooks/services.yml and playbooks/web.yml

If you want Terraform only:

./setup --no-ansible

If you want Ansible only (requires an existing inventory/hosts.yml):

./setup --ansible-only

First-time init

New infra (no vault yet):

brew install sops age jq argon2        # + bitwarden-cli for the recovery key
scripts/secrets/age-operator-keygen.sh          # operator key -> Keychain (sops-age/infra); prints its public key
scripts/secrets/age-recovery-keygen.sh          # recovery key -> personal Bitwarden; prints its public key
# Another repo's keys (forge-agent, app-skeleton): pass its name to both keygen scripts (default infra)
$EDITOR .sops.yaml                      # set both public keys as the age recipients
scripts/secrets/init-secrets.sh                 # generate service secrets -> secrets/vault.sops.yml (encrypted)
sops secrets/vault.sops.yml             # add the manual keys listed in secrets/secrets-manifest.yml (tokens, hostnames)
scripts/secrets/vault-set.py set TF_VAR_linode_token   # one per variable listed in .env.example (value from stdin)
./setup --workspace linode-staging      # staging first, then --workspace linode --confirm-prod

New machine, existing infra: clone, then load an age key (Keychain sops-age/infra, or export SOPS_AGE_KEY="$(bw get notes sops-age-recovery-infra)") and run ./setup. scripts/secrets/init-secrets.sh refuses to overwrite an existing vault.

Prereqs (local)

• sops and age (secrets), jq, argon2; bw for the recovery key
• terraform
• ansible
• python3 (for helper scripts)
• pip / python3 -m pip
• SSH access to the hosts

If your SSH key is passphrase-protected, you must load it into your agent before running Ansible non-interactively:

ssh-add --apple-use-keychain ~/.ssh/id_ed25519

DNS (Cloudflare)

Create A/CNAME records that point to the correct server IP.

Active records:

• jfraeys.com -> A record to web server IPv4
• services.jfraeys.com -> A record to services server IPv4
• web-ssh.jfraeys.com -> A/AAAA to the web server, DNS-only (not proxied): SSH from CI (coaching-site's DEMO_REFRESH_SSH_HOST); follows web at cutover
• git.jfraeys.com -> A/CNAME to services (Forgejo)
• auth.jfraeys.com -> A/CNAME to services (Authelia)
• vault.jfraeys.com -> A/CNAME to services (Vaultwarden)
• vpn.jfraeys.com -> A/CNAME to services (Headscale control plane)
• demo.jfraeys.com -> A/CNAME to services (shared demo/sandbox gateway, path-routed by app slug)
• app.jfraeys.com -> A/CNAME to services (reserved for future permanent/prod apps — nothing uses this yet)

Commented out (unused):

• grafana.jfraeys.com -> A/CNAME to services (Grafana - currently disabled)
• prometheus.jfraeys.com -> A/CNAME to services (Prometheus - currently disabled)

To enable, uncomment the records in terraform/modules/dns/main.tf.

TLS

Traefik uses Let’s Encrypt via Cloudflare DNS-01.

You must provide a Cloudflare API token in your local environment when running Ansible:

• CF_DNS_API_TOKEN
• CF_ZONE_API_TOKEN

SSO (Authelia OIDC)

Authelia is exposed at:

• https://auth.jfraeys.com (issuer)
• https://auth.jfraeys.com/.well-known/openid-configuration (discovery)

Grafana is configured via roles/grafana using the Generic OAuth provider.

Forgejo is configured via roles/forgejo using the Forgejo admin CLI with --provider=openidConnect and --auto-discover-url.

Note: Forgejo pages that ask for an "OpenID URI" are legacy OpenID 2.0 and are not used for OIDC.

Access control groups (LLDAP)

LLDAP has no declarative group/user provisioning — groups and users are created manually via the LLDAP web UI (https:// LLDAP's exposed hostname, admin login). Authelia maps a group's cn directly to an access_control group:<cn> subject (see groups_filter / group_name_attribute in roles/authelia/templates/configuration.yml.j2), so creating a group in LLDAP is sufficient — no Authelia-side registration needed beyond the access_control rules already templated.

Groups are generated from inventory/group_vars/all.yml's client_sites list — roles/authelia/templates/configuration.yml.j2 loops over every entry with auth: authelia to build its access_control rules, and roles/lldap's lldap_required_groups derives the same groups from the same list, so nothing needs to stay manually in sync:

• admin — full access: all infra services (Grafana, Forgejo, Vaultwarden, Headscale) plus every client site.
• client-<slug>-admin (e.g. client-coaching-admin) — full admin-panel access to that one client's site, including destructive actions.
• client-<slug>-employee (e.g. client-coaching-employee) — read/edit access to that client's admin panel; destructive HTTP methods (DELETE/PUT/PATCH) are blocked by the access_control rule regardless of group membership.

Adding a new client only requires: add one entry to client_sites (see that variable's own comment for the schema), then add users to the two new LLDAP groups via the LLDAP UI once LLDAP creates them — no LLDAP/Authelia code changes needed beyond the one data entry.

Email (Postfix + Postmark)

Transactional email is delivered via Postfix relay to Postmark:

• Sender: notifications@jfraeys.com
• Relay: smtp.postmarkapp.com:2525
• Auth: Server token authentication

Services using email:

• Authelia (password resets)
• Alertmanager (monitoring alerts)
• Forgejo (CI/CD notifications)

DNS Records for Email

Terraform manages these Cloudflare records:

Record	Type	Purpose
YYYYMMDDDDpm._domainkey	TXT	DKIM signature
pm-bounces	CNAME	Return-path for bounces
_dmarc	TXT	DMARC policy

Postmark validates these during account setup.

Vault Variables

Add to the vault (scripts/secrets/vault-set.py set KEY from stdin, then scripts/secrets/vault-pr.py; rotate with ./infra rotate; key names and purposes: secrets/secrets-manifest.yml, .env.example):

Email (Postfix + Postmark):

POSTFIX_RELAYHOST_USERNAME: "your-postmark-server-token"
POSTFIX_RELAYHOST_PASSWORD: "your-postmark-server-token"
AUTHELIA_SMTP_SENDER: "notifications@jfraeys.com"
AUTHELIA_SMTP_IDENTIFIER: "jfraeys.com"

Backups (Restic):

RESTIC_REPOSITORY: "s3:https://us-east-1.linodeobjects.com/mybucket/backups"
RESTIC_PASSWORD: "strong-encryption-password"
# Also used to init the Terraform state S3 backend, not Restic-specific —
# see scripts/lib/setup/terraform.sh.
LINODE_OBJECT_STORAGE_ACCESS_KEY_ID: "your-linode-access-key"
LINODE_OBJECT_STORAGE_SECRET_ACCESS_KEY: "your-linode-secret-key"
# Optional:
LINODE_OBJECT_STORAGE_REGION: "us-east-1"
RESTIC_KEEP_DAILY: 7
RESTIC_KEEP_WEEKLY: 4
RESTIC_KEEP_MONTHLY: 6
INFRA_BACKUP_ONCALENDAR: "daily" # systemd calendar spec

# Optional dead-man's-switch heartbeats (create a free check per job at
# healthchecks.io, paste its ping URL here). Empty/unset disables that job's
# heartbeat. Works independently of Alertmanager/Prometheus (see Notes) —
# still fires even if services itself is unreachable and can't send its own
# alert email.
BACKUP_HEARTBEAT_URL: "" # infra-backup (daily)
BACKUP_VERIFY_HEARTBEAT_URL: "" # backup-verify (monthly)
BACKUP_RESTORE_TEST_HEARTBEAT_URL: "" # backup-restore-test (weekly)

# OVH Object Storage credentials (bucket/region are non-secret, see
# .env.example's OVH_OBJECT_STORAGE_BUCKET/_REGION) — generated via
# scripts/migration/ovh-setup-wizard.sh Stage 5. A separate bucket from
# LINODE_OBJECT_STORAGE_* above (still the live Linode bucket today).
OVH_OBJECT_STORAGE_ACCESS_KEY_ID: "your-ovh-s3-access-key"
OVH_OBJECT_STORAGE_SECRET_ACCESS_KEY: "your-ovh-s3-secret-key"

# Optional: secondary (dual-write) Restic target on the OVH bucket above —
# roles/backups writes to BOTH repositories on every scheduled backup when
# this is set, non-fatally logging (not failing) if only the secondary
# fails. Exists to prove OVH backups are actually restorable — see
# backup-verify's secondary-repo check — before RESTIC_REPOSITORY itself
# is ever switched over. Leave unset for no behavior change. Reuses
# RESTIC_PASSWORD above as its own encryption password — restic allows
# this, no separate password needed.
RESTIC_SECONDARY_REPOSITORY: "s3:https://s3.bhs.io.cloud.ovh.net/your-ovh-bucket/infra-backup"

Alerting (set exactly one — only relevant if Alertmanager/Prometheus are enabled, see Notes):

# Slack option:
ALERTMANAGER_SLACK_WEBHOOK_URL: "https://hooks.slack.com/services/..."
ALERTMANAGER_SLACK_CHANNEL: "#alerts"
ALERTMANAGER_SLACK_USERNAME: "alertmanager"

# Discord option:
ALERTMANAGER_DISCORD_WEBHOOK_URL: "https://discord.com/api/webhooks/..."

Secrets (SOPS + age)

Secrets live in secrets/vault.sops.yml, encrypted to the operator and recovery age keys in .sops.yaml and committed, so any machine holding an age key can run playbooks. Design: docs/secrets-management-sops.md; key recovery: docs/secrets-recovery.md; who owns what: docs/secrets-inventory.md.

• Edit/inspect: sops secrets/vault.sops.yml (age key from macOS Keychain, or SOPS_AGE_KEY).
• ./setup decrypts to secrets/vault.yml for its run and deletes it on exit.
• Direct ansible-playbook runs must go through scripts/secrets/with-vault.sh — playbooks skip a missing secrets/vault.yml silently, so a bare run deploys with every secret empty.
• Rotate with ./infra rotate <service> (./infra help lists the other credentials); it opens the vault PR and prints the rest: merge, ./infra approve, git pull, ./infra deploy --tags .... The plaintext secrets/vault.yml is gitignored and never committed.
• The legacy Ansible Vault file, if still present, is secrets/vault.yml.ansible-vault (retired after the staging deploy passes).

Notes:

• inventory/hosts.yml and inventory/host_vars/web.yml are generated by setup and intentionally gitignored

Playbooks

• playbooks/services.yml: deploy observability + forgejo on services
• playbooks/web.yml: deploy app-side dependencies on web
• playbooks/tests/test_config.yml: smoke test host config and deployed stacks
• playbooks/validate-*.yml: staging smoke tests (app, connectivity, hardening, reboot, services)

Configuration split

• Vault (secrets/vault.sops.yml): everything needed to provision or deploy (API tokens, passwords, access keys, sensitive TF_VAR_* values)
• .env no longer exists as a source: its keys live in the vault (.env.example lists the variable names). Per-machine values such as ANSIBLE_PRIVATE_KEY_FILE are exported in your shell.

Object storage for demo apps

If a demo app needs its own S3-compatible bucket, mint a dedicated access key scoped as tightly as possible — don't reuse LINODE_OBJECT_STORAGE_ACCESS_KEY_ID/LINODE_OBJECT_STORAGE_SECRET_ACCESS_KEY above, which are this infra repo's own credentials for the Terraform state backend + Restic backups, not meant to be shared with app-level code.

S3_ACCESS_KEY_ID/S3_SECRET_ACCESS_KEY/S3_BUCKET/S3_ENDPOINT/S3_REGION are deprecated, dead fallback names in this repo (see secrets/secrets-manifest.yml — confirmed dead 2026-08-30, InvalidAccessKeyId) — don't wire a new app up against them. No app role in this repo currently consumes generic S3_* vars at all; per this repo's app deployment contract, a demo app's own object-storage credentials and variable naming belong in that app's own repo, not here.

Grafana provisioning

Grafana is provisioned with Prometheus and Loki datasources via the Grafana provisioning mechanism (no manual UI setup required).

Note: Grafana is deployed but DNS records are commented out. Access via grafana.jfraeys.com by uncommenting the records in terraform/modules/dns/main.tf, or access directly via the services host IP.

Host vars

Set inventory/host_vars/web.yml:

• public_ipv4: public IPv4 of jfraeys.com

Used by roles/anubis and roles/traefik.

Forgejo Actions runner (ephemeral)

As of 2026-08-27 there is no persistent runner anywhere — roles/forgejo_runner_ubuntu (Docker-based, self-hosted label) was removed after repeated resource contention with everything else on services (see docs/incidents.md). roles/ci_ephemeral_provisioner runs on services instead: on a queued ephemeral-large-labeled job, it provisions a throwaway Linode VM, registers roles/forgejo_runner_native on it in ephemeral mode (playbooks/ci-ephemeral.yml), runs exactly one job via forgejo-runner one-job, then destroys the VM.

• The ephemeral provisioner's own required secrets are documented in roles/ci_ephemeral_provisioner/defaults/main.yml.
• Workflows must use runs-on: ephemeral-large — self-hosted matches no runner at all.

AI Scrapers Blocklist

Forgejo includes a weekly cron job (roles/forgejo/files/update-ai-scrapers.sh) that updates robots.txt to block AI scrapers (GPTBot, ClaudeBot, etc.).

OIDC Configuration

Forgejo is configured with:

• Group claim mapping from Authelia (groups)
• Admin group: admins
• Auto-discovery from https://auth.jfraeys.com/.well-known/openid-configuration
• ENABLE_AUTO_REGISTRATION=true + ACCOUNT_LINKING=auto — a new local Forgejo account is created automatically on first Authelia login, username/email taken straight from the OIDC claims. Required together on this Forgejo version for silent account linking to work at all (see CHANGELOG.md 2026-08-21) — Authelia's own login gate is what actually controls who can reach this, not Forgejo's registration setting.
• Git-over-SSH (git.jfraeys.com:2222) is restricted to the tailnet (roles/firewall's DOCKER-USER rules + Headscale's ACL) — clone/push over SSH requires an enrolled device; HTTPS is unaffected.

Nothing to manually re-register — each ephemeral runner registers fresh and is destroyed after its one job, so a bad registration only ever affects the next cycle, not a long-lived host.

Required Forgejo Actions secrets

The workflows reference these repo secrets. They are replicas of vault values (the vault is authoritative), listed in .forgejo/ci-secrets.yml and applied with scripts/secrets/sync-ci-secrets.py (check by default, writes nothing; after changing a vault value run ./infra secrets sync --apply --force --vault-key <KEY>, e.g. --vault-key TF_VAR_linode_token):

• LINODE_TOKEN — Terraform provisioning for staging nodes.
• CLOUDFLARE_API_TOKEN — Traefik's DNS-01 cert issuance during the services-check step (see Terraform/staging validation below).
• CI_FORGEJO_COMMENT_TOKEN — plan comments, issues and the promotion PR (its own repo-scoped token; the vault key is the same name).
• HEADSCALE_API_KEY — staging mints tailnet pre-auth keys through Headscale's API; created and rotated by scripts/secrets/provision-headscale-api-key.sh (90-day expiry).
• LINODE_OBJECT_STORAGE_ACCESS_KEY_ID / LINODE_OBJECT_STORAGE_SECRET_ACCESS_KEY — Terraform state backend; created by scripts/secrets/provision-ci-state-key.py.

Restoring them after losing the Forgejo instance: docs/runbooks/ci-secrets-restore.md. BW_SESSION is no longer used.

Plus FORGEJO_RUNNER_REGISTRATION_TOKEN — set on whichever repo runs the corresponding workflow (an onboarded app's own CI dispatches this repo's staging validation and needs its own copy of secrets it references directly).

Testing

Molecule (Local Role Testing)

Local only: CI does not run Molecule, because the ephemeral runner has no Docker by design.

Test individual roles locally with Docker before pushing:

# Test a specific role
cd roles/hardening
molecule test

# Test all roles
for role in roles/*/molecule; do
  role_name=$(dirname "$role" | sed 's|roles/||')
  echo "Testing $role_name..."
  (cd "roles/$role_name" && molecule test)
done

Common commands:

• molecule test - Full test cycle
• molecule create && molecule converge - Create and apply role
• molecule verify - Run verification tests
• molecule login - SSH into test container
• molecule destroy - Clean up

Prerequisites:

# macOS with Colima
colima start
export DOCKER_HOST="unix://${HOME}/.colima/default/docker.sock"

# Ensure venv is activated
source .venv/bin/activate

Staging Validation

Full integration testing with real cloud VMs:

# Create staging nodes, run validation, destroy
./setup --workspace linode-staging

# Keep nodes on failure for debugging
./setup --workspace linode-staging --keep-on-failure

# Re-run Ansible on existing staging nodes
./setup --workspace linode-staging --reuse-staging

Day-to-day Operations (manage CLI)

On the operator machine run these through ./infra (./infra health check, ./infra backup status, ...), which forwards to ./manage. ./manage (root of this repo) is the day-to-day ops companion to setup — health checks, backup status/restore, LLDAP user management, firewall rules, VPN nodes, and remote-CLI install/update. roles/manage_cli deploys it automatically to /opt/infra-manage (symlinked as infra-manage) on every services/web deploy, so it's always available there too.

manage --help
manage user list                 # LLDAP-backed, no --host needed
manage vpn status                 # Headscale-backed, no --host needed
manage backup status --host services
manage health check --host web

Commands tied to a single-host service — user (LLDAP) and vpn (Headscale), both only ever running on services — default to that host automatically, whether run from services itself or elsewhere; --host still works to override. health/backup/firewall default to localhost and need an explicit --host services/--host web when run from anywhere else, since either host is a valid target for those.

Deploy

Services:

./infra deploy --limit services   # Ansible-only deploy of the services host, promotion-gated

Web:

scripts/secrets/with-vault.sh ansible-playbook -i inventory/hosts.yml playbooks/web.yml

Reviewing production Terraform

Production is planned and applied from your machine, not CI: the vault holds every input and credential (CI cannot decrypt it, and a CI plan with Terraform's defaults wants to destroy live resources).

scripts/release/plan-prod.sh          # plan + drift (refresh-only) of the linode workspace; read-only
scripts/release/plan-prod.sh drift    # only: did the live infrastructure change outside Terraform?
PLAN_PROD_VERBOSE=1 scripts/release/plan-prod.sh plan   # full attribute-level diff

Exit 0 = clean, 2 = changes or drift, 1 = error. Apply with ./setup --workspace linode --confirm-prod. The staging-validate promotion PR is a reminder to run this before merging; CI does not apply production.

Terraform

Infrastructure is managed via modular Terraform in terraform/:

# Deploy infrastructure
./setup --no-ansible

# Or full deploy with Ansible
./setup

Workspaces and their tfvars. Region, instance type and image differ per provider and are not vault values. Each production/DR workspace has its own file, which ./setup passes by default (an explicit --var-file wins): linode uses terraform/environments/linode/prod.tfvars, ovh uses ovh/prod.tfvars, ovh-dr uses ovh/dr.tfvars, linode-dr uses linode/dr.tfvars (copied from dr.example.tfvars). Per-node sizes: ovh_node_flavors = { services = "<flavor>" } on OVH and linode_node_types = { services = "<plan>" } on Linode; nodes not named keep instance_type. Passthrough commands get the same file, e.g. ./setup --workspace ovh --confirm-prod -- plan -no-color.

Modules:

• modules/compute - Compute instances (provider-agnostic interface)
• modules/dns - Cloudflare DNS records

Provider portability: Swapping Linode for OVH requires only rewriting the compute module.

./setup will export TF_VAR_* from the decrypted vault (age key from Keychain or SOPS_AGE_KEY) and then run Terraform with a saved plan.

Notes

• Alertmanager/Grafana/Prometheus/Loki: each gated by its own *_enabled var in inventory/group_vars/all.yml (Traefik's routers/cert requests for these hostnames are gated on the same vars, not left live regardless). Currently: alertmanager_enabled/prometheus_enabled are true (re-enabled since the 2026-08-21 memory-headroom note that used to apply to all four); loki_enabled/grafana_enabled remain false — see FUTURE_IMPROVEMENTS.md's Infrastructure Capacity Upgrades section for why. Backup-job alerting works regardless via the Healthchecks.io heartbeat (see Backups section below).
• Loki is published as 127.0.0.1:3100:3100 (loopback-only, Docker-internal) — Grafana reads it over the internal monitoring Docker network, not the public IP. No UFW rule needed or present for it.
• Watchtower is enabled with label-based updates.
• Traefik: Uses file provider exclusively (Docker socket access removed). Services have static router definitions in /opt/traefik/dynamic/base.yml.
• Postfix: Relays through Postmark port 2525 (avoids ISP blocking on 587).
• Hardening: SSH config and unattended-upgrades managed via hardening role to prevent cloud-init drift.

Role layout

Services host (services, see playbooks/services.yml):

• roles/docker - supports both rootless and classic root Docker (docker_rootless_enabled, role default true). services overrides this to false (inventory/host_vars/services.yml) — rootless Docker there caused the "Disable system Docker when using rootless" task to stop docker.service but not docker.socket, so the next docker/docker compose call anywhere later in the same run silently restarted the daemon via systemd socket activation, intermittently breaking container egress. Treat rootful as this host's actual, current config, not an aspirational default — see SECURITY_CHECKLIST.md.
• roles/firewall (Cloudflare-allowlisted UFW + DOCKER-USER chains)
• roles/traefik (file provider only - no Docker socket)
• roles/lldap (directory backend for Authelia)
• roles/authelia (SSO / OIDC provider)
• roles/exporters (node-exporter + cAdvisor)
• roles/alertmanager (uses localhost:25 Postfix relay; optional - commented out in services.yml, same as prometheus/loki/grafana below — has nothing to alert on without Prometheus running anyway)
• roles/prometheus (optional - commented out in services.yml)
• roles/loki (optional - commented out in services.yml)
• roles/grafana (optional - commented out in services.yml)
• roles/forgejo
• roles/ci_ephemeral_provisioner (ephemeral Forgejo Actions runner — see "Forgejo Actions runner (ephemeral)" above)
• roles/vaultwarden (self-hosted password manager, SSO via Authelia OIDC)
• roles/headscale (VPN control plane — see VPN Access)
• roles/watchtower
• roles/postfix (Postmark SMTP relay for transactional email)
• roles/backups
• roles/fail2ban (Docker-based fail2ban)

Applied separately to every host via playbooks/hardening.yml:

• roles/hardening (SSH hardening, unattended-upgrades, per-host tuning via host_vars)

Web host (web, see playbooks/web.yml) — bare-metal, no Docker. The app itself is not deployed by this repo at all — it deploys itself via its own repo's Ansible playbooks/CD. This repo only provisions the generic base layer:

• roles/tailscale (VPN client — see VPN Access)
• roles/fail2ban (native backend, no Docker on this host)
• roles/hardening (hardening_docker: false, enable_container_scanning: false)

VPN Access (Headscale + Tailscale)

An app's admin panel (on the web host) can be network-restricted to a self-hosted tailnet, in addition to whatever app-level auth it uses. The VPN is self-hosted Headscale (control plane, roles/headscale on services) + the official Tailscale client apps (data plane — still WireGuard under the hood, but auto-configured instead of hand-managed). This replaced a previous hand-rolled wireguard role: that VPS was accidentally deleted and no peers ever re-joined, so there was no live state to migrate, and running a second VPN stack alongside Headscale would have been unnecessary weight on this infra's resources. DERP (NAT-traversal relay) is self-hosted too (derp.server.enabled in roles/headscale) — no dependency on Tailscale Inc.'s infrastructure for the data plane.

Human enrollment (e.g. whoever needs to reach the admin panel): install the official Tailscale app (Windows/macOS/iOS/Android — polished GUI, no config files) and log in. That login goes through Authelia — the same SSO page already used for Grafana/Forgejo/Vaultwarden. Headscale auto-approves the device on successful login; there's no separate approval step.

Server enrollment (web, via roles/tailscale) is unattended, using a bounded-expiration (~24h) pre-auth key instead of OIDC:

manage vpn preauthkey
# copy the printed key into vault as TAILSCALE_AUTHKEY, then:
ansible-playbook playbooks/web.yml --tags tailscale

The key goes inert on its own after expiry — no manual vault cleanup needed, and it can be regenerated any time via the same command (useful for disaster recovery / host replacement).

By default the key is single-use — the first successful tailscale up consumes it. Staging (./setup --workspace linode-staging) destroys and recreates both nodes on every run, so a single-use key only survives one staging run before services.yml's tailscale role fails with backend error: authkey already used. Pass --reusable for a key that survives repeated staging runs within its expiration window — this also switches the default expiration from 24h to 168h (7 days), since surviving many runs across several days is the point:

manage vpn preauthkey --tags tag:services --reusable

Anyone holding a reusable key can enroll multiple devices with it until it expires — keep the expiration short and don't share it outside this repo's own automation.

ACL: every non-infra node on the tailnet can reach web's nginx on tcp:443 only — no SSH, no other hosts, no lateral movement between client devices. See roles/headscale/templates/acl.hujson.j2.

Day-to-day operations: manage vpn status (list nodes), manage vpn revoke <node> (offboard a device — it can re-enroll afterward).

Roles: roles/headscale (control plane, services host), roles/tailscale (client, web host).

App Deployment (web host)

The web host is bare-metal, no Docker. The app running on it is deployed entirely by that app's own repo — its own Ansible playbooks for staging, its own webhook/CD pipeline for production — never by this repo. playbooks/web.yml only provisions the generic base layer (firewall, tailscale, fail2ban, hardening); it has no knowledge of what app, if any, runs on top of it.

Refer to that app's own repo/CI for its deploy/rollback commands and troubleshooting.