Promote to production #77

Merged

jfraeys merged 266 commits from main into prod 2026-10-13 15:34:33 -04:00

Conversation 0 Commits 266 Files changed 208 +8786 -1783

jfraeys commented 2026-10-12 21:05:09 -04:00

Owner

Copy link

Promotion: staging passed. Run scripts/plan-prod.sh locally before merging

Merging this PR is the approval record; it deploys nothing. Then check out the validated commit and run scripts/promote-prod.sh (or scripts/deploy-services.sh --tags ... --limit services for an Ansible-only change); both refuse any other commit.

Staging validation: passed on commit fd26dc385e (this run)
Production drift check: clean

Ansible preview (staging clone, synthetic secrets — approximate, not exact prod values)

PLAY [Deploy the Docker-based services stack] **********************************
skipping: no hosts matched

PLAY RECAP *********************************************************************

### Promotion: staging passed. Run `scripts/plan-prod.sh` locally before merging Merging this PR is the approval record; it deploys nothing. Then check out the validated commit and run `scripts/promote-prod.sh` (or `scripts/deploy-services.sh --tags ... --limit services` for an Ansible-only change); both refuse any other commit. **Staging validation:** passed on commit fd26dc385e8c86c9f40d2159a759c9a2a75e151e (this run) **Production drift check:** clean ### Ansible preview (staging clone, synthetic secrets — approximate, not exact prod values) ``` PLAY [Deploy the Docker-based services stack] ********************************** skipping: no hosts matched PLAY RECAP ********************************************************************* ```

jfraeys added 175 commits 2026-10-12 21:05:09 -04:00

feat(ci_ephemeral_provisioner): watch multiple repos, not just coaching-site ... 8028aee489

GitOps plan phase 1, step 0 (prerequisite for everything else): the
ephemeral CI runner controller only ever watched jfraeys/coaching-site's
queued jobs -- new infra workflows targeting runs-on: ephemeral-large
would queue forever with nothing picking them up.

Turns out job *discovery* (find_queued_jobs(), via
/admin/runners/jobs) was already instance-wide -- only get_task_status()
is actually repo-scoped by Forgejo's own API design, and each job object
only carries a numeric repo_id, not an owner/repo string. So the real
fix is smaller than "poll N repos": resolve repo_id -> (owner, repo) for
a configured watch list once per reconcile pass that needs it, filter
discovered jobs down to watched repos, and thread owner/repo through
cycle.json so get_task_status() (and VM-reuse's next-job selection, which
has the same instance-wide-discovery gap) query the right repo.

One repo-scoped token already covers every repo its owner can access
(Forgejo scopes are account-wide permission levels, not per-repo ACLs),
so no per-repo token/secret was needed -- just the existing
CI_EPHEMERAL_FORGEJO_REPO_TOKEN.

- ci_ephemeral_provisioner_forgejo_owner/_repo (single pair) ->
  ci_ephemeral_provisioner_watched_repos (list), defaults to
  [jfraeys/coaching-site, jfraeys/infra]
- CI_EPHEMERAL_FORGEJO_OWNER/_REPO -> CI_EPHEMERAL_WATCHED_REPOS
  (comma-separated)
- get_task_status(task_id) -> get_task_status(task_id, owner, repo)
- main()'s job selection and both VM-reuse call sites now filter to
  jobs whose repo_id resolves to a watched repo, and store owner/repo
  on the cycle -- a job from an unrecognized repo reusing our label is
  left alone, not guessed at

feat(gitops): Terraform plan/promote/apply/drift-check workflows ... 6ac857905f

GitOps plan phase 1, steps 1a-1d. Real credentials CI already has for
staging (LINODE_TOKEN, CLOUDFLARE_API_TOKEN) extend cleanly to owning
the Terraform layer end-to-end -- no invariant conflict, unlike the
Ansible layer (see roles/gitops_watcher, phase 4, not yet built).

New: .forgejo/workflows/terraform-plan.yml (1a) -- posts `tofu plan`
against the pinned `linode` workspace as a PR comment on any PR touching
terraform/**. Read-only, never applies. Explicit workspace select always
-- never the orphaned "default" workspace (docs/incidents.md: ungated by
local.skip_dns, ~62 unrelated resources).

New: .forgejo/workflows/drift-check.yml (1d) -- scheduled (6h) +
workflow_dispatch `tofu plan -refresh-only -detailed-exitcode` across
linode/ovh/ovh-dr (matrix), opens a Forgejo issue on drift. This is the
direct, automatic fix for the incident that prompted this whole plan:
Terraform declared cloudflare_record.proxied=true for vpn.jfraeys.com,
live had drifted to false. -refresh-only already does this comparison
correctly; the gap was purely "nobody runs it on a schedule."

Modified: .forgejo/workflows/staging-validate.yml -- added a `promote`
job (needs: validate) implementing 1b (promotion gate) and setting up
1c's trigger:
  - Pre-promotion production drift check: same -refresh-only mechanism
    as drift-check.yml, but run synchronously at promotion time --
    answers "is prod already wrong, don't build on top of it" rather
    than only "did something change since the last deploy" (drift-check
    still runs on its own schedule too, for drift introduced *between*
    promotions).
  - On clean drift check: opens/updates a main->prod PR with the staging
    summary, drift-check result, and a synthetic-secrets Ansible preview
    attached. Merging that PR is the entire production approval event --
    no separate approve command.
  - Removed the old "Run locally to deploy: ./setup --confirm-prod"
    message -- this replaces it.
  - Implemented as a same-workflow `needs:` job dependency rather than a
    separate promote-to-prod.yml relying on workflow_run/workflow_call,
    since Forgejo Actions' support for cross-workflow triggers wasn't
    confirmed -- needs: is unambiguously supported and guarantees
    ordering either way. Flagged as an open question in the plan;
    resolved this way during implementation.

New: .forgejo/workflows/terraform-apply.yml (1c) -- applies Terraform
automatically on push to `prod` (i.e. exactly when the promotion PR
above gets merged). Deliberately does not touch Ansible-managed config
-- roles/gitops_watcher (phase 4) owns that with real secrets this CI
job never has. Includes a placeholder step for signaling gitops_watcher
once that role exists.

All four files' YAML validated with pyyaml, embedded bash blocks synax-
checked with `bash -n`, embedded Python snippets compile-checked --
no live Forgejo Actions runner available in this environment to test
an actual run.

Merge pull request 'feat(gitops): Terraform plan/promote/apply/drift-check workflows + multi-repo CI provisioner' (#2) from phase1/gitops-workflows into main ... e7380bd9bb

Reviewed-on: #2

fix(scripts): detect Colima's XDG docker socket path ... 83c2b6d0cf

Newer Colima puts its socket at ~/.config/colima/default/docker.sock, which
config.sh never checked, so DOCKER_HOST stayed unset and Molecule's Python
Docker client failed with FileNotFoundError even though the docker CLI
(which resolves it through its context) worked.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(molecule): install Ansible collections in --setup ... 85b95615ef

Bare ansible-core ships no collections, and the repo-relative
./ansible/collections in ansible.cfg doesn't resolve when Molecule runs from a
role's directory. The firewall role's first collection module
(community.general.ufw) surfaced this: converge failed with "couldn't resolve
module/action". Install requirements.yml into ~/.ansible/collections.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(molecule): define traefik_certresolver in the firewall scenario ... d978e70213

The role's firewall_restrict_web_to_cloudflare default derives from it, and
undefined it made the Cloudflare-IP task's `when:` error out. Reproduced on
unmodified main: the firewall scenario already failed at converge before any
UFW change, so the Molecule step in staging-validate.yml could not have
passed for this role.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(hardening): don't start the reboot-alert timer under Molecule ... 2d193681c8

Molecule's fake systemctl (docker-systemctl-replacement) can't start .timer
units, so the hardening scenario failed at converge on "Enable and start
reboot-required alert timer". Reproduced against an identical roles/hardening
on main, so it already failed there; the timer was added after the scenario
was written. Skip only the start (still enable) when molecule_test is set --
production behavior is unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(molecule): pin the fake systemctl to a commit and verify its checksum ... 6df66e3c9a

hardening's prepare.yml fetched systemctl3.py from an unpinned master URL, so
an upstream change could break the scenario with no change here, and nothing
verified the file that runs as root in the container. Pin to
gdraheim/docker-systemctl-replacement 15a2509 (v1.7.1311, byte-identical to
master at the time) with its sha256. Also drop ignore_errors, which would
have turned a checksum failure into a confusing converge error later.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(security): verify downloaded binaries with checksums; make docker-bench actually run ... ec10c76277

Binaries pinned to a version but fetched with no integrity check (a tampered
release or hijacked upstream account would go unnoticed) now carry a
SHA-256 from the vendor's published sums, each also recomputed independently
from the artifact: OpenTofu 1.12.6 (provisioner and runner roles, amd64 and
arm64), Forgejo runner 13.0.0 (runner and cache-host roles), Docker Compose
v2.27.0 fallback. Bumping a version without its checksum now fails loudly.

docker-bench-security was fetched from upstream's master with no
verification, and also never worked: the script finds functions/ and
tests/ relative to its working directory, but only the single .sh was
copied into /usr/local/sbin and the unit set no WorkingDirectory, so the
weekly scan has just logged "No such file or directory" (the download's
failed_when: false hid the rest). Now: pinned release commit (v1.6.1) with
a checksum, installed as a directory, unit runs from it. This makes a
previously dead scan start producing output on the next deploy. Stale
standalone copy is removed.

Not changed: the pipe-to-shell installers (crowdsec, trivy, nodesource,
yq), GPG keys, and authelia:latest. forgejo_runner_macos is also
untouched: its version "11" 404s on both hosts, so there's nothing valid
to checksum.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(security): replace pipe-to-shell installs and pin remaining downloads by checksum ... 8ce6dab2dd

- crowdsec/nodesource: explicit apt repo setup with checksum-pinned keys
- yq: pinned version, per-arch sha256, curl -f
- docker/trivy/tailscale apt keys: checksum pinned
- cloud-init: pin ansible==12.3.0
- trivy scan script: no self-install from main
- macOS runner: require caller-supplied URL + sha256 (no upstream darwin build)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix/pin-downloads' (#3) from fix/pin-downloads into main ... fdc1871688

Reviewed-on: #3

cleanup: remove forgejo_runner_macos role (no upstream darwin binary) ... d245c64e09

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'cleanup: remove forgejo_runner_macos role (no upstream darwin binary)' (#4) from cleanup/remove-macos-runner into main ... 4fe1d5d5d9

Reviewed-on: #4

ci: add download-hygiene guard, lint workflow and weekly pin-drift check; fix molecule hardening ... 64a6d430c7

- scripts/check-downloads.py: hygiene (pipe-to-shell, get_url checksum, pip pin) and drift (key hashes, yq, release URLs)
- lint.yml: hygiene + shellcheck block; ansible-lint/yamllint advisory
- pin-drift.yml: weekly upstream comparison
- hardening: skip reboot-required timer start under molecule (no systemd)
- config.sh: detect Colima's ~/.config socket path

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(lint): clean ansible-lint, align Python/tooling versions ... 5c22521ae1

- fqcn fixes (include_vars, apt, ufw, posix.mount), play name, comment spacing
- .ansible-lint: skip stylistic rules, exclude molecule scaffolding
- lint.yml: install pinned tooling from requirements.txt, dummy vault file; all steps block
- requirements.txt: add ansible-lint, yamllint
- .python-version 3.13 (newest supported by ansible-core 2.19); molecule-test.sh builds .venv to match

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(molecule-test): fall back to system python3 when .python-version isn't installable ... c624e8e63e

The Debian 12 CI runner has no uv or python3.13; --setup must not abort there.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

ci(staging): allow trialing a different image (Debian 13) without changing the default ... 1f366b2b07

- staging-validate workflow_dispatch gains an image input (default linode/debian12)
- staging-debian13.tfvars for local ./setup --var-file runs

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'ci/guards-and-drift' (#6) from ci/guards-and-drift into main ... aac80235f2

Reviewed-on: #6

fix(tailscale): stop leaking the preauth key in failure output ... e8cd934769

The tailscale up command line carries --authkey; Ansible printed it in full when
the join failed (e.g. 'authkey expired'). no_log the task and fail separately
with stderr, which never contains the key.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(ci-ephemeral): circuit breaker for repeated provisioning failures ... 9c83d50b29

An expired Headscale preauth key made the provisioner create and destroy a billed
VM every ~3.5 min indefinitely. Now N consecutive provisioning failures (default 3)
pause new cycles for a cooldown (default 30 min), then allow one trial cycle; a cycle
reaching running resets it. Failures count once per cycle, and known causes (expired
key, rejected SSH key, checksum mismatch) log a fixed hint without echoing output.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(ci-ephemeral): alert when the provisioner circuit breaker opens ... 7ee8144f9b

The breaker exports circuit_open/consecutive_failures via node-exporter's textfile
collector; CIEphemeralProvisionerCircuitOpen (warning, for 2m) fires on it and routes
through the existing Alertmanager receiver. Metrics rewrite every tick so clearing
breaker.json clears the alert; a metrics write failure never blocks provisioning.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(tailscale): stop leaking the preauth key in failure output' (#8) from fix/tailscale-authkey-no-log into main b871805951

Merge branch 'main' into feat/provisioner-circuit-breaker ... 44167f4bf6

# Conflicts:
#	roles/ci_ephemeral_provisioner/tasks/main.yml

Merge pull request 'feat(ci-ephemeral): provisioner circuit breaker and alert' (#9) from feat/provisioner-circuit-breaker into main 092af98d96

fix(lint): run on the ephemeral runner without sudo or system pip; add step timeouts ... 9cc8402f71

The runner user has no sudo (only 'ansible' does) and Debian 12's system Python is
externally managed, so the apt shellcheck install and bare pip install could not work.
Use a venv from requirements.txt, with shellcheck from PyPI (shellcheck-py), and
per-step timeouts so a hang fails fast and names the step.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(ci): make drift/plan/apply workflows non-interactive and surface tofu errors ... acaee9b1f0

The drift job ran tofu init/plan with no TF_INPUT=0, no backend credentials and no values for
root_pass/user_password, so tofu waited for typed input until the job timeout. Set TF_INPUT=0,
pass the state backend settings (bucket/region/endpoint non-secret, key pair from repo secrets),
use placeholders for the refresh-only plan, and fail the step on tofu exit 1 (previously
swallowed, so a failed run looked green).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

ci(drift): schedule only the permanently-provisioned linode workspace ... 6d49fc1a73

ovh needs OVH credentials this workflow doesn't pass; ovh-dr isn't permanently provisioned, so a
scheduled refresh-only plan has nothing to compare and bills a runner VM every 6h.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(lint): run on the ephemeral runner without sudo or system pip' (#10) from fix/lint-runner-no-sudo into main 38eadd212d

Merge branch 'main' into fix/drift-check-noninteractive 443469a97e

Merge pull request 'fix(ci): non-interactive drift/plan/apply workflows; surface tofu errors' (#11) from fix/drift-check-noninteractive into main 6410980d40

docs: incidents for the provisioner key-expiry loop, Debian 13 failure, and CI hangs ... b3dc361188

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

test: molecule scenarios for crowdsec and forgejo_runner_native ... 4e5847de1c

Both run only the image_bake subset (--tags via ansible_args) on a Debian 13 container: crowdsec
checks the pinned repo key, sources file, removal of the old key, and package resolution;
forgejo_runner_native checks the NodeSource key checksum, Node major, OpenTofu version, runner
binary checksum, user and baked tools. Package installs needing live systemd are skipped under
molecule_test; /etc/hosts uses unsafe_writes there (bind mount). Verify plays are tagged always:
ansible_args --tags would otherwise skip every assertion silently.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'docs: incidents for the provisioner key-expiry loop, Debian 13 failure, CI hangs' (#12) from docs/incidents-2026-10-02 into main 63a83ef85b

Merge pull request 'test: molecule scenarios for crowdsec and forgejo_runner_native' (#13) from test/molecule-crowdsec-runner into main a3d26d58e5

Merge pull request 'ci(staging): allow trialing a different image (Debian 13) without changing the default' (#5) from staging/try-debian13 into main c39f4ed91a

Merge branch 'main' into fix/molecule-tooling ... aabd9e95ec

# Conflicts:
#	roles/hardening/tasks/main.yml
#	scripts/lib/setup/config.sh

Merge pull request 'fix(molecule): tooling fixes (pinned systemctl, collections in --setup, firewall scenario, Colima socket)' (#14) from fix/molecule-tooling into main ... c324310a21

Reviewed-on: #14

feat(ci-runner): install gnupg so release jobs can sign artifacts ... 21308be1e7

forgejo_runner_native now installs gnupg (tags base and image_bake). apt alone only ships
gpgv, which cannot sign or import keys, and nothing else in the ephemeral playbook pulls
gnupg in. The base run installs it on each ephemeral VM; rebuilding the Packer image bakes
it in.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'feat(ci-runner): install gnupg so release jobs can sign artifacts' (#16) from ci/runner-gnupg into main ... 4bc75afc6f

Reviewed-on: #16

fix(ci-runner): let the provisioner watch every repo of a trusted owner ... f7d75f9b0a

ephemeral-large jobs from repos not in ci_ephemeral_provisioner_watched_repos are ignored by
design, so jobs from jfraeys/forge-agent stayed queued forever. The list now accepts owner/*
(every repo of that owner the repo token can see, resolved on each pass, so new repos need no
redeploy) and defaults to jfraeys/*. The list remains the trust boundary: there is deliberately
no wildcard for every repo on the instance.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(ci-runner): let the provisioner watch every repo of a trusted owner' (#17) from ci/provisioner-watch-owner into main ... 5431e6324e

Reviewed-on: #17

ci-runner: bake the UFW package into the ephemeral image ... 70b5f7aad3

The Install UFW task was tagged base only, so every ephemeral VM installed the package from the network.
Tagging it image_bake too moves that into the image. The UFW rules and enable stay per-VM: Packer's ansible
provisioner connects through a local proxy port, so baked rules would allow the wrong port and a baked
enable would deny SSH on 22 at boot.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'ci-runner: bake the UFW package into the ephemeral image' (#18) from ci/bake-ufw-package into main ... c6ccd853a1

Reviewed-on: #18

feat(ci-runner): bake the auditd package into the ephemeral image ... 50b0123f04

Tag hardening's alerting include and its auditd install with image_bake so
the package is installed during the Packer bake instead of on every VM
start. Only the install is baked; the other alerting.yml tasks are untagged
and still run at provision time.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'feat(ci-runner): bake the auditd package into the ephemeral image' (#19) from ci-runner-bake-auditd into main ... 8ecfca6914

Reviewed-on: #19

docs(changelog): record the auditd package bake ... ae7f270f84

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'docs(changelog): record the auditd package bake' (#20) from changelog-auditd-bake into main ... d8b86575ba

Reviewed-on: #20

docs(runbooks): mark Vaultwarden/Moodle as non-blocking in DR failback gate ... d1787ddb2a

Vaultwarden isn't used (SOPS/Vault instead) and Moodle isn't deployed yet,
so their failback rows shouldn't gate the drill alongside Forgejo,
Authelia, and lldap, which are the actual production dependencies.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

fix(terraform): scope ovh_object_storage_iam out of staging/DR workspaces ... 2737fc53d4

A routine `linode-staging` provision destroyed production's real OVH
secondary-backup S3 IAM policy: the module was gated only on
ovh_backup_s3_username being set, not on workspace, so staging's own
state independently tracked and could destroy the same live resource
production's state manages. Excludes staging and ovh-dr from the
module's count so only the canonical production workspace touches it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

docs(incidents): record AUTHELIA_OIDC_PRIVATE_KEY_PEM stray-quote corruption ... aa6fe3cc95

Found while deploying linode-staging for the lldap/Authelia/Forgejo
failback drills: a stray leading quote in vault corrupted the PEM header,
which would have crash-looped Authelia on the next production redeploy.
Production was never actually down — its running config predates the
corruption — but this was a live time bomb until the user fixed it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

fix(lldap): correct restore procedure's volume name and rsync dependency ... 982fdedf3c

The first real lldap restore drill (linode-staging, 2026-09-23) found the
documented volume path was wrong: Compose auto-prefixes the unnamed
lldap_data volume with its project name, giving lldap_lldap_data on disk,
not lldap_data. This affects production identically, not just staging.
Also swaps rsync for cp -a in the restore step since rsync isn't
installed anywhere in this repo. Checks off the corresponding drill-gate
item now that the corrected procedure has actually been run and verified
(marker user survived the restore).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

fix(tailscale): run read-only status/ip commands under --check ... 4a9b3ac99c

tailscale status --json and tailscale ip -4 are command tasks, which Ansible
skips in check mode. The empty stdout then broke from_json and left
tailscale_own_ipv4 undefined, so any --check run of a play using this role
failed before reaching its own tasks.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

docs: record the .env authkey override incident, repo-token read:user scope, and the baked gnupg image ... 05cd8695b8

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

docs(tailscale): note that read-only command tasks feeding set_fact need check_mode: false ... ebfff8a226

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(deploy-services): keep the caller's authkey over .env, mint 168h keys ... 33917f8d2b

The script sourced .env after the caller's exports, so a stale
CI_EPHEMERAL_TAILSCALE_AUTHKEY line there overrode a freshly minted key and
the deploy kept writing the expired one. Snapshot the caller's
CI_EPHEMERAL_TAILSCALE_AUTHKEY / FORGEJO_RUNNER_APP_SKELETON_DEPLOY_KEY,
drop whatever .env or vault set for them (with a warning), then restore the
caller's. The self-minted key now lasts 168h instead of 4h: the provisioner
reuses it for every VM cycle until the next deploy.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(ci_ephemeral_provisioner): don't blank the installed app-skeleton deploy key ... f56be67b34

The install task wrote the exported key unconditionally, so a deploy run
without FORGEJO_RUNNER_APP_SKELETON_DEPLOY_KEY replaced the installed key
with an empty file. force is now true only when a key was supplied; the file
is still created when absent because the provisioner script opens it every
cycle.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

docs(incidents): update the authkey incident for the script and role fixes ... d5a1c93032

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

docs(ci_ephemeral_provisioner): note why the deploy key task uses force, not when ... 52126669c7

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(ci_ephemeral_provisioner): alert before the Tailscale auth key expires ... 431bd8a77e

deploy-services.sh records the key's expiry when it mints one
(CI_EPHEMERAL_TAILSCALE_AUTHKEY_EXPIRES_AT, Unix seconds); the role writes it
as a node-exporter textfile metric, and Prometheus warns 48h before expiry
and goes critical once expired, reaching the existing Alertmanager email
route. A deploy without an expiry leaves the last known metric in place; a
malformed value fails the play.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

docs(ci_ephemeral_provisioner): drop the wrong history from the app-skeleton deploy key comment ... abcb33f04c

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(tailscale): run read-only status/ip commands under --check' (#21) from jfraeys/fix/tailscale-check-mode into main ... bc5acc3167

Reviewed-on: #21

Merge pull request 'docs: record the .env authkey override incident and the 2026-10-08 changes' (#22) from jfraeys/docs/authkey-env-override-incident into main ... a50c66110d

Reviewed-on: #22

Merge pull request 'fix(deploy-services): keep the caller's authkey over .env, and don't blank the app-skeleton key' (#23) from jfraeys/fix/deploy-services-authkey-env into main ... ff26593797

Reviewed-on: #23

Merge pull request 'feat(ci_ephemeral_provisioner): alert before the Tailscale auth key expires' (#24) from jfraeys/feat/authkey-expiry-alert into main ... 8fcae645dc

Reviewed-on: #24

chore(lint): make .yamllint comply with ansible-lint's requirements ... d5a5337b88

ansible-lint warned on every CI run that the custom .yamllint is incompatible:
comments.min-spaces-from-content must be 1, braces.max-spaces-inside 1 and
octal-values forbid implicit/explicit octals. Set those; yamllint still
reports no findings across roles, playbooks, inventory and .forgejo.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

chore(ci): pause the scheduled drift check until its secrets exist ... b25b7ca4d1

The infra repo has none of the repo secrets drift-check reads (LINODE_TOKEN,
CLOUDFLARE_API_TOKEN, OBJECT_STORAGE_ACCESS_KEY_ID/SECRET_ACCESS_KEY,
FORGEJO_ADMIN_TOKEN), so every 6-hourly run built an ephemeral VM and failed
in tofu init with an empty backend bucket/region. Keep workflow_dispatch;
note where each value lives (vault, or .env TF_VAR_* for the two the vault
lacks) and record that the 2026-10-02 incident's 'not yet added' is still
true.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'chore(lint): make .yamllint comply with ansible-lint's requirements' (#25) from jfraeys/chore/yamllint-ansible-lint-compat into main ... 1564a06c70

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'chore(ci): pause the scheduled drift check until its secrets exist' (#26) from jfraeys/chore/pause-drift-check-schedule into main ... 747c34919b

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

chore(secrets): add age recovery recipient and P0 secrets inventory ... 3802c6af30

Add a second age recipient (recovery key, private half in personal
Bitwarden) to .sops.yaml, a keygen script that stores it without printing
it, the P0 credential ownership inventory, and the recovery-key section of
secrets-recovery.md. Recovery-only decrypt drill not yet recorded.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

docs(secrets): record recovery-only decrypt drill result ... 54a9ec557e

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

docs(secrets): record out-of-scope drill; close P1 gate ... e115128aa7

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'jfraeys/chore/secrets-p0-p1-recovery-key' (#27) from jfraeys/chore/secrets-p0-p1-recovery-key into main ... de6847f52a

Reviewed-on: #27

chore(secrets): allow committing *.sops.yml; keep all recipients on rotation ... e63c90114e

.gitignore's secrets/* also hid vault.sops.yml, so the encrypted vault could
never reach git. rotate-secret.sh and rotate-ovh-s3-credentials.sh parsed a
single recipient from .sops.yaml and would have dropped the recovery key;
both now use scripts/lib/sops-recipients.py.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'chore(secrets): allow committing *.sops.yml; keep all recipients on rotation' (#28) from jfraeys/chore/secrets-p2-gitignore-sops into main 203c06e5f1

feat(secrets): cut vault over to SOPS+age ... ddd89984db

Add secrets/vault.sops.yml (36 keys, encrypted to the operator and recovery
recipients) and scripts/sops-cutover.sh that produced it. The Ansible Vault
original is kept locally as secrets/vault.yml.ansible-vault until staging and
a real rotation pass.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

docs(secrets): record cutover and fresh-clone recovery drill ... c59d2e3f82

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

chore(secrets): rotate vaultwarden secrets under SOPS ... 9364d77cc4

First real rotation after the SOPS cutover: VAULTWARDEN_ADMIN_TOKEN and the
shared OIDC client secret pair (Authelia + Vaultwarden). Both recipients
verified in the re-encrypted file; only those three keys changed. Not yet
deployed: replay the authelia and vaultwarden roles together.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

chore(secrets): record 2026-10-09 vaultwarden rotation in manifest ... c442ea9a32

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(secrets): make direct playbook runs and CI work after the SOPS cutover ... e5693bb101

Playbooks include_vars secrets/vault.yml only when present, so a bare
ansible-playbook after the cutover ran with every secret empty. Add
scripts/with-vault.sh (decrypt for the command's duration, always clean up),
use it in deploy-services.sh and the rotate-secret.sh next-steps, and add
authelia to the vaultwarden redeploy tags (shared OIDC client secret).
./setup now continues without secrets in CI, which deliberately has no age
key, instead of exiting 2 now that vault.sops.yml is tracked.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(secrets): make rotate-secret.yml work under SOPS ... 12e88709d6

Accept vault.sops.yml, drop the .vault_pass requirement when SOPS is live,
refresh the plaintext vault.yml after rotation so the replay play deploys the
new values (with-vault.sh's copy predates them), and refuse to run outside
with-vault.sh so the plaintext copy is always cleaned up.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'feat(secrets): cut vault over to SOPS+age (P2)' (#29) from jfraeys/chore/secrets-p2-cutover into main ... b0961e8d3c

Reviewed-on: #29

docs(secrets): update README and runbooks for the SOPS vault ... 9ba27b15a0

README secrets section, deploy examples (with-vault.sh instead of
--ask-vault-pass), .env-vs-vault split and BW_SESSION note; vault-lockout
runbook marked legacy-only; ovh-setup-wizard hints point at sops. Working
Ansible Vault code paths are untouched until the legacy vault is retired.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

docs(secrets): add Ansible Vault retirement checklist ... 5d7dcf6a02

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(secrets): make first-time init work end to end under SOPS ... 3f0a031b3b

Add scripts/init-secrets.sh (generate service secrets straight into an
encrypted vault.sops.yml; tested in a scratch repo: 24 keys, 2 recipients) and
scripts/age-operator-keygen.sh (operator key into the Keychain). ./setup now
warns when no vault exists at all instead of continuing silently, and the
README gains prereqs and a First-time init section.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

chore(scripts): remove two uncalled shell functions ... ff47630f3b

create_password_item (init-vaultwarden.sh) and log_rotate (rotate-secret.sh)
are defined but referenced nowhere.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(terraform): gzip cloud-init user_data to stay under Linode's 16 KiB limit ... 01681eda30

The rendered services cloud-init (11.9 KB template plus Docker GPG key,
Cloudflare IPs, SSH key) exceeded the 16384-byte decoded user_data limit,
failing every staging provision. base64gzip drops it to ~4.5 KB; cloud-init
decompresses gzip user_data natively. metadata is already in ignore_changes
on both instance resources, so existing nodes are not replaced.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(setup): put mktemp's Xs last so staging temp files work on macOS ... 739af494c3

BSD mktemp doesn't substitute XXXXXX when a suffix follows, so it created a
literal staging_tailscale_key.XXXXXX.err and every later run failed with an
empty path ("cat: : No such file"), aborting the ephemeral Tailscale key mint.
Drop the .log/.err suffixes in all seven templates.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(setup): always delete the plaintext vault.yml, even after staging traps ... b260c4bbc8

setup registered the SOPS plaintext cleanup as an EXIT trap, but staging.sh
replaces the EXIT trap and clear_staging_trap removes it entirely, so every
staging run left secrets/vault.yml on disk (found as with-vault.sh's
leftover guard firing). Move the deletion into common_vault_remove_plaintext
and call it from cleanup(), both staging EXIT handlers, and by restoring
cleanup() as the EXIT trap in clear_staging_trap.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'docs(secrets): SOPS docs, first-time init, dead-code cleanup' (#30) from jfraeys/chore/secrets-docs-cleanup into main ce2c6dc238

Merge pull request 'fix(staging): gzip cloud-init user_data; mktemp templates that work on macOS' (#31) from jfraeys/fix/cloud-init-gzip into main 052aba8b52

Merge pull request 'fix(setup): always delete plaintext vault.yml, even after staging traps' (#32) from jfraeys/fix/setup-plaintext-vault-cleanup into main a5a1498c76

fix(ci): pass the state backend settings explicitly to every tofu-init workflow ... 08558ac45b

terraform-plan.yml assumed the runner baseline already had the S3 backend
settings and said to confirm on first real run; run 254 failed at tofu init
("bucket must not be empty", "Missing region"). Declare the bucket/region/
endpoint and the OBJECT_STORAGE_* repo secrets at workflow level in
terraform-plan, terraform-apply, staging-validate and staging-reboot-check,
matching drift-check. The two repo secrets must exist for this to take effect.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(ci): bucket-scoped Linode key for the Terraform state backend ... a66ab5296d

Add scripts/provision-ci-state-key.py (creates the limited key, stores it in
the SOPS vault first, then sets the Forgejo OBJECT_STORAGE_* repo secrets;
nothing secret is printed), the two vault keys it wrote, their manifest
entries, and the finding that state and restic share one bucket.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(ci): pass state backend settings to every tofu-init workflow' (#33) from jfraeys/fix/ci-state-backend-env into main 272d26efb7

Merge pull request 'feat(ci): bucket-scoped Linode key for the Terraform state backend' (#34) from jfraeys/chore/ci-state-backend-key into main b34c08cf2b

fix(manage): keep ssh stderr out of the Headscale JSON stream ... 33a92a8370

_headscale ran ssh with 2>&1, so any ssh warning or error merged into the
stdout that callers pipe to jq, failing a staging run's ephemeral Tailscale
key mint with "jq: parse error: Invalid numeric literal". stderr now stays on
stderr; every caller parses stdout JSON or discards it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(ci): drop the Molecule step from staging-validate ... fdadb35016

The ephemeral runner is a Docker-less host executor by design, so Molecule's
docker driver fails with "Unable to contact the Docker daemon" (run 257)
before any staging step. The step dates from the old Docker runner. Molecule
stays a local Colima step; README notes it is not run in CI.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(ci): drop the Molecule step from staging-validate' (#35) from jfraeys/fix/staging-validate-drop-molecule into main 9897c9e162

Merge pull request 'fix(manage): keep ssh stderr out of the Headscale JSON stream' (#36) from jfraeys/fix/headscale-ssh-stderr into main ... 3751848de3

Reviewed-on: #36

chore(secrets): P3 - record owner/replicas/privileges/expiry/rotation per secret ... f676ff5e1b

Every manifest entry now carries an owner, replicas, privileges, expiry and
rotation procedure (unknowns recorded as "unknown", not guessed), plus 13
entries for secrets that lived only in .env or the vault: OS_PASSWORD,
TF_VAR_* provider tokens, Zedmail/Postfix API keys, OVH_APPLICATION_SECRET and
VAULT_DEPLOY_TOKEN. The 11 manifest entries with no key in the real vault are
marked unset rather than dropped. Original fields are unchanged and
`manage secret list-expiring` still reads the file.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(secrets): copy every .env key into the vault (one source of truth) ... 6a7a7315cc

Adds scripts/migrate-dotenv-to-vault.py and applies it: 67 .env keys moved
into secrets/vault.sops.yml under their existing names (the roles already
read VAR with the env as fallback, so vault values need no role changes).
OVH_OBJECT_STORAGE_ACCESS_KEY_ID/SECRET_ACCESS_KEY differed between .env and
the vault; the vault copy was kept (it already wins everywhere). .env still
exists and is still read; removing it is the next change.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(secrets): retire .env - the SOPS vault is the single source of truth ... 8c4c5813b7

setup, deploy-services.sh, backup-terraform.sh, rotate-ovh-s3-credentials.sh
and recover-secondary-repo.sh no longer read .env; they use the vault.
common_vault_load_env now also exports the OS_/OVH_/DOMAIN/DESEC_/ZEDMAIL_/
AUTHELIA_SMTP_SENDER|IDENTIFIER names Terraform and config.sh read from the
environment. The OVH and Cloudflare setup wizards read/write the vault through
the new scripts/vault-set.py (stdin, atomic, verified) instead of .env.
provision-ci-state-key.py shares scripts/lib/sopsvault.py. The spent
migration script is removed. Docs and .env.example (now a name catalog) updated.

Verified: effective environment identical for 67/69 keys with .env gone (the
other two are the stale OVH_OBJECT_STORAGE_* values, vault wins as before), and
`./setup --workspace linode-staging -- plan` succeeds with .env moved aside.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'feat(secrets): retire .env - SOPS vault is the single source of truth (P4)' (#38) from jfraeys/feat/p4-dotenv-into-vault into main fe0a364583

chore(scripts): remove two spent one-off scripts ... 554121e938

sops-cutover.sh already ran (it now refuses to: vault.sops.yml exists) and
recover-secondary-repo.sh already ran on 2026-09-23 (RESTIC_SECONDARY_REPOSITORY
in the vault points at the fresh infra-backup-v2 repo it created), and it was
Ansible-Vault-only and refused to run under SOPS. Porting a completed one-off
would be dead weight. The retirement checklist is updated.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'chore(scripts): remove two spent one-off scripts' (#39) from jfraeys/fix/recover-secondary-sops into main 8624c9d41b

chore(lint): scope ansible-lint to files it can analyze ... f744676165

Exclude role templates (71 .j2), shipped files, Python tests and markdown from
ansible-lint's directory expansion: it processed 167 files and now processes
about 93, with identical (zero) findings. Local wall time is unchanged
(~13s, dominated by per-playbook syntax checks), so the gain is a smaller scope
and no "Directory expansion discovered 166 file(s)" noise, not speed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(playbooks): apply the apt lock wait to every apt task, not one ... ba096b4d35

web.yml set lock_timeout: 120 as a parameter on the single "Install UFW" task;
every other apt task (crowdsec, fail2ban, hardening, ...) kept apt's 60s
default and failed on a fresh VM whose unattended-upgrades held the dpkg lock
(crowdsec=1.8.1 install, staging run 2026-10-10). Move it to play-level
module_defaults (as ci-ephemeral.yml already does) in web, services and
hardening, at 300s. Verified in a scratch play that an apt task inside a role
gets lock_timeout 300 with the default and 60 without it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'chore(lint): scope ansible-lint to files it can analyze' (#40) from jfraeys/chore/ansible-lint-exclude into main c9d9b82b9f

Merge pull request 'fix(playbooks): apply the apt lock wait to every apt task' (#41) from jfraeys/fix/apt-lock-module-defaults into main e129679bb2

fix(ci): run lint when its own config changes ... 24a2dca340

lint.yml only triggered on roles/, playbooks/, scripts/, terraform/ and itself,
so a change to .ansible-lint or .yamllint (e.g. the exclude_paths narrowing in
#40) never ran lint in CI and could not be verified there.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(ci): run lint when its own config changes' (#42) from jfraeys/fix/lint-trigger-config-files into main 4f547e1127

feat(ci): P5 - declare CI secrets as vault replicas and sync them; drop BW_SESSION ... dd10a6eb32

.forgejo/ci-secrets.yml maps each Forgejo repo secret to the vault key it
replicates; scripts/sync-ci-secrets.py checks (default, read-only) or writes
them (--apply, --force, --only), reports names and statuses only and exits
nonzero on any failure. Tested against the real repo with a throwaway
non-secret secret (good key written, missing vault key reported, rerun skipped,
--force rewrote, deleted afterwards). BW_SESSION is removed from the two
workflows that referenced it. README and a restore runbook updated.

Not applied: LINODE_TOKEN, CLOUDFLARE_API_TOKEN, FORGEJO_ADMIN_TOKEN are still
missing as repo secrets; writing them is a separate decision.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(ci): rotation scripts keep the Forgejo secret replicas current ... 629bffb57c

sync-ci-secrets.py gains --vault-key (sync only the secrets replicating the
given vault keys; a no-op if none do). rotate-secret.sh records every key it
changes and syncs those; rotate-ovh-s3-credentials.sh does the same for its
pair; provision-ci-state-key.py drops its own Forgejo write code for the shared
sync. If the vault write succeeds but the sync fails, the script reports the
replica as STALE, prints the fix and exits nonzero. Stacked on the P5 PR.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'feat(ci): rotation scripts keep the Forgejo secret replicas current' (#44) from jfraeys/feat/rotation-ci-sync into main ... edd8e609d9

Reviewed-on: #44

fix(ci): rename the FORGEJO_ADMIN_TOKEN Forgejo secret to CI_FORGEJO_ADMIN_TOKEN ... e3b3952e18

Forgejo forbids secret names starting with FORGEJO_, GITEA_, GITHUB_ or a
digit, so secrets.FORGEJO_ADMIN_TOKEN could never exist and always resolved
empty in drift-check, terraform-plan/apply and staging-validate (PR-comment and
status posts were silently failing). The secret is now CI_FORGEJO_ADMIN_TOKEN
(vault key unchanged). sync-ci-secrets.py validates names client-side against
that rule before calling the API.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(ci): rename the FORGEJO_ADMIN_TOKEN Forgejo secret to CI_FORGEJO_ADMIN_TOKEN' (#45) from jfraeys/feat/rotation-ci-sync into main 2aefe56876

Merge pull request 'chore(secrets): P3 ownership and consumer mapping in the manifest' (#37) from jfraeys/chore/p3-manifest-ownership into main 3f9e3746b0

docs(terraform): point comments and the deSEC output at the vault, not .env ... dd814b78ab

.env is retired (all keys live in the SOPS vault). Updates the stale references
in comments, a variable description and the desec_nameservers_reminder output.
Touching terraform/ also exercises terraform-plan's tofu init with the new
backend secrets.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(ci): give terraform-plan its required variables and a diagnosable comment step ... b9391c6eac

Run 268 got past tofu init (backend configured) but then failed on two required
variables with no default (root_pass, user_password) that drift-check already
supplies as placeholders; both only feed ignore_changes attributes and this job
never applies. The PR-comment step now tolerates a missing plan file and prints
the HTTP status (never the token) instead of failing with a bare curl exit 22.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

chore(secrets): fold vault.example.yml into the manifest and remove it ... 6a230ac541

The manifest becomes the single catalog of vault keys: ownership fields are
current (all .env-owned entries now vault-owned, CI replicas recorded, known
token scopes and expiry noted: notably FORGEJO_ADMIN_TOKEN is read:admin +
write:admin only and cannot comment on PRs), and a config: section documents
the non-secret vault keys the example used to describe. vault.example.yml is
removed and every reference updated. .forgejo/ci-secrets.yml no longer maps
CI_FORGEJO_ADMIN_TOKEN to that admin-only vault token, so a --force sync cannot
overwrite the hand-set PR-comment token with one that cannot comment.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(ci): give CI its own scoped Forgejo token (CI_FORGEJO_COMMENT_TOKEN) ... 6cbc3d4e72

The vault's FORGEJO_ADMIN_TOKEN is a read:admin/write:admin service-account
token and gets 403 on issues and PRs, so CI could never comment or open the
promotion PR with it. CI now uses CI_FORGEJO_COMMENT_TOKEN, a repo-scoped
token with its own vault key, mapped in .forgejo/ci-secrets.yml and recorded
in the manifest; the five workflow references are renamed. A read-only test
shows it reads pulls/issue comments (200) and has no user scope (403).
The old hand-set CI_FORGEJO_ADMIN_TOKEN repo secret is now unreferenced.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(ci): do not give terraform-plan placeholder passwords yet ... 20ae1bb8c9

With placeholders the plan of the production workspace runs on with Terraform's
defaults instead of prod's real inputs and prints a false "7 to destroy" (live
Cloudflare records and rulesets) into PR comments. Until CI has prod's inputs,
the job should stop at the missing required variables with a clear error. The
comment-step diagnostics (HTTP status, missing plan file) stay.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'docs(terraform): point comments and outputs at the vault, not .env' (#46) from jfraeys/chore/terraform-env-references into main e00ba137ae

Merge pull request 'chore(secrets): fold vault.example.yml into the manifest and remove it' (#47) from jfraeys/chore/manifest-absorbs-example into main ca7b85e176

feat(terraform): review production from the operator machine; retire the CI jobs ... 58505593cc

CI cannot plan or apply production faithfully: prod's inputs and the deSEC/OVH
credentials live only in the SOPS vault, and a CI plan with Terraform's defaults
wanted to destroy 7 live Cloudflare resources (terraform-plan run 270). Add
scripts/plan-prod.sh (plan + refresh-only drift of the linode workspace,
read-only, exit 0/2/1, optional full output) and remove terraform-plan.yml,
drift-check.yml, terraform-apply.yml and the pre-promotion drift gate in
staging-validate (the promotion PR now says to run the script first). Docs,
manifest used_by and the CI-secrets restore runbook are updated.

First real run against prod: no adds or destroys, 8 in-place DNS updates.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'feat(terraform): review production from the operator machine; retire the CI jobs' (#48) from jfraeys/chore/prod-plan-local into main aa87a03b1b

feat(web): placeholder site for the apex and www, owned by infra until jfraeys_com exists ... 43114c5176

New generic role placeholder_site: a static "coming soon" page and an nginx
vhost for the apex and www (00-placeholder.conf, loaded before the coaching-site
vhost so it wins the duplicate names), validated with nginx -t before reload and
disabled again if nginx rejects it. Reuses the existing apex Let's Encrypt
certificate and skips itself if that is missing. Wired into playbooks/web.yml
under the placeholder_site tag. Dry run against the production web host (check
mode, --diff) shows exactly the expected five changes.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'feat(web): placeholder site for the apex and www, owned by infra' (#49) from jfraeys/feat/placeholder-site into main ac33d4d677

feat(secrets): P6 agent guardrails and secret scanning ... a639550376

Claude deny rules for plaintext secret paths and decrypt commands,
gitleaks (staged pre-commit hook, full-history CI scan with a redacted
baseline of existing findings), and a structural check that tracked
SOPS files are fully encrypted and no .env/tfstate/plaintext vault is tracked.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'feat(secrets): P6 agent guardrails and secret scanning' (#50) from jfraeys/feat/p6-agent-safety into main 8955e9f923

fix(setup): never prompt for a Vaultwarden URL outside a terminal or the legacy vault path ... 62fac4847a

staging-validate hung until the runner deadline at "Vaultwarden URL not configured":
setup called the Bitwarden config prompt unconditionally and `read` blocked on stdin.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(setup): no Vaultwarden prompt in CI' (#53) from jfraeys/fix/setup-no-vaultwarden-prompt into main 69bc60b15d

fix(ci): give staging workflows throwaway node passwords so tofu never prompts ... 4818eb6475

root_pass/user_password have no defaults and used to arrive via the vault (BW_SESSION).
CI has no vault, so `tofu plan` (and destroy) waited on stdin and hung until the step deadline.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

chore(ci): poll the ephemeral runner provisioner every 30s and keep idle VMs warm for 30 min ... f059e88266

Cuts queue latency (up to 90s -> 30s per tick) and lets back-to-back runs reuse a
provisioned VM instead of paying the ~4.5 min cold start. Max VM lifetime goes to
60 min so the idle window is not cut short.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(ci): throwaway node passwords for staging workflows' (#54) from jfraeys/fix/staging-throwaway-passwords into main d6abb3ad6f

Merge pull request 'chore(ci): faster ephemeral runner pickup and warm VM reuse' (#55) from jfraeys/chore/ci-runner-latency into main 7b60abeaa3

fix(ci): wait up to 300s for the dpkg lock on ephemeral runner VMs ... 7277a0f7ab

A fresh VM's unattended-upgrades held the lock past the 120s timeout and failed
"Install UFW" in the base run (cycle 1791736325); matches web/services/hardening.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(ci): 300s dpkg lock wait on ephemeral runner VMs' (#56) from jfraeys/fix/ci-ephemeral-apt-lock into main 237327ad47

fix(ci): supply throwaway provider creds and an SSH key to the staging workflows ... b80c09f365

CI has no vault, so plan failed at OVH/deSEC provider init and at the Linode module's
missing ~/.ssh public key. Shared scripts/lib/ci-staging-env.sh replaces the inline
password generation in both workflows.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(ci): throwaway provider creds and SSH key for staging workflows' (#58) from jfraeys/fix/ci-staging-env into main 9291ed75db

fix(ci): let staging plan with placeholder OVH credentials and an absolute SSH key path ... c8c0898508

The OVH provider validates credentials online, so placeholders returned 403 at init.
ovh_ignore_init_error (default false) is set true only by the CI staging env. The key
path is absolute so the module's file() no longer depends on ~ expansion.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(ci): OVH placeholder creds and absolute SSH key path for staging' (#59) from jfraeys/fix/ci-staging-ovh-ssh into main b929008f91

fix(ci): derive the SSH public key when a reused runner VM has only the private half ... 7656b781de

Run 434 still failed: the runner had a private id_ed25519 with no .pub, so key generation
was skipped and the module's file() found nothing.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(ci): derive the SSH public key on reused runner VMs' (#60) from jfraeys/fix/ci-staging-pubkey into main ca298718f3

fix(ansible): drop vault_password_file from ansible.cfg, ansible-profile.cfg, lint and packer ... 95671b4b53

Staging run 440 reached both validation plays and died on "vault password file secrets/.vault_pass
was not found". No tracked file is Ansible-Vault encrypted any more (vault.yml is a transient SOPS
decrypt), so the setting only breaks runs that lack the file.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(ansible): drop the Ansible Vault password file setting' (#61) from jfraeys/fix/drop-ansible-vault-password-file into main 640c035aba

fix(ci): create ~/.ansible/cp on the staging runner so Ansible's SSH sockets can bind ... 29db4f09b9

Run 444 passed plan/apply/SSH readiness, then both validation plays were UNREACHABLE with
"unix_listener: cannot bind to path ~/.ansible/cp/...: No such file or directory".

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(ci): create ~/.ansible/cp on the staging runner' (#62) from jfraeys/fix/ci-staging-ansible-cp into main 51552c49ce

feat(ci): mint staging tailnet pre-auth keys through Headscale's API ... d462f0ac7d

The ephemeral runner has no SSH route to the services host, so `manage vpn preauthkey` failed
at "Could not resolve hostname services" in staging step 6. With HEADSCALE_API_KEY set it now
uses the REST API (key passed to curl on stdin). HEADSCALE_API_KEY is vault-authoritative,
replicated to Forgejo by ci-secrets.yml and provisioned/rotated by provision-headscale-api-key.sh.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'feat(ci): mint staging pre-auth keys through Headscale's API' (#63) from jfraeys/feat/headscale-api-preauthkey into main 1a7aa9e8b7

feat(ci): staging decrypts its own SOPS file with a staging-only age key ... cf5f60d82f

Run 452 reached the services play and failed at "LLDAP_ADMIN_PASSWORD is required": CI has no vault.
secrets/staging.sops.yml carries the production key set with throwaway values, encrypted to operator,
recovery and a staging-only key (vault key / Forgejo secret STAGING_SOPS_AGE_KEY). setup decrypts it
in CI; the staging key is refused on the production vault. Generator: scripts/gen-staging-secrets.py.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'feat(ci): staging SOPS file with a staging-only age key' (#64) from jfraeys/feat/staging-sops into main 17f9fc75e7

fix(headscale): bootstrap the TLS cert on a fresh node ... 5e17fb5be7

The role extracted the vpn cert from Traefik's acme.json before starting Headscale, but Traefik only
requests it once the Headscale router exists, so a fresh node deadlocked (staging run 457). A short-lived
self-signed placeholder lets Headscale start; the role then waits for the real cert and restarts once.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(forgejo): make the OIDC auth-source task actually retry while the first DB migration runs ... 28b3176fc5

retries/delay without `until` are ignored by Ansible, so the task never waited and failed with
"no such table: login_source" on a fresh node (masked by ignore_errors in staging, fatal in production).
Staging keeps a single attempt: its untrusted ACME CA means the task cannot succeed there anyway.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(headscale): bootstrap the TLS cert on a fresh node' (#65) from jfraeys/fix/headscale-fresh-node-cert into main addfdefc68

fix(ci): staging secrets keep valid formats for config-like keys ... c9b31724ff

Run 463 failed at backups: RESTIC_REPOSITORY had to be an absolute path or URL, got "staging-unused".
Config keys from the manifest keep production values; RESTIC_REPOSITORY is a local path, the secondary
repository stays off and the alert recipient is example.invalid. Empty strings stay empty.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(ci): staging secrets keep valid formats for config-like keys' (#66) from jfraeys/fix/staging-secrets-formats into main 0b588aed84

fix(crowdsec): wait for the container before re-verifying the bouncer ... 5bf9a20f2a

Run 467 failed with "No such container: crowdsec-crowdsec-1" immediately after flush_handlers, which runs
every pending handler on the host and can restart the container. Retry up to 12 x 5s.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(crowdsec): wait for the container before re-verifying the bouncer' (#67) from jfraeys/fix/crowdsec-verify-retry into main 976e0261af

fix(ansible): make apt itself wait for the dpkg lock ... 1f5a496835

module_defaults lock_timeout only covers python-apt's cache lock; the apt-get it spawns failed at once on
unattended-upgrades' lock (staging run 471, backups: Install restic). A pre-task drops an apt.conf.d file with
DPkg::Lock::Timeout 600 in services, web, hardening and ci-ephemeral.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(ansible): make apt itself wait for the dpkg lock' (#68) from jfraeys/fix/apt-dpkg-lock-timeout into main 96002059a0

chore(ci): dump Docker/systemd state from the staging node when a deploy fails ... 6bfb4c8629

Run 475 lost the crowdsec container after a Docker restart handler and the node was destroyed before anything
could be inspected. Print docker ps -a, docker info highlights, unit state and the docker journal on failure.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'chore(ci): dump Docker/systemd state from the staging node when a deploy fails' (#69) from jfraeys/chore/staging-failure-diagnostics into main 2d1a3db79b

fix(ci): run the node diagnostics as a script file ... b2e767cf85

The runner's ansible-core 2.14 cannot parse a quoted multi-line -a string ("Unable to parse argument string"),
so the failure dump in run 479 never printed. Use ansible.builtin.script with a temp file instead.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(ci): run the node diagnostics as a script file (ansible-core 2.14 can't parse the multi-line -a)' (#70) from jfraeys/fix/staging-diagnostics-script into main a3e25e9998

fix(ansible): wait for background apt/dpkg to finish before the play ... 8f3782c8d8

Run 487 failed at hardening: Install auditd on /var/cache/apt/archives/lock, which DPkg::Lock::Timeout does
not cover. The old flock-based wait in services.yml could not see apt's fcntl locks. Replace it with a
process-name wait, and add it to web, hardening and ci-ephemeral.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(ansible): wait for background apt/dpkg to finish before the play (covers the archives lock)' (#71) from jfraeys/fix/apt-quiet-before-play into main a211688f57

fix(ansible): wait on apt's real locks, not process names ... cf40b4c0cf

#71's name-based wait always saw the permanent unattended-upgrade-shutdown daemon and timed out after 15 min
(staging run 491). Test the fcntl locks apt takes instead, via playbooks/files/apt-quiet.py.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(ansible): wait on apt's real locks, not process names' (#72) from jfraeys/fix/apt-quiet-lock-test into main f0c6e3f764

fix(docker): restart on the daemon.json change before any container exists ... bcfd240b64

Docker's containerd-snapshotter default vs the pinned overlay2 storage-driver: a deferred restart handler switched
backends after containers existed and hid them all (staging run 495). Flush handlers right after the daemon.json
task so the restart happens first. Incident recorded.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(docker): restart on the daemon.json change before any container exists' (#73) from jfraeys/fix/docker-restart-before-containers into main ac0296e612

fix(ci): staging parity: plain prod image, explicit role skip, documented differences ... b148fa0634

Skip the CI-provisioner roles explicitly (they cannot run on an isolated node), boot staging from the same
plain image as production (use_prebaked_image=false in the validation env), and document every known
difference in docs/staging-parity.md.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(ci): staging parity: plain prod image, explicit role skip, documented differences' (#74) from jfraeys/fix/staging-skip-ci-provisioner into main c16bfaa480

fix(ansible): load vault vars under always so --skip-tags cannot drop them ... f9e4fe8e2e

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(ansible): load vault vars under always so --skip-tags cannot drop them' (#75) from jfraeys/fix/vault-vars-always-load into main bd4aa0aebc

jfraeys added 3 commits 2026-10-12 21:11:43 -04:00

fix(ci): staging deploys playbooks/web.yml even without an app repo ... 846985476c

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(ci): cross-host connectivity check after staging deploys ... 3967eaeb7c

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(ci): staging deploys playbooks/web.yml even without an app repo' (#76) from jfraeys/fix/staging-deploy-web-base-layer into main 6480f5c5d3

jfraeys added 3 commits 2026-10-12 21:45:17 -04:00

fix(firewall): LLDAP admin port is tailnet-only (Docker-published ports bypass UFW) ... 0cf03c6ae1

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

docs(incidents): do not claim LLDAP logs were reviewed ... 9cab7903ef

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(firewall): LLDAP admin port is tailnet-only' (#78) from jfraeys/fix/lldap-admin-tailnet-only into main ... 254ce5d63b

Reviewed-on: #78

jfraeys added 2 commits 2026-10-12 22:24:17 -04:00

fix(ci): provisioner liveness check ignores host keys of recycled VM IPs ... 3f1765f8e4

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(ci): provisioner liveness check ignores host keys of recycled VM IPs' (#79) from jfraeys/fix/provisioner-recycled-ip-hostkey into main d82c1ba254

jfraeys added 2 commits 2026-10-12 22:30:59 -04:00

fix(deploy): services deploy defaults to the production inventory only ... df754a56ea

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(deploy): services deploy defaults to the production inventory only' (#80) from jfraeys/fix/deploy-services-limit-doc into main ff61299a88

jfraeys added 2 commits 2026-10-12 22:32:41 -04:00

fix(ansible): default inventory is production only ... bc6853230a

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(ansible): default inventory is production only' (#81) from jfraeys/fix/inventory-default-production-only into main 305bbd51e7

jfraeys added 2 commits 2026-10-12 23:01:23 -04:00

fix(ci): staging runs on role/playbook/inventory changes; promotion PR names the staged commit ... 471fe0fff2

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(ci): staging runs on deploy-affecting changes; promotion PR names the staged commit' (#82) from jfraeys/fix/staging-trigger-paths into main 53dd78dc27

jfraeys referenced this pull request 2026-10-12 23:12:26 -04:00

chore(cloudflare): remove the ineffective vpn security_level Page Rule #89

jfraeys added 2 commits 2026-10-12 23:14:46 -04:00

fix(firewall): Postfix 25 is tailnet-only; staging audits published ports ... fd6a0bf6ec

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(firewall): Postfix 25 is tailnet-only; staging audits all published ports' (#83) from jfraeys/fix/tailnet-only-ports-and-open-port-check into main 019f5c3007

jfraeys added 2 commits 2026-10-12 23:14:55 -04:00

fix(ssh): operator scripts tell a changed host key apart from missing data ... 5a350a4f01

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(ssh): operator scripts tell a changed host key apart from missing data' (#84) from jfraeys/fix/operator-ssh-hostkey-recycled-ip into main 0a00c17a96

jfraeys added 2 commits 2026-10-12 23:15:06 -04:00

feat(ci): alert when CI jobs wait too long or the provisioner stops running ... 6e606e9036

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'feat(ci): alert when CI jobs wait too long or the provisioner stops running' (#85) from jfraeys/feat/provisioner-queue-stall-alert into main faa798c50b

jfraeys added 2 commits 2026-10-12 23:15:16 -04:00

feat(ci): run the production-only hardening checks after the staging deploy ... d854fea9ae

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'feat(ci): run the production-only hardening checks after the staging deploy' (#86) from jfraeys/fix/post-deploy-hardening-checks into main fbba703900

jfraeys added 2 commits 2026-10-12 23:33:27 -04:00

feat(ci): staging prints per-phase timings and the slowest Ansible tasks ... c1391c52d7

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'feat(ci): staging prints per-phase timings and the slowest Ansible tasks' (#87) from jfraeys/feat/staging-timing-profile into main a82502aaf8

jfraeys added 2 commits 2026-10-12 23:33:37 -04:00

fix(dns): vpn is DNS-only in code, matching live; the proxied record would break Headscale on 8443 ... 6119b30b82

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(dns): vpn is DNS-only in code, matching live' (#88) from jfraeys/fix/vpn-dns-only into main 73c06bfa7a

jfraeys added 2 commits 2026-10-12 23:33:46 -04:00

chore(cloudflare): remove the ineffective vpn security_level Page Rule ... e95f268aa8

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'chore(cloudflare): remove the ineffective vpn security_level Page Rule' (#89) from jfraeys/fix/remove-vpn-page-rule into main e263b27737

jfraeys added 2 commits 2026-10-12 23:52:19 -04:00

fix(dns): remove the unused mail record (returns Cloudflare 526) ... c95a61b603

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(dns): remove the unused mail.jfraeys.com record (Cloudflare 526)' (#90) from jfraeys/fix/remove-unused-mail-record into main fd9ef851ac

jfraeys added 2 commits 2026-10-13 08:50:34 -04:00

fix(ci): the published-port audit allows Headscale's direct control port ... 133210b66e

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(ci): published-port audit allows Headscale's direct control port (8443)' (#91) from jfraeys/fix/audit-allow-headscale-direct-port into main 14a18488e8

jfraeys added 2 commits 2026-10-13 09:25:14 -04:00

fix(ci): queue-age metric resets when Forgejo answers an empty queue with null ... eedede5beb

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(ci): queue-age metric resets when the queue drains (Forgejo returns null)' (#92) from jfraeys/fix/queue-metrics-null-queue into main d3437b2f1e

jfraeys added 2 commits 2026-10-13 09:47:33 -04:00

docs(secrets): remediation register and gate status; exposure fields in manifest ... 2cc23a6db8

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'docs(secrets): remediation register and gate status' (#51) from jfraeys/chore/p6-close-r-track into main d6be449370

jfraeys added 2 commits 2026-10-13 09:48:06 -04:00

fix(molecule-test): refuse a system python3 newer than ansible-core 2.19 supports ... 4dbe5ec880

When neither uv nor python<.python-version> exists, the script fell back to the system
python3, which is 3.14 on Homebrew machines and silently produced an unsupported venv. Error
out above 3.13; the Debian 12 runner's 3.11 still falls back with a warning.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(molecule-test): refuse a system python3 newer than 3.13' (#15) from fix/molecule-refuse-python314 into main b6c0322774

jfraeys added 3 commits 2026-10-13 09:50:58 -04:00

fix(secrets): rotate exposed CI SSH key and Authelia OIDC key; SOPS-aware key regen script ... fb21518f45

Both values were printed in full during a redaction incident (docs/incidents.md).
Vault holds fresh keys; production redeploy pending.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge main into p2-rotate-exposed; re-apply the two rotated keys onto main's vault ... 29f95d8e42

Resolved by taking main's secrets/vault.sops.yml and re-applying
AUTHELIA_OIDC_PRIVATE_KEY_PEM and CI_EPHEMERAL_ANSIBLE_SSH_KEY from this branch
(verified: every other key equals main's, same key set). The regenerate script keeps
this branch's rewrite with main's --limit services hint.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(secrets): rotate exposed CI SSH key and Authelia OIDC key' (#52) from jfraeys/fix/p2-rotate-exposed into main 5c338ba9e8

jfraeys added 2 commits 2026-10-13 10:20:32 -04:00

fix(secrets): load_env fails closed on a decrypt error; plan-prod never prompts ... 95a75dfcb3

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(secrets): load_env fails closed on a decrypt error; plan-prod never prompts' (#93) from jfraeys/fix/load-env-fail-closed into main d41fc4ad18

jfraeys added 2 commits 2026-10-13 10:23:44 -04:00

feat(ci): LLDAP admin page must answer over the tailnet; add scripts/check-exposure.py for production ... b737c6ae32

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'feat(ci): LLDAP admin page must answer over the tailnet; add an operator exposure check for production' (#94) from jfraeys/feat/exposure-check into main edb605add2

jfraeys referenced this pull request 2026-10-13 10:55:31 -04:00

feat(deploy): production gate tied to the promotion PR (replaces the staging marker files) #96

jfraeys added 2 commits 2026-10-13 10:59:07 -04:00

docs(secrets): close P2, record the rotation and the two gitleaks dispositions ... 6d71386307

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'docs(secrets): close P2, record the rotation and the two gitleaks dispositions' (#95) from jfraeys/docs/close-p2-r-register into main 36c6490bb6

jfraeys added 4 commits 2026-10-13 11:16:21 -04:00

feat(deploy): production gate: deploy only the commit staging validated and the promotion PR approved ... f56ddaa4da

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

ci(lint): run the promotion gate tests ... 5ceb5d56ef

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(deploy): the gate accepts a merged descendant that changed only non-deploy files ... 6236ebd79e

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'feat(deploy): production gate tied to the promotion PR (replaces the staging marker files)' (#96) from jfraeys/feat/promotion-gate into main 1e9a89596b

jfraeys added 2 commits 2026-10-13 11:18:44 -04:00

docs+chore: describe the real deploy flow (CI validates, you deploy) and remove the staging marker mechanism ... a401c26bc1

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'docs+chore: describe the real deploy flow and remove the staging marker mechanism (stacked on #96)' (#97) from jfraeys/chore/gitops-docs-cleanup into main 6c24d41e44

jfraeys added 2 commits 2026-10-13 13:03:09 -04:00

feat(secrets): one expiry classifier and a manifest-only expiry report ... 44ba182f43

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'feat(secrets): one expiry classifier and a manifest-only expiry report (P8, 1/3)' (#98) from jfraeys/feat/secrets-expiry-report into main caa9b6d26d

jfraeys added 2 commits 2026-10-13 13:03:21 -04:00

feat(secrets): rotation lock for vault read-modify-write cycles ... b35100b7dc

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'feat(secrets): rotation lock for vault read-modify-write cycles (P8, 2/3; stacked on #98)' (#99) from jfraeys/feat/rotation-lock into main d42233a0aa

jfraeys added 2 commits 2026-10-13 13:03:33 -04:00

feat(secrets): monthly expiry report workflow and the manifest facts the repo knows ... d05d8e0b96

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'feat(secrets): monthly expiry report workflow and the manifest facts the repo knows (P8, 3/3; stacked on #99)' (#100) from jfraeys/feat/secrets-expiry-workflow into main 71b9deb713

jfraeys added 3 commits 2026-10-13 13:03:42 -04:00

feat(ops): tool version registry, monthly upstream survey, and the update/security runbook ... 86986b754f

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

ops: tool versions report runs weekly (Mondays), one issue per month refreshed each week ... 1e4d538f6d

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'feat(ops): tool version registry, monthly upstream survey, and the update/security runbook (stacked on #100)' (#105) from jfraeys/feat/tool-version-procedure into main 01b30c782c

jfraeys added 3 commits 2026-10-13 13:03:56 -04:00

feat(ops): secret expiry and tool version drift as Prometheus metrics and alerts ... 168d0f4b02

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(ops): hourly check for unexpectedly public ports, alerting through Prometheus ... 253d2df8e4

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'feat(ops): alert when a port is published to the internet that nothing allows (stacked on #106)' (#107) from jfraeys/feat/exposure-drift-alert into main 27357d42d1

jfraeys added 2 commits 2026-10-13 13:04:04 -04:00

fix(exporters): trim cAdvisor to the 1 GB services host, and check PostfixDown's series survives ... 8cbf03f763

cAdvisor used ~93 MiB at defaults. Only container_last_seen{name=...} is consumed, so:
docker-only, 30s housekeeping, no container labels/events, cpu/memory/network/oom metrics.
Staging now queries Prometheus for container_last_seen{name="postfix-postfix-1"}.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(exporters): trim cAdvisor memory on the 1 GB services host' (#108) from jfraeys/fix/cadvisor-memory-footprint into main 8c17997806

jfraeys added 2 commits 2026-10-13 13:27:47 -04:00

ci(staging): skip a push run once main has moved on, and keep promote from reporting it ... e6b5390ec7

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'ci(staging): skip superseded push runs instead of validating every commit in a burst' (#110) from jfraeys/ci/skip-superseded-staging into main 6f50a7b7da

jfraeys added 3 commits 2026-10-13 13:52:31 -04:00

docs+test(secrets): P7 per-project SOPS convention, contract section, key-isolation test ... e1b61bd68c

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

docs(secrets): link the P7 PRs ... c67dbf94d7

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'docs+test(secrets): P7 per-project SOPS convention and key-isolation test' (#109) from jfraeys/docs/p7-project-secrets into main ... 7e7f6eba9e

Reviewed-on: #109

jfraeys added 2 commits 2026-10-13 13:53:34 -04:00

fix(ci): a task Forgejo no longer lists no longer stalls the runner cycle until its deadline ... 9cf2de1f71

get_task_status read only the newest 50 tasks and returned None for both an API error and a missing task, so a cycle
whose job vanished (e.g. a run cancelled mid-job) waited ~55 min. It now pages the listing, distinguishes 'not listed'
from an error, and asks the VM whether the one-job runner is still alive.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(ci): a task Forgejo no longer lists no longer stalls the runner cycle' (#111) from jfraeys/fix/provisioner-missing-task into main ... bb1cf2fba1

Reviewed-on: #111

jfraeys added 2 commits 2026-10-13 13:56:10 -04:00

fix(ci): staging-reboot-check runs on ephemeral-large (no self-hosted runner exists) ... aeb92323b4

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(ci): staging-reboot-check runs on ephemeral-large' (#113) from jfraeys/fix/reboot-check-runner into main 76d65dc410

jfraeys referenced this pull request 2026-10-13 14:25:41 -04:00

Next: immediate steps (merge, deploy, verify) after the 2026-10-13 session #116

jfraeys referenced this pull request 2026-10-13 14:25:44 -04:00

Infra backlog: capacity, CI verification, alerts, older loose ends #121

jfraeys added 2 commits 2026-10-13 14:30:33 -04:00

fix(ci): skip a staging run only when a newer commit that starts staging exists ... 32b00178a4

A later commit touching no staged path (docs, a manual workflow) started no run but still marked the earlier run stale:
#542 skipped in 5s, so nothing validated the tree and promote was skipped too.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(ci): skip a staging run only when a newer commit that starts staging exists' (#123) from jfraeys/fix/skip-only-if-restaged into main f4c842c42c

jfraeys added 3 commits 2026-10-13 14:30:48 -04:00

fix(rotate-secret playbook): pass --force after its own confirmation (the script's prompt has no terminal) ... 2daa4ad7a2

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

fix(rotate-secret playbook): refresh the plaintext vault with env bash (macOS /bin/bash 3.2 has no local -n) ... 43fde2a063

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(rotate-secret playbook): the script's second confirmation prompt failed the run' (#115) from jfraeys/fix/rotate-playbook-force into main 565953a78a

jfraeys added 2 commits 2026-10-13 14:31:01 -04:00

fix(rotate-secret): --save-to-bitwarden stored the description instead of the secret, never verified, and the playbook hint skipped with-vault.sh ... b5936be2bb

jq chained '.notes = $content | .notes = $notes', so the secret was overwritten; bw create/edit results were discarded
and success was always logged. bw_store_verified (scripts/lib/bitwarden.sh) stores the secret in the note, syncs, reads it
back and compares, failing loudly otherwise. Services without a Bitwarden item now say so. The replay hint carries the
with-vault.sh prefix the playbook's guard requires.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'fix(rotate-secret): verify --save-to-bitwarden, and store the secret rather than the description' (#114) from jfraeys/fix/rotate-secret-bitwarden into main ed1c6bdca9

jfraeys added 2 commits 2026-10-13 14:31:19 -04:00

chore(deps): patch updates: Forgejo 16.0.6, Vaultwarden 1.37.4, Headscale 0.29.5, Authelia 4.39.28, CrowdSec 1.8.1 ... 6dc380d177

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'chore(deps): patch updates: Forgejo 16.0.6, Vaultwarden 1.37.4, Headscale 0.29.5, Authelia 4.39.28, CrowdSec 1.8.1' (#103) from jfraeys/chore/deps-patch-batch-1 into main 682b72abc0

jfraeys added 2 commits 2026-10-13 14:31:32 -04:00

chore(deps): OpenTofu 1.12.6 to 1.12.7 (Go security advisories) in the provisioner and runner roles ... 4cb680e5f3

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'chore(deps): OpenTofu 1.12.6 to 1.12.7 in the provisioner and runner roles' (#104) from jfraeys/chore/deps-opentofu-1.12.7 into main 7dcc38b65c

jfraeys added 2 commits 2026-10-13 14:56:41 -04:00

feat(plan-prod): unlock mode for a stuck state lock, and show the holder in the lock error ... 886b46405e

plan-prod.sh unlock <ID> re-reads the lock, refuses if it is not exactly that ID, refuses while a local tofu runs when the
holder is this machine, and requires retyping the ID. A lock error now prints Lock Info (Who, Created) instead of cutting it off.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'feat(plan-prod): unlock mode for a stuck state lock' (#124) from jfraeys/feat/plan-prod-unlock into main fd26dc385e

jfraeys referenced this pull request 2026-10-13 15:08:50 -04:00

Roadmap: remaining infra and secrets work (tracker) #122

jfraeys added 2 commits 2026-10-13 15:25:09 -04:00

chore(secrets): rotated Vaultwarden and Authelia secrets (2026-10-13), re-applied onto main's vault ... ed9109fd12

Applies the 8 keys rotated on the operator machine on top of main (which already holds the #52 rotation of
AUTHELIA_OIDC_PRIVATE_KEY_PEM and CI_EPHEMERAL_ANSIBLE_SSH_KEY); main's PEM is kept, it is the deployed one.
Not deployed yet: Vaultwarden and Authelia need a deploy before these values are live.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Merge pull request 'chore(secrets): rotated Vaultwarden and Authelia secrets (2026-10-13)' (#125) from jfraeys/chore/rotated-secrets-2026-10-13 into main 221396dc2c

jfraeys merged commit c3ef211b68 into prod 2026-10-13 15:34:33 -04:00

jfraeys referenced this pull request from a commit 2026-10-13 15:34:34 -04:00

Merge pull request 'Promote to production' (#77) from main into prod

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

jfraeys/infra!77

No description provided.