P8: close rotation hygiene (rotation drill, manifest triage) #117

Open
opened 2026-10-13 14:25:41 -04:00 by jfraeys · 5 comments
Owner

Gate: "Expiry reporting and rotation locking work". Status 2026-10-13: code merged and report verified; the real rotation drill is half done.

Done

  • Code merged: #98 classifier + report, #99 rotation lock, #100 monthly workflow, #114 verified Bitwarden save, #115 rotate-secret playbook fixes.
  • Report workflow verified live: two workflow_dispatch runs (#536, #537) produced one issue (#112), created then updated on rerun.

Open

  • Finish the real rotation drill. Vaultwarden and a set of Authelia secrets were rotated on 2026-10-13; the values are merged (#125) and DEPLOYED (group deploy, #116). Forgejo SSO login verified after the deploy. Verified after the deploy: Forgejo SSO login and Vaultwarden SSO login (Grafana is disabled in this inventory, so not applicable). Remaining before ticking this: the new admin token at /admin (tracked in #116), then the Bitwarden re-save and the live lock test below. The drill also found two real defects, both fixed: the playbook's second confirmation prompt (#115) and /bin/bash 3.2 lacking local -n (#115).
  • Confirm the Bitwarden copy: after #114, ./scripts/rotate-secret.sh vaultwarden --save-to-bitwarden must print Saved and verified ..., and the item's note body must be the token.
  • Verify the rotation lock live: run two rotate-secret.sh at once; the second must refuse with the owner's details.
  • Triage the 48 unknown entries in secrets/secrets-manifest.yml (listed in #112). Most have a rotate_every_days schedule but no rotated_at; fill only what is true. Today's rotations give real rotated_at: 2026-10-13 dates for the keys in #125.
  • Reconcile AUTHELIA_OIDC_PRIVATE_KEY_PEM: manifest says rotated 2026-10-10, docs/secrets-remediation.md says 2026-10-13 (PR #52).
  • Confirm a failed monthly run (expired/overdue) is visible to you.
  • Set the P8 row in docs/secrets-remediation.md to Met with the evidence (#536/#537, #112, the drill).
Gate: "Expiry reporting and rotation locking work". **Status 2026-10-13: code merged and report verified; the real rotation drill is half done.** **Done** - [x] Code merged: #98 classifier + report, #99 rotation lock, #100 monthly workflow, #114 verified Bitwarden save, #115 rotate-secret playbook fixes. - [x] Report workflow verified live: two `workflow_dispatch` runs (#536, #537) produced one issue (#112), created then updated on rerun. **Open** - [ ] Finish the real rotation drill. Vaultwarden and a set of Authelia secrets were rotated on 2026-10-13; the values are merged (#125) and DEPLOYED (group deploy, #116). Forgejo SSO login verified after the deploy. Verified after the deploy: Forgejo SSO login and Vaultwarden SSO login (Grafana is disabled in this inventory, so not applicable). Remaining before ticking this: the new admin token at `/admin` (tracked in #116), then the Bitwarden re-save and the live lock test below. The drill also found two real defects, both fixed: the playbook's second confirmation prompt (#115) and `/bin/bash` 3.2 lacking `local -n` (#115). - [ ] Confirm the Bitwarden copy: after #114, `./scripts/rotate-secret.sh vaultwarden --save-to-bitwarden` must print `Saved and verified ...`, and the item's note body must be the token. - [ ] Verify the rotation lock live: run two `rotate-secret.sh` at once; the second must refuse with the owner's details. - [ ] Triage the 48 `unknown` entries in `secrets/secrets-manifest.yml` (listed in #112). Most have a `rotate_every_days` schedule but no `rotated_at`; fill only what is true. Today's rotations give real `rotated_at: 2026-10-13` dates for the keys in #125. - [ ] Reconcile `AUTHELIA_OIDC_PRIVATE_KEY_PEM`: manifest says rotated 2026-10-10, `docs/secrets-remediation.md` says 2026-10-13 (PR #52). - [ ] Confirm a failed monthly run (expired/overdue) is visible to you. - [ ] Set the P8 row in `docs/secrets-remediation.md` to Met with the evidence (#536/#537, #112, the drill).
Author
Owner

P8 update 2026-10-14: #143 merged. The manifest now records rotated_at: 2026-10-13 for the 8 keys in #125 plus POSTFIX_SENDER_ALERTS_APIKEY (#133); expiry report unknown 48 -> 44, ok 13 -> 18. The AUTHELIA_OIDC_PRIVATE_KEY_PEM reconcile item is resolved: manifest 2026-10-10 is when the key was generated, docs' 2026-10-13 was the PR #52 merge and redeploy, not a conflict (remediation row reworded).

Not done, deliberately: the other 44 unknown entries have no rotation evidence, so I did not invent dates; the ones with no rotate_every_days / expiry need a policy decision from you (not a recorded fact). Still open: Bitwarden re-save, /admin token check, the live two-at-once rotation-lock test (it rotates real secrets, so it is your call), confirm a failed monthly run is visible to you, set the P8 row to Met.

**P8 update 2026-10-14:** #143 merged. The manifest now records `rotated_at: 2026-10-13` for the 8 keys in #125 plus `POSTFIX_SENDER_ALERTS_APIKEY` (#133); expiry report unknown 48 -> 44, ok 13 -> 18. The `AUTHELIA_OIDC_PRIVATE_KEY_PEM` reconcile item is resolved: manifest 2026-10-10 is when the key was generated, docs' 2026-10-13 was the PR #52 merge and redeploy, not a conflict (remediation row reworded). Not done, deliberately: the other 44 unknown entries have no rotation evidence, so I did not invent dates; the ones with no `rotate_every_days` / expiry need a policy decision from you (not a recorded fact). Still open: Bitwarden re-save, /admin token check, the live two-at-once rotation-lock test (it rotates real secrets, so it is your call), confirm a failed monthly run is visible to you, set the P8 row to Met.
Author
Owner

P8 update 2026-10-14: the rotation drill's /admin check now passes: the new Vaultwarden admin token works (after #157 fixed the compose $ escaping that had mangled every hashed token; see #116). The Bitwarden copy holds the bare token (the *_PLAIN= prefix is gone from the script). Remaining for P8: the live two-at-once rotation-lock test, confirming a failed monthly run is visible to you, the policy decision for manifest entries without a schedule, then the P8 row to Met. Also new since the last update: rotations open their own chore(secrets) PR and set the manifest's rotated_at (#151), so the manual manifest step from #143 is automated.

**P8 update 2026-10-14:** the rotation drill's /admin check now passes: the new Vaultwarden admin token works (after #157 fixed the compose `$` escaping that had mangled every hashed token; see #116). The Bitwarden copy holds the bare token (the `*_PLAIN=` prefix is gone from the script). Remaining for P8: the live two-at-once rotation-lock test, confirming a failed monthly run is visible to you, the policy decision for manifest entries without a schedule, then the P8 row to Met. Also new since the last update: rotations open their own `chore(secrets)` PR and set the manifest's `rotated_at` (#151), so the manual manifest step from #143 is automated.
Author
Owner

P8 decisions, 2026-10-14 (operator: "go with the recommendations")

  • Live rotation-lock test: passed. Held the real lock (rotation_lock.py hold), then ran a real rotate-secret.sh lldap: it refused at once, exit 3, another rotation is running: pid … on Jays-Mac-Studio.local … lock drill, before touching the vault; lock free afterwards, vault unchanged.
  • Failed monthly run is visible: accepted via the Prometheus alerts. SecretExpired (critical) and SecretRotationOverdue (warning) fire from live metrics (14 series on services) through Alertmanager to phone push and email (delivery verified earlier); rules are tested in CI. The workflow's own failure in Actions stays as a second signal.
  • Policy for unscheduled secrets: #159. 18 access credentials scheduled (180 d; OVH access key 90 d), 9 exposure-only with the reason recorded (secrets that encrypt or sign, plus heartbeat URLs / repository location). Unknown 44 -> 32. #159 also makes ./infra rotate lldap safe (it rotated LLDAP_KEY_SEED, which would lock everyone out, and a rotated admin password never reached LLDAP).

Rotation order for the 32 with a schedule but no date (each rotation now records its date via the PR it opens, #151; deploy what the output names, together):

Wave 1, one command each:

  1. ./infra rotate forgejo (FORGEJO_RUNNER_REGISTRATION_TOKEN, FORGEJO_API_TOKEN, FORGEJO_METRICS_TOKEN). Check the printed deploy tags include prometheus (it scrapes with the metrics token).
  2. ./infra rotate headscale (HEADSCALE_OIDC_CLIENT_SECRET + AUTHELIA_OIDC_HEADSCALE_CLIENT_SECRET).
  3. ./infra rotate ci-ephemeral-wakeup, then the same value in Forgejo's webhook settings.
  4. ./infra rotate lldap (admin password, JWT secret), only after #159 is deployed.

Wave 2, provider UI then scripts/secrets/vault-set.py set KEY and scripts/secrets/vault-pr.py --what <label> --key KEY, deploy, revoke the old value:

  • Zedmail: POSTFIX_SENDER_BOOKING/CONTACT/NOREPLY_APIKEY, ZEDMAIL_API_KEY, AUTHELIA_SMTP_USERNAME/PASSWORD.
  • Forgejo UI: FORGEJO_ADMIN_TOKEN, CI_EPHEMERAL_FORGEJO_REPO_TOKEN, VAULT_DEPLOY_TOKEN, FORGEJO_RUNNER_APP_SKELETON_DEPLOY_KEY.
  • Providers (Terraform/CI only, then ./infra secrets sync): TF_VAR_cloudflare_api_token, TF_VAR_desec_api_token, TF_VAR_linode_token, OS_PASSWORD, OVH_APPLICATION_SECRET.
  • LLDAP UI: AUTHELIA_LDAP_SERVICE_PASSWORD (the authelia-service account), deploy authelia.
  • Alert endpoints: ALERTMANAGER_DISCORD/SLACK_WEBHOOK_URL, ALERTMANAGER_NTFY_TOPIC (365 d; resubscribe the phone).

Wave 3, needs a decision first:

  • GRAFANA_ADMIN_PASSWORD: Grafana is disabled in this inventory; remove the key instead of rotating it?
  • TF_VAR_root_pass / TF_VAR_user_password: only used when a VM is created; changing the vault does not change existing hosts. Rotate on the hosts too, or treat as "next rebuild"?
  • POSTGRES_PASSWORD: needs an ALTER USER in the database plus a coordinated redeploy of its consumers.
  • STAGING_SOPS_AGE_KEY: follow its documented procedure (new key, recipient in .sops.yaml, re-encrypt staging, CI secret).
**P8 decisions, 2026-10-14 (operator: "go with the recommendations")** - [x] **Live rotation-lock test: passed.** Held the real lock (`rotation_lock.py hold`), then ran a real `rotate-secret.sh lldap`: it refused at once, exit 3, `another rotation is running: pid … on Jays-Mac-Studio.local … lock drill`, before touching the vault; lock free afterwards, vault unchanged. - [x] **Failed monthly run is visible: accepted via the Prometheus alerts.** `SecretExpired` (critical) and `SecretRotationOverdue` (warning) fire from live metrics (14 series on services) through Alertmanager to phone push and email (delivery verified earlier); rules are tested in CI. The workflow's own failure in Actions stays as a second signal. - [x] **Policy for unscheduled secrets: #159.** 18 access credentials scheduled (180 d; OVH access key 90 d), 9 exposure-only with the reason recorded (secrets that encrypt or sign, plus heartbeat URLs / repository location). Unknown 44 -> 32. #159 also makes `./infra rotate lldap` safe (it rotated LLDAP_KEY_SEED, which would lock everyone out, and a rotated admin password never reached LLDAP). **Rotation order for the 32 with a schedule but no date** (each rotation now records its date via the PR it opens, #151; deploy what the output names, together): Wave 1, one command each: 1. `./infra rotate forgejo` (FORGEJO_RUNNER_REGISTRATION_TOKEN, FORGEJO_API_TOKEN, FORGEJO_METRICS_TOKEN). Check the printed deploy tags include `prometheus` (it scrapes with the metrics token). 2. `./infra rotate headscale` (HEADSCALE_OIDC_CLIENT_SECRET + AUTHELIA_OIDC_HEADSCALE_CLIENT_SECRET). 3. `./infra rotate ci-ephemeral-wakeup`, then the same value in Forgejo's webhook settings. 4. `./infra rotate lldap` (admin password, JWT secret), only after #159 is deployed. Wave 2, provider UI then `scripts/secrets/vault-set.py set KEY` and `scripts/secrets/vault-pr.py --what <label> --key KEY`, deploy, revoke the old value: - Zedmail: POSTFIX_SENDER_BOOKING/CONTACT/NOREPLY_APIKEY, ZEDMAIL_API_KEY, AUTHELIA_SMTP_USERNAME/PASSWORD. - Forgejo UI: FORGEJO_ADMIN_TOKEN, CI_EPHEMERAL_FORGEJO_REPO_TOKEN, VAULT_DEPLOY_TOKEN, FORGEJO_RUNNER_APP_SKELETON_DEPLOY_KEY. - Providers (Terraform/CI only, then `./infra secrets sync`): TF_VAR_cloudflare_api_token, TF_VAR_desec_api_token, TF_VAR_linode_token, OS_PASSWORD, OVH_APPLICATION_SECRET. - LLDAP UI: AUTHELIA_LDAP_SERVICE_PASSWORD (the `authelia-service` account), deploy authelia. - Alert endpoints: ALERTMANAGER_DISCORD/SLACK_WEBHOOK_URL, ALERTMANAGER_NTFY_TOPIC (365 d; resubscribe the phone). Wave 3, needs a decision first: - GRAFANA_ADMIN_PASSWORD: Grafana is disabled in this inventory; remove the key instead of rotating it? - TF_VAR_root_pass / TF_VAR_user_password: only used when a VM is created; changing the vault does not change existing hosts. Rotate on the hosts too, or treat as "next rebuild"? - POSTGRES_PASSWORD: needs an `ALTER USER` in the database plus a coordinated redeploy of its consumers. - STAGING_SOPS_AGE_KEY: follow its documented procedure (new key, recipient in .sops.yaml, re-encrypt staging, CI secret).
Author
Owner

Rotation wave 1 done, 2026-10-15 (all rotated, merged, approved, deployed, verified):

  • forgejo (FORGEJO_METRICS_TOKEN, #168; deployed with prometheus, scrape up)
  • headscale (OIDC client secret on both sides, #170/#171; deployed headscale,authelia; Forgejo Authelia login verified)
  • ci-ephemeral-wakeup (#172/#176; receiver verified on the new secret; Forgejo webhook updated and verified: test delivery 202)
  • lldap (admin password + JWT secret, #177/#178; container restarted healthy; Authelia login verified)

Found on the way, all fixed: vault-pr commits failed in its temporary worktree (hook needs .venv) and the deploy gate now refuses an uncommitted vault (#167); ./infra approve retries a mistyped confirmation (#174); ./infra secrets get KEY copies one value to the clipboard without printing it (#175). Webhook: a leftover user-level Forgejo hook (created 2026-09-23) still signed with the old secret and produced a 401 beside every 202; deleted. Editing an existing hook's secret did not take effect (twice); a freshly created hook did.

Wave 2: provider-issued credentials (24 unknown in the report minus the 5 decisions below). Procedure for each: issue the new value at the provider, scripts/secrets/vault-set.py set KEY (stdin), scripts/secrets/vault-pr.py --what <label> --key KEY, merge, ./infra approve, git pull, deploy the role, verify, revoke the old value.

  • Cloud/DNS: TF_VAR_linode_token, TF_VAR_cloudflare_api_token, TF_VAR_desec_api_token
  • OVH: OVH_APPLICATION_SECRET, OS_PASSWORD (OVH access key ID is 90d, already rotated via ./infra rotate ovh-s3)
  • Forgejo tokens: FORGEJO_ADMIN_TOKEN, CI_EPHEMERAL_FORGEJO_REPO_TOKEN, FORGEJO_RUNNER_APP_SKELETON_DEPLOY_KEY, VAULT_DEPLOY_TOKEN
  • Mail: ZEDMAIL_API_KEY, AUTHELIA_SMTP_USERNAME, AUTHELIA_SMTP_PASSWORD, POSTFIX_SENDER_BOOKING_APIKEY, POSTFIX_SENDER_CONTACT_APIKEY, POSTFIX_SENDER_NOREPLY_APIKEY
  • Alert channels: ALERTMANAGER_DISCORD_WEBHOOK_URL, ALERTMANAGER_SLACK_WEBHOOK_URL, ALERTMANAGER_NTFY_TOPIC
  • AUTHELIA_LDAP_SERVICE_PASSWORD (the bind account's password lives in LLDAP too, so both change together)

Wave 3: decisions for the operator (each needs a yes/no, not just an action): GRAFANA_ADMIN_PASSWORD (Grafana is disabled in this inventory: remove the key?), TF_VAR_root_pass / TF_VAR_user_password (only used at provisioning; rotation does not change existing hosts), POSTGRES_PASSWORD (the database keeps the old one until changed inside Postgres), STAGING_SOPS_AGE_KEY (rotating means re-encrypting staging).

Report today: expired 0, overdue 0, unknown 24, ok 38.

**Rotation wave 1 done, 2026-10-15** (all rotated, merged, approved, deployed, verified): - [x] **forgejo** (`FORGEJO_METRICS_TOKEN`, #168; deployed with prometheus, scrape up) - [x] **headscale** (OIDC client secret on both sides, #170/#171; deployed `headscale,authelia`; Forgejo Authelia login verified) - [x] **ci-ephemeral-wakeup** (#172/#176; receiver verified on the new secret; Forgejo webhook updated and verified: test delivery 202) - [x] **lldap** (admin password + JWT secret, #177/#178; container restarted healthy; Authelia login verified) Found on the way, all fixed: `vault-pr` commits failed in its temporary worktree (hook needs `.venv`) and the deploy gate now refuses an uncommitted vault (#167); `./infra approve` retries a mistyped confirmation (#174); `./infra secrets get KEY` copies one value to the clipboard without printing it (#175). Webhook: a leftover user-level Forgejo hook (created 2026-09-23) still signed with the old secret and produced a 401 beside every 202; deleted. Editing an existing hook's secret did not take effect (twice); a freshly created hook did. **Wave 2: provider-issued credentials (24 `unknown` in the report minus the 5 decisions below).** Procedure for each: issue the new value at the provider, `scripts/secrets/vault-set.py set KEY` (stdin), `scripts/secrets/vault-pr.py --what <label> --key KEY`, merge, `./infra approve`, `git pull`, deploy the role, verify, revoke the old value. - [ ] Cloud/DNS: `TF_VAR_linode_token`, `TF_VAR_cloudflare_api_token`, `TF_VAR_desec_api_token` - [ ] OVH: `OVH_APPLICATION_SECRET`, `OS_PASSWORD` (OVH access key ID is 90d, already rotated via `./infra rotate ovh-s3`) - [ ] Forgejo tokens: `FORGEJO_ADMIN_TOKEN`, `CI_EPHEMERAL_FORGEJO_REPO_TOKEN`, `FORGEJO_RUNNER_APP_SKELETON_DEPLOY_KEY`, `VAULT_DEPLOY_TOKEN` - [ ] Mail: `ZEDMAIL_API_KEY`, `AUTHELIA_SMTP_USERNAME`, `AUTHELIA_SMTP_PASSWORD`, `POSTFIX_SENDER_BOOKING_APIKEY`, `POSTFIX_SENDER_CONTACT_APIKEY`, `POSTFIX_SENDER_NOREPLY_APIKEY` - [ ] Alert channels: `ALERTMANAGER_DISCORD_WEBHOOK_URL`, `ALERTMANAGER_SLACK_WEBHOOK_URL`, `ALERTMANAGER_NTFY_TOPIC` - [ ] `AUTHELIA_LDAP_SERVICE_PASSWORD` (the bind account's password lives in LLDAP too, so both change together) **Wave 3: decisions for the operator** (each needs a yes/no, not just an action): `GRAFANA_ADMIN_PASSWORD` (Grafana is disabled in this inventory: remove the key?), `TF_VAR_root_pass` / `TF_VAR_user_password` (only used at provisioning; rotation does not change existing hosts), `POSTGRES_PASSWORD` (the database keeps the old one until changed inside Postgres), `STAGING_SOPS_AGE_KEY` (rotating means re-encrypting staging). Report today: expired 0, overdue 0, unknown 24, ok 38.
Author
Owner

Wave 3 decisions applied, 2026-10-15 (operator: "go with your recommendations"):

  • GRAFANA_ADMIN_PASSWORD: key removed from the vault (#182) and the manifest entry dropped (#179). Grafana stays disabled; the role still asserts the key if it is ever enabled.
  • TF_VAR_root_pass, TF_VAR_user_password: rotated (#182, promoted #183, ci_ephemeral_provisioner deployed). New VMs only: Linode ignores root_pass/user_data changes on staging and production nodes (checked in terraform/modules/compute/linode/main.tf); OVH admin_pass is an in-place update, not a replacement. Manifest descriptions say so.
  • POSTGRES_PASSWORD: no consumer, so unscheduled (expiry: none: reserved, no consumer); rotate with ALTER USER first if something starts using it.
  • STAGING_SOPS_AGE_KEY: stays on its 365-day schedule (cannot decrypt production); no action now.

Tooling found and fixed on the way: vault-set.py delete KEY (#179), vault-pr.py accepts keys with lowercase such as TF_VAR_* (#181). Report: unknown 24 -> 22 (and 19 of those are the wave 2 list above).

**Wave 3 decisions applied, 2026-10-15** (operator: "go with your recommendations"): - [x] `GRAFANA_ADMIN_PASSWORD`: key removed from the vault (#182) and the manifest entry dropped (#179). Grafana stays disabled; the role still asserts the key if it is ever enabled. - [x] `TF_VAR_root_pass`, `TF_VAR_user_password`: rotated (#182, promoted #183, `ci_ephemeral_provisioner` deployed). New VMs only: Linode ignores `root_pass`/`user_data` changes on staging and production nodes (checked in `terraform/modules/compute/linode/main.tf`); OVH `admin_pass` is an in-place update, not a replacement. Manifest descriptions say so. - [x] `POSTGRES_PASSWORD`: no consumer, so unscheduled (`expiry: none: reserved, no consumer`); rotate with `ALTER USER` first if something starts using it. - [ ] `STAGING_SOPS_AGE_KEY`: stays on its 365-day schedule (cannot decrypt production); no action now. Tooling found and fixed on the way: `vault-set.py delete KEY` (#179), `vault-pr.py` accepts keys with lowercase such as `TF_VAR_*` (#181). Report: unknown 24 -> 22 (and 19 of those are the wave 2 list above).
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
jfraeys/infra#117
No description provided.