P8: close rotation hygiene (rotation drill, manifest triage) #117
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Gate: "Expiry reporting and rotation locking work". Status 2026-10-13: code merged and report verified; the real rotation drill is half done.
Done
workflow_dispatchruns (#536, #537) produced one issue (#112), created then updated on rerun.Open
/admin(tracked in #116), then the Bitwarden re-save and the live lock test below. The drill also found two real defects, both fixed: the playbook's second confirmation prompt (#115) and/bin/bash3.2 lackinglocal -n(#115)../scripts/rotate-secret.sh vaultwarden --save-to-bitwardenmust printSaved and verified ..., and the item's note body must be the token.rotate-secret.shat once; the second must refuse with the owner's details.unknownentries insecrets/secrets-manifest.yml(listed in #112). Most have arotate_every_daysschedule but norotated_at; fill only what is true. Today's rotations give realrotated_at: 2026-10-13dates for the keys in #125.AUTHELIA_OIDC_PRIVATE_KEY_PEM: manifest says rotated 2026-10-10,docs/secrets-remediation.mdsays 2026-10-13 (PR #52).docs/secrets-remediation.mdto Met with the evidence (#536/#537, #112, the drill).P8 update 2026-10-14: #143 merged. The manifest now records
rotated_at: 2026-10-13for the 8 keys in #125 plusPOSTFIX_SENDER_ALERTS_APIKEY(#133); expiry report unknown 48 -> 44, ok 13 -> 18. TheAUTHELIA_OIDC_PRIVATE_KEY_PEMreconcile item is resolved: manifest 2026-10-10 is when the key was generated, docs' 2026-10-13 was the PR #52 merge and redeploy, not a conflict (remediation row reworded).Not done, deliberately: the other 44 unknown entries have no rotation evidence, so I did not invent dates; the ones with no
rotate_every_days/ expiry need a policy decision from you (not a recorded fact). Still open: Bitwarden re-save, /admin token check, the live two-at-once rotation-lock test (it rotates real secrets, so it is your call), confirm a failed monthly run is visible to you, set the P8 row to Met.P8 update 2026-10-14: the rotation drill's /admin check now passes: the new Vaultwarden admin token works (after #157 fixed the compose
$escaping that had mangled every hashed token; see #116). The Bitwarden copy holds the bare token (the*_PLAIN=prefix is gone from the script). Remaining for P8: the live two-at-once rotation-lock test, confirming a failed monthly run is visible to you, the policy decision for manifest entries without a schedule, then the P8 row to Met. Also new since the last update: rotations open their ownchore(secrets)PR and set the manifest'srotated_at(#151), so the manual manifest step from #143 is automated.P8 decisions, 2026-10-14 (operator: "go with the recommendations")
rotation_lock.py hold), then ran a realrotate-secret.sh lldap: it refused at once, exit 3,another rotation is running: pid … on Jays-Mac-Studio.local … lock drill, before touching the vault; lock free afterwards, vault unchanged.SecretExpired(critical) andSecretRotationOverdue(warning) fire from live metrics (14 series on services) through Alertmanager to phone push and email (delivery verified earlier); rules are tested in CI. The workflow's own failure in Actions stays as a second signal../infra rotate lldapsafe (it rotated LLDAP_KEY_SEED, which would lock everyone out, and a rotated admin password never reached LLDAP).Rotation order for the 32 with a schedule but no date (each rotation now records its date via the PR it opens, #151; deploy what the output names, together):
Wave 1, one command each:
./infra rotate forgejo(FORGEJO_RUNNER_REGISTRATION_TOKEN, FORGEJO_API_TOKEN, FORGEJO_METRICS_TOKEN). Check the printed deploy tags includeprometheus(it scrapes with the metrics token)../infra rotate headscale(HEADSCALE_OIDC_CLIENT_SECRET + AUTHELIA_OIDC_HEADSCALE_CLIENT_SECRET)../infra rotate ci-ephemeral-wakeup, then the same value in Forgejo's webhook settings../infra rotate lldap(admin password, JWT secret), only after #159 is deployed.Wave 2, provider UI then
scripts/secrets/vault-set.py set KEYandscripts/secrets/vault-pr.py --what <label> --key KEY, deploy, revoke the old value:./infra secrets sync): TF_VAR_cloudflare_api_token, TF_VAR_desec_api_token, TF_VAR_linode_token, OS_PASSWORD, OVH_APPLICATION_SECRET.authelia-serviceaccount), deploy authelia.Wave 3, needs a decision first:
ALTER USERin the database plus a coordinated redeploy of its consumers.Rotation wave 1 done, 2026-10-15 (all rotated, merged, approved, deployed, verified):
FORGEJO_METRICS_TOKEN, #168; deployed with prometheus, scrape up)headscale,authelia; Forgejo Authelia login verified)Found on the way, all fixed:
vault-prcommits failed in its temporary worktree (hook needs.venv) and the deploy gate now refuses an uncommitted vault (#167);./infra approveretries a mistyped confirmation (#174);./infra secrets get KEYcopies one value to the clipboard without printing it (#175). Webhook: a leftover user-level Forgejo hook (created 2026-09-23) still signed with the old secret and produced a 401 beside every 202; deleted. Editing an existing hook's secret did not take effect (twice); a freshly created hook did.Wave 2: provider-issued credentials (24
unknownin the report minus the 5 decisions below). Procedure for each: issue the new value at the provider,scripts/secrets/vault-set.py set KEY(stdin),scripts/secrets/vault-pr.py --what <label> --key KEY, merge,./infra approve,git pull, deploy the role, verify, revoke the old value.TF_VAR_linode_token,TF_VAR_cloudflare_api_token,TF_VAR_desec_api_tokenOVH_APPLICATION_SECRET,OS_PASSWORD(OVH access key ID is 90d, already rotated via./infra rotate ovh-s3)FORGEJO_ADMIN_TOKEN,CI_EPHEMERAL_FORGEJO_REPO_TOKEN,FORGEJO_RUNNER_APP_SKELETON_DEPLOY_KEY,VAULT_DEPLOY_TOKENZEDMAIL_API_KEY,AUTHELIA_SMTP_USERNAME,AUTHELIA_SMTP_PASSWORD,POSTFIX_SENDER_BOOKING_APIKEY,POSTFIX_SENDER_CONTACT_APIKEY,POSTFIX_SENDER_NOREPLY_APIKEYALERTMANAGER_DISCORD_WEBHOOK_URL,ALERTMANAGER_SLACK_WEBHOOK_URL,ALERTMANAGER_NTFY_TOPICAUTHELIA_LDAP_SERVICE_PASSWORD(the bind account's password lives in LLDAP too, so both change together)Wave 3: decisions for the operator (each needs a yes/no, not just an action):
GRAFANA_ADMIN_PASSWORD(Grafana is disabled in this inventory: remove the key?),TF_VAR_root_pass/TF_VAR_user_password(only used at provisioning; rotation does not change existing hosts),POSTGRES_PASSWORD(the database keeps the old one until changed inside Postgres),STAGING_SOPS_AGE_KEY(rotating means re-encrypting staging).Report today: expired 0, overdue 0, unknown 24, ok 38.
Wave 3 decisions applied, 2026-10-15 (operator: "go with your recommendations"):
GRAFANA_ADMIN_PASSWORD: key removed from the vault (#182) and the manifest entry dropped (#179). Grafana stays disabled; the role still asserts the key if it is ever enabled.TF_VAR_root_pass,TF_VAR_user_password: rotated (#182, promoted #183,ci_ephemeral_provisionerdeployed). New VMs only: Linode ignoresroot_pass/user_datachanges on staging and production nodes (checked interraform/modules/compute/linode/main.tf); OVHadmin_passis an in-place update, not a replacement. Manifest descriptions say so.POSTGRES_PASSWORD: no consumer, so unscheduled (expiry: none: reserved, no consumer); rotate withALTER USERfirst if something starts using it.STAGING_SOPS_AGE_KEY: stays on its 365-day schedule (cannot decrypt production); no action now.Tooling found and fixed on the way:
vault-set.py delete KEY(#179),vault-pr.pyaccepts keys with lowercase such asTF_VAR_*(#181). Report: unknown 24 -> 22 (and 19 of those are the wave 2 list above).