fix(rotate): correct the next steps after a rotation; retire the rotate-secret playbook #149

Merged
jfraeys merged 5 commits from jfraeys/fix/rotate-playbook-replay-vars into main 2026-10-14 12:30:17 -04:00
Owner

Found while rotating Vaultwarden on 2026-10-14: scripts/with-vault.sh ansible-playbook playbooks/rotate-secret.yml -e secret_name=vaultwarden rotated the vault (play 1) and then failed in the replay play with 'dry_run_mode' is undefined, leaving a rotated, uncommitted vault and nothing deployed.

Net change of this PR

  • scripts/rotate-secret.sh (SOPS mode): the printed next steps were wrong: a bare with-vault.sh ansible-playbook ... services.yml (skips the promotion gate and the Tailscale key mint), "run again with --save-to-bitwarden" to get the plaintext (that rotates again and changes the value), no commit/merge of the vault change, no verification. Now, in order: save the plaintext to personal Bitwarden now (it exists nowhere else); commit and merge the vault change as a chore(secrets) PR; ./infra deploy --tags <tags> --limit services in one command; service-specific verification (vaultwarden: /admin takes the plaintext token, not the hash). Dry runs say nothing changed. Legacy vault.yml text unchanged.
  • playbooks/rotate-secret.yml is removed. Besides the dry_run_mode bug, its replay play applied only the role matching secret_name straight to the host: for vaultwarden that deploys Vaultwarden with the new OIDC client secret while Authelia keeps the old one (SSO breaks until Authelia is deployed; the script's own tags are vaultwarden,authelia), it skips the promotion gate and services.yml's pre-tasks, and it deploys a vault change that is not merged yet. ./infra deploy cannot be called from inside a play (interactive, runs ansible itself), and nothing else referenced the playbook except comments and two doc lines; ./infra rotate (daily-commands.md) already does the rotation. Doc references removed; the .ansible-lint exclusion for it is gone.

The first commits on this branch fixed the playbook before the decision to retire it; the net diff does not touch it any more.

Checked: ansible-lint roles playbooks (the CI command) passes, bash -n, test-infra-cli, test-rotation-lock and test-bitwarden-save pass; the next-steps text was rendered with stub variables for vaultwarden, ci-ephemeral-wakeup, authelia (dry run) and lldap (legacy).

🤖 Generated with Claude Code

https://claude.ai/code/session_01Ge5tovcwdEUBtPwUktsZZZ

Found while rotating Vaultwarden on 2026-10-14: `scripts/with-vault.sh ansible-playbook playbooks/rotate-secret.yml -e secret_name=vaultwarden` rotated the vault (play 1) and then failed in the replay play with `'dry_run_mode' is undefined`, leaving a rotated, uncommitted vault and nothing deployed. **Net change of this PR** - `scripts/rotate-secret.sh` (SOPS mode): the printed next steps were wrong: a bare `with-vault.sh ansible-playbook ... services.yml` (skips the promotion gate and the Tailscale key mint), "run again with --save-to-bitwarden" to get the plaintext (that rotates again and changes the value), no commit/merge of the vault change, no verification. Now, in order: save the plaintext to personal Bitwarden now (it exists nowhere else); commit and merge the vault change as a `chore(secrets)` PR; `./infra deploy --tags <tags> --limit services` in one command; service-specific verification (vaultwarden: `/admin` takes the plaintext token, not the hash). Dry runs say nothing changed. Legacy `vault.yml` text unchanged. - **`playbooks/rotate-secret.yml` is removed.** Besides the `dry_run_mode` bug, its replay play applied only the role matching `secret_name` straight to the host: for vaultwarden that deploys Vaultwarden with the new OIDC client secret while Authelia keeps the old one (SSO breaks until Authelia is deployed; the script's own tags are `vaultwarden,authelia`), it skips the promotion gate and `services.yml`'s pre-tasks, and it deploys a vault change that is not merged yet. `./infra deploy` cannot be called from inside a play (interactive, runs ansible itself), and nothing else referenced the playbook except comments and two doc lines; `./infra rotate` (daily-commands.md) already does the rotation. Doc references removed; the `.ansible-lint` exclusion for it is gone. The first commits on this branch fixed the playbook before the decision to retire it; the net diff does not touch it any more. Checked: `ansible-lint roles playbooks` (the CI command) passes, `bash -n`, test-infra-cli, test-rotation-lock and test-bitwarden-save pass; the next-steps text was rendered with stub variables for vaultwarden, ci-ephemeral-wakeup, authelia (dry run) and lldap (legacy). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Ge5tovcwdEUBtPwUktsZZZ
fix(rotate): define dry_run_mode in the replay play too
Some checks failed
Lint / lint (pull_request) Has started running
Secret scan / scan (pull_request) Has been cancelled
c45506b6ed
Play vars do not carry across plays, so the replay play's `when: dry_run_mode`
conditions failed with "'dry_run_mode' is undefined" right after a successful
rotation (the vault was updated, nothing was replayed). Same default expression
as the first play.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ge5tovcwdEUBtPwUktsZZZ
Author
Owner

Second commit: the line-24 lint/syntax-check error (the mandatory filter in the play name, plus the same problem in the second play's name) is fixed instead of excluded. Both play names use default('(secret_name not set)'), an explicit first task fails with the same message when secret_name is not set, and the .ansible-lint exclusion for this playbook is removed. Verified locally: ansible-lint roles playbooks (the CI command) passes on 106 files, yamllint passes, and --syntax-check passes with and without -e secret_name.

Second commit: the line-24 lint/syntax-check error (the `mandatory` filter in the play name, plus the same problem in the second play's name) is fixed instead of excluded. Both play names use `default('(secret_name not set)')`, an explicit first task fails with the same message when `secret_name` is not set, and the `.ansible-lint` exclusion for this playbook is removed. Verified locally: `ansible-lint roles playbooks` (the CI command) passes on 106 files, yamllint passes, and `--syntax-check` passes with and without `-e secret_name`.
fix(rotate): play names no longer need secret_name at parse time; drop the lint exclusion
Some checks failed
Lint / lint (pull_request) Has started running
Secret scan / scan (pull_request) Has been cancelled
d8970619ce
`mandatory(...)` and a bare `{{ secret_name }}` in the two play names made
syntax-check (and ansible-lint) fail without -e secret_name, so the playbook was
excluded from lint. Use default() in both names, check secret_name in an explicit
first task with the same message, and remove the .ansible-lint exclusion.
ansible-lint on the full CI command (106 files) and yamllint pass; syntax-check
passes with and without secret_name.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ge5tovcwdEUBtPwUktsZZZ
Author
Owner

Third commit: the next steps printed after a rotation (scripts/rotate-secret.sh SOPS mode, and the playbook's messages) are corrected. Before: replay with a bare with-vault.sh ansible-playbook ... services.yml (skips the promotion gate and the Tailscale key mint), 'run again with --save-to-bitwarden' to get the plaintext (that rotates again, changing the value), and no mention of committing the vault change or verifying. Now, in order: (1) save the plaintext to personal Bitwarden now, it exists nowhere else; (2) commit and merge the vault change as a chore(secrets) PR; (3) ./infra deploy --tags <tags> --limit services in one command; (4) service-specific verification (vaultwarden: /admin takes the plaintext token, not the $argon2id hash). Dry runs say nothing was changed; the legacy vault.yml text is unchanged. Rendered for vaultwarden, ci-ephemeral-wakeup, authelia (dry run) and lldap (legacy) with stub variables; test-infra-cli, test-rotation-lock and test-bitwarden-save pass. Also noted in the playbook: its replay play deploys the roles directly, without the ./infra deploy gate.

Third commit: the next steps printed after a rotation (`scripts/rotate-secret.sh` SOPS mode, and the playbook's messages) are corrected. Before: replay with a bare `with-vault.sh ansible-playbook ... services.yml` (skips the promotion gate and the Tailscale key mint), 'run again with --save-to-bitwarden' to get the plaintext (that rotates again, changing the value), and no mention of committing the vault change or verifying. Now, in order: (1) save the plaintext to personal Bitwarden now, it exists nowhere else; (2) commit and merge the vault change as a chore(secrets) PR; (3) `./infra deploy --tags <tags> --limit services` in one command; (4) service-specific verification (vaultwarden: /admin takes the plaintext token, not the $argon2id hash). Dry runs say nothing was changed; the legacy vault.yml text is unchanged. Rendered for vaultwarden, ci-ephemeral-wakeup, authelia (dry run) and lldap (legacy) with stub variables; test-infra-cli, test-rotation-lock and test-bitwarden-save pass. Also noted in the playbook: its replay play deploys the roles directly, without the ./infra deploy gate.
fix(rotate): print the right next steps after a rotation
Some checks are pending
Lint / lint (pull_request) Successful in 1m57s
Secret scan / scan (pull_request) Has started running
67386f813b
The SOPS-mode next steps told the operator to replay with a bare
`with-vault.sh ansible-playbook ... services.yml` (skips the promotion gate and
the Tailscale key mint), to get plaintext by "running again with
--save-to-bitwarden" (that rotates again and changes the value), and never said
to commit and merge the vault change or how to verify. Now, in order: save the
plaintext to Bitwarden now (it exists nowhere else), commit and merge the vault
change as a chore(secrets) PR, deploy together with `./infra deploy --tags ...`,
verify (vaultwarden: /admin takes the plaintext token, not the hash). Dry runs say
nothing changed. The legacy vault.yml text is unchanged. The playbook's own
messages now say the replay play deploys without the gate and that the PR is
still needed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ge5tovcwdEUBtPwUktsZZZ
jfraeys changed title from fix(rotate): define dry_run_mode in the replay play too to fix(rotate): correct the next steps after a rotation; retire the rotate-secret playbook 2026-10-14 12:22:23 -04:00
chore(rotate): retire playbooks/rotate-secret.yml; ./infra rotate is the one entry point
Some checks failed
Lint / lint (pull_request) Has started running
Secret scan / scan (pull_request) Has been cancelled
154b2c2173
Its replay play deployed only the role matching secret_name straight to the host:
for vaultwarden that left Authelia on the old OIDC client secret (the script's own
tags are vaultwarden,authelia), it skipped the promotion gate and services.yml's
pre-tasks, and deployed a vault change that was not merged yet. ./infra deploy
cannot be called from inside a play (interactive, runs ansible itself), so the
play is removed with the rest of the playbook: nothing else referenced it besides
comments and two doc lines, and ./infra rotate (documented in daily-commands.md)
already does the rotation and prints the right next steps. Removes the doc
references; the .ansible-lint exclusion for it was already dropped.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ge5tovcwdEUBtPwUktsZZZ
Author
Owner

Fourth commit: every script and doc that told the operator to rotate or deploy the old way now points at the new mechanism. Scripts: regenerate-ci-ephemeral-ssh-key.sh and rotate-ovh-s3-credentials.sh now say merge the vault change as a chore(secrets) PR, then ./infra deploy --tags <tags> --limit services (was a bare ansible-playbook); rotate-secret.sh's legacy branch no longer says to re-run to save plaintext. Docs: SECURITY_CHECKLIST (./infra rotate), vault-lockout and mail-pipeline-failure (./infra deploy --tags ..., the latter also drops stale vault.yml wording and names vault-set.py), README, deployment.md and updating-tool-versions.md (./infra deploy instead of deploy-services.sh / with-vault.sh ansible-playbook). Left on purpose: ./setup's internal ansible calls, staging/validation hints in playbooks/tests, ROLLBACK.md's test playbook, the web deploy (no ./infra equivalent), node-failure.md's node-rebuild step, the legacy non-SOPS branch in rotate-secret.sh, and with-vault.sh/deploy-services.sh themselves. ansible-lint (CI command), test-infra-cli, test-promotion, test-rotation-lock and test-bitwarden-save pass.

Fourth commit: every script and doc that told the operator to rotate or deploy the old way now points at the new mechanism. **Scripts:** `regenerate-ci-ephemeral-ssh-key.sh` and `rotate-ovh-s3-credentials.sh` now say merge the vault change as a chore(secrets) PR, then `./infra deploy --tags <tags> --limit services` (was a bare ansible-playbook); `rotate-secret.sh`'s legacy branch no longer says to re-run to save plaintext. **Docs:** SECURITY_CHECKLIST (`./infra rotate`), vault-lockout and mail-pipeline-failure (`./infra deploy --tags ...`, the latter also drops stale vault.yml wording and names vault-set.py), README, deployment.md and updating-tool-versions.md (`./infra deploy` instead of deploy-services.sh / with-vault.sh ansible-playbook). **Left on purpose:** ./setup's internal ansible calls, staging/validation hints in playbooks/tests, ROLLBACK.md's test playbook, the web deploy (no ./infra equivalent), node-failure.md's node-rebuild step, the legacy non-SOPS branch in rotate-secret.sh, and with-vault.sh/deploy-services.sh themselves. ansible-lint (CI command), test-infra-cli, test-promotion, test-rotation-lock and test-bitwarden-save pass.
chore: point rotation and deploy instructions at ./infra rotate / ./infra deploy
All checks were successful
Lint / lint (pull_request) Successful in 1m54s
Secret scan / scan (pull_request) Successful in 15s
64eb1b373d
Scripts: regenerate-ci-ephemeral-ssh-key.sh and rotate-ovh-s3-credentials.sh now
say to merge the vault change as a chore(secrets) PR and then run
`./infra deploy --tags <tags> --limit services` instead of a bare
ansible-playbook; rotate-secret.sh's legacy branch no longer tells the operator to
re-run to save plaintext (that rotates again).
Docs: SECURITY_CHECKLIST (./infra rotate), vault-lockout and mail-pipeline-failure
(./infra deploy --tags ...; the latter also drops the stale vault.yml wording and
names vault-set.py), README and runbooks (./infra deploy instead of calling
deploy-services.sh / with-vault.sh ansible-playbook directly).
Left as is on purpose: ./setup's internal ansible calls, the staging/validation
hints in playbooks/tests, ROLLBACK.md's test-playbook run, the web deploy (no
./infra equivalent), the node-rebuild step in node-failure.md, the legacy
non-SOPS branch of rotate-secret.sh, and with-vault.sh/deploy-services.sh themselves.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ge5tovcwdEUBtPwUktsZZZ
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
jfraeys/infra!149
No description provided.