Operator tooling: automation candidates and scripts/ organization (2026-10-14 survey) #153

New issue

Open

opened 2026-10-14 12:43:36 -04:00 by jfraeys · 1 comment

jfraeys commented

2026-10-14 12:43:36 -04:00

Owner

Survey of infra, manage, setup and scripts/ after the rotation work. Done so far: #151 (rotation scripts open the chore(secrets) PR and set the manifest's rotated_at themselves), #152 (./infra is the one front door: day-2 ops forwarded to ./manage, every rotation, checks, keys). Below is what is left, ranked by value.

Automation candidates

infra approve <PR>: re-run the promotion check and ./infra plan, show the result, ask for a typed confirmation, then merge the promotion PR with tea. Merges from the Forgejo UI have not taken several times (#129, #133, #134; see #116), and the CLI merge has worked every time. Keeps the approval explicit.
Post-deploy verification in infra deploy --tags: after the deploy, run manage health check --service <tag> for each deployed tag, and check-exposure.py after a traefik, firewall or hardening deploy. Today both are separate manual steps that are easy to skip.
Fail fast on non-blocking stdio in deploy-services.sh: a backgrounded run (! in Claude Code) fails inside Ansible with "requires blocking IO" only after the Tailscale preauth key was minted. Check before minting and say "run it in a terminal tab".
infra wait <PR|staging>: wait for CI with modest polling via the API (a 20 s git ls-remote loop hit Forgejo's 429).
Weekly trivy result as an issue: feed trivy-scan into forgejo-upsert-issue.py, like the monthly expiry report (#112), so a new CRITICAL image is visible without ssh.
Local vault backups: infra status reports the count and oldest age of ~/.local/state/infra/vault.sops.yml.* (encrypted copies); deleting stays manual.
(low) infra tidy: prune merged local branches and stale worktrees.

scripts/ organization

scripts/ is flat: operator commands, CI-only tools, migration-only wizards and Ansible-Vault leftovers sit side by side.

Kind	Scripts	Proposal
Operator commands	plan-prod, promote-prod, deploy-services, rotate-secret, rotate-ovh-s3-credentials, regenerate-ci-ephemeral-ssh-key, provision-headscale-api-key, provision-ci-state-key, vault-set, vault-pr, sync-ci-secrets, secrets-expiry-report, check-exposure, check-tool-versions, age-*-keygen, init-secrets, with-vault	stay where they are (CI, docs and tests use these paths); reachable through `./infra` after #152
CI / deploy-time helpers	check-downloads, check-secret-hygiene, forgejo-upsert-issue, gen-staging-secrets, export-infra-metrics	stay (workflows and hooks reference them)
Migration-only	cloudflare-migration-setup-wizard, export-cloudflare-zone, ovh-setup-wizard (labelled MIGRATION-ONLY in #147)	move to `scripts/migration/`, delete when each migration ends
Ansible-Vault leftovers	init-vaultwarden, verify-vaultwarden-token	delete or port at #120 (checklist there)
Dev	molecule-test	stay

Recommendation: no mass move (high churn in workflows/docs for little gain); move only the migration-only three, and add this table to scripts/CLAUDE.md so a reader can tell a command from a helper.

Not changed on purpose

./manage stays a separate script: roles/manage_cli installs it on the hosts as infra-manage, where git, tea and the vault are not available.
./setup stays the engine; staging, DR and bootstrap call it directly.

Survey of `infra`, `manage`, `setup` and `scripts/` after the rotation work. Done so far: #151 (rotation scripts open the `chore(secrets)` PR and set the manifest's `rotated_at` themselves), #152 (`./infra` is the one front door: day-2 ops forwarded to `./manage`, every rotation, checks, keys). Below is what is left, ranked by value. ## Automation candidates - [ ] **`infra approve <PR>`**: re-run the promotion check and `./infra plan`, show the result, ask for a typed confirmation, then merge the promotion PR with `tea`. Merges from the Forgejo UI have not taken several times (#129, #133, #134; see #116), and the CLI merge has worked every time. Keeps the approval explicit. - [ ] **Post-deploy verification in `infra deploy --tags`**: after the deploy, run `manage health check --service <tag>` for each deployed tag, and `check-exposure.py` after a `traefik`, firewall or `hardening` deploy. Today both are separate manual steps that are easy to skip. - [ ] **Fail fast on non-blocking stdio** in `deploy-services.sh`: a backgrounded run (`!` in Claude Code) fails inside Ansible with "requires blocking IO" only after the Tailscale preauth key was minted. Check before minting and say "run it in a terminal tab". - [ ] **`infra wait <PR|staging>`**: wait for CI with modest polling via the API (a 20 s `git ls-remote` loop hit Forgejo's 429). - [ ] **Weekly trivy result as an issue**: feed `trivy-scan` into `forgejo-upsert-issue.py`, like the monthly expiry report (#112), so a new CRITICAL image is visible without ssh. - [ ] **Local vault backups**: `infra status` reports the count and oldest age of `~/.local/state/infra/vault.sops.yml.*` (encrypted copies); deleting stays manual. - [ ] (low) **`infra tidy`**: prune merged local branches and stale worktrees. ## scripts/ organization `scripts/` is flat: operator commands, CI-only tools, migration-only wizards and Ansible-Vault leftovers sit side by side. | Kind | Scripts | Proposal | |---|---|---| | Operator commands | plan-prod, promote-prod, deploy-services, rotate-secret, rotate-ovh-s3-credentials, regenerate-ci-ephemeral-ssh-key, provision-headscale-api-key, provision-ci-state-key, vault-set, vault-pr, sync-ci-secrets, secrets-expiry-report, check-exposure, check-tool-versions, age-*-keygen, init-secrets, with-vault | stay where they are (CI, docs and tests use these paths); reachable through `./infra` after #152 | | CI / deploy-time helpers | check-downloads, check-secret-hygiene, forgejo-upsert-issue, gen-staging-secrets, export-infra-metrics | stay (workflows and hooks reference them) | | Migration-only | cloudflare-migration-setup-wizard, export-cloudflare-zone, ovh-setup-wizard (labelled MIGRATION-ONLY in #147) | move to `scripts/migration/`, delete when each migration ends | | Ansible-Vault leftovers | init-vaultwarden, verify-vaultwarden-token | delete or port at #120 (checklist there) | | Dev | molecule-test | stay | Recommendation: no mass move (high churn in workflows/docs for little gain); move only the migration-only three, and add this table to `scripts/CLAUDE.md` so a reader can tell a command from a helper. ## Not changed on purpose - `./manage` stays a separate script: `roles/manage_cli` installs it on the hosts as `infra-manage`, where git, tea and the vault are not available. - `./setup` stays the engine; staging, DR and bootstrap call it directly.

jfraeys referenced this issue

2026-10-15 18:53:51 -04:00

Cleanup pass: stale references, duplicated helpers, legacy code, root docs #204

jfraeys commented

2026-10-16 20:42:42 -04:00

Author

Owner

Operator tooling added or fixed this cycle (all merged): manage user roles/access/grant/revoke (#248); manage health recognises Compose containers and monitors Prometheus/Alertmanager (#249); manage backup restore validates the snapshot before any remote work (#226); manage node list works on macOS, no timeout binary (#224); scripts/secrets/with-vault.sh --env (#228); scripts/checks/check-exposure.py login-redirect checks (#250); infra-state-snapshot in the backup run (#222, #223); ./infra moved into scripts/lib/infra (#205). Candidates still open: a tailnet-reach smoke script; a one-command OVH rehearsal runner; a Headscale stale-node cleaner (list + delete with confirmation).

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

jfraeys/infra#153

No description provided.

Rows
Columns