Roadmap: remaining infra and secrets work (tracker) #122

Open
opened 2026-10-13 14:25:54 -04:00 by jfraeys · 3 comments
Owner

Tracking issue for everything left after the 2026-10-13 session. Status as of ~19:50 EDT (updated). #77 and #126 merged; rotated secrets and the provisioner fix are deployed (Forgejo and Vaultwarden SSO verified; Grafana is disabled by design); the Terraform apply is done and clean (./infra plan: no changes, no drift). CI green, queue empty, main = 221396d, operator checkout synced, #125 merged, staging #573 passed and #77 validated at fd26dc3. Work top to bottom; later items depend on earlier ones.

# Item State
#116 Immediate steps: merges, deploys, verification In progress. Next (operator): release the stale state lock, run plan-prod.sh, merge #77, deploy secrets together; then deploy metrics/cAdvisor (host memory ~95 MB free)
#117 P8 rotation hygiene Code merged, report verified. Real rotation drill pending deploy of #125; manifest triage open
#118 P7 project keys and drills Convention merged everywhere. Keys, secrets/dev.sops.yml and drills are operator-machine work
#119 R remediation Not started (Trash, shell history, transcripts)
#120 Retire Ansible Vault Blocked on #117, #118, #119 and a full end-to-end check
#121 Infra backlog Partly done (#110/#111/#113/#108 merged); capacity, deploy verification and older loose ends open

Merged today: #98, #99, #100 (P8 code), #103, #104 (dependency bumps), #105, #107, #108, #109 (P7 convention), #110, #111, #113, #114, #115, #123, #124; forge-agent #3, app-skeleton #1.

Open PRs: #138 (promotion to production: merge after ./infra plan; approval only), #101, #102 (green; merge or close), #106 (redundant, close). #136 (trivy report keeps failing images) and #137 (staging TCP probe retries) are merged; staging run API 607 (UI #416) is green, validated at a1250c6. Staging run API 599 (UI #408) passed; promotion PR #135 is open (validated at bbc8565 = main head). Branches cleaned up (merged ones deleted).

Blockers right now: none; no alert is firing and everything merged is deployed and verified (rotated secrets, provisioner fix, metrics/cAdvisor, phone push, alert email, journald cap; Terraform clean). Needs a look (#121): trivy-scan (119 CRITICAL across 9 of 15 images on 2026-10-11, before the #103 patch updates); its report was dropping the failing images, fixed in #136 (merged; deploy waits for promotion PR #138 and ./infra deploy --tags hardening --limit services, see #116). The backup-restore-test failures were investigated: the scheduled 2026-10-12 run passed fully (primary and OVH secondary restore), and the two red units were manual systemd-run runs without the service environment. Open checks: the new Vaultwarden admin token at /admin; the other three Postfix sender keys; the Bitwarden re-save and live rotation-lock test (#117). Resolved today: stale state lock, SSO invalid_client, provisioner deploy, Terraform apply, host memory, phone push, alert email, alerts@ key, journald cap and disk.

Gate status lives in docs/secrets-remediation.md; update its rows as each issue closes.

Tracking issue for everything left after the 2026-10-13 session. **Status as of ~19:50 EDT (updated).** #77 and #126 merged; rotated secrets and the provisioner fix are deployed (Forgejo and Vaultwarden SSO verified; Grafana is disabled by design); the Terraform apply is done and clean (`./infra plan`: no changes, no drift). CI green, queue empty, `main` = `221396d`, operator checkout synced, #125 merged, staging #573 passed and #77 validated at `fd26dc3`. Work top to bottom; later items depend on earlier ones. | # | Item | State | |---|---|---| | #116 | Immediate steps: merges, deploys, verification | In progress. Next (operator): release the stale state lock, run `plan-prod.sh`, merge #77, deploy secrets together; then deploy metrics/cAdvisor (host memory ~95 MB free) | | #117 | P8 rotation hygiene | Code merged, report verified. Real rotation drill pending deploy of #125; manifest triage open | | #118 | P7 project keys and drills | Convention merged everywhere. Keys, `secrets/dev.sops.yml` and drills are operator-machine work | | #119 | R remediation | Not started (Trash, shell history, transcripts) | | #120 | Retire Ansible Vault | Blocked on #117, #118, #119 and a full end-to-end check | | #121 | Infra backlog | Partly done (#110/#111/#113/#108 merged); capacity, deploy verification and older loose ends open | **Merged today:** #98, #99, #100 (P8 code), #103, #104 (dependency bumps), #105, #107, #108, #109 (P7 convention), #110, #111, #113, #114, #115, #123, #124; forge-agent #3, app-skeleton #1. **Open PRs:** #138 (promotion to production: merge after `./infra plan`; approval only), #101, #102 (green; merge or close), #106 (redundant, close). #136 (trivy report keeps failing images) and #137 (staging TCP probe retries) are merged; staging run API 607 (UI #416) is green, validated at `a1250c6`. Staging run API 599 (UI #408) passed; promotion PR **#135** is open (validated at `bbc8565` = main head). Branches cleaned up (merged ones deleted). **Blockers right now:** none; **no alert is firing** and everything merged is deployed and verified (rotated secrets, provisioner fix, metrics/cAdvisor, phone push, alert email, journald cap; Terraform clean). Needs a look (#121): `trivy-scan` (119 CRITICAL across 9 of 15 images on 2026-10-11, before the #103 patch updates); its report was dropping the failing images, fixed in #136 (merged; deploy waits for promotion PR #138 and `./infra deploy --tags hardening --limit services`, see #116). The `backup-restore-test` failures were investigated: the scheduled 2026-10-12 run passed fully (primary and OVH secondary restore), and the two red units were manual `systemd-run` runs without the service environment. Open checks: the new Vaultwarden admin token at `/admin`; the other three Postfix sender keys; the Bitwarden re-save and live rotation-lock test (#117). Resolved today: stale state lock, SSO `invalid_client`, provisioner deploy, Terraform apply, host memory, phone push, alert email, `alerts@` key, journald cap and disk. Gate status lives in `docs/secrets-remediation.md`; update its rows as each issue closes.
Author
Owner

Tracker update 2026-10-14 ~12:00 EDT (the body above is stale):

# State
#116 All merged work is deployed and verified; runner-sweep verification pending ~12:22 (see #116, #121).
#117 P8 Code merged; manifest dates recorded (#143); real drill, Bitwarden re-save, lock test and policy decisions are yours.
#118 P7 forge-agent keys done and drilled (operator + recovery, encrypted dev.sops.yml, integration passes; forge-agent #6 merged). Row in docs/secrets-remediation.md is "Keys met; gate test 2 pending". Open: gate test 2 (an app deploys through the contract), app-skeleton keys/TODOs, decision for fetch_ml/afk_bot. The wizard is retired (#147).
#119 R Not started; history search needs you or a permitted script.
#120 Blocked on P8, P7 gate test 2 and R; retirement checklist added there.
#121 trivy 8 -> 3 failing (postfix, lldap, vaultwarden accepted); runner leak fixed (#148); backup-terraform.sh removed; capacity and test-forge items added.

Open PRs: none.

**Tracker update 2026-10-14 ~12:00 EDT (the body above is stale):** | # | State | |---|---| | #116 | All merged work is deployed and verified; runner-sweep verification pending ~12:22 (see #116, #121). | | #117 P8 | Code merged; manifest dates recorded (#143); real drill, Bitwarden re-save, lock test and policy decisions are yours. | | #118 P7 | forge-agent keys done and drilled (operator + recovery, encrypted dev.sops.yml, integration passes; forge-agent #6 merged). Row in docs/secrets-remediation.md is "Keys met; gate test 2 pending". Open: gate test 2 (an app deploys through the contract), app-skeleton keys/TODOs, decision for fetch_ml/afk_bot. The wizard is retired (#147). | | #119 R | Not started; history search needs you or a permitted script. | | #120 | Blocked on P8, P7 gate test 2 and R; retirement checklist added there. | | #121 | trivy 8 -> 3 failing (postfix, lldap, vaultwarden accepted); runner leak fixed (#148); backup-terraform.sh removed; capacity and test-forge items added. | Open PRs: none.
Author
Owner

Tracker update 2026-10-15: rotation wave 1 is done and deployed (forgejo, headscale, ci-ephemeral-wakeup, lldap); details and the wave 2 checklist (19 provider-issued credentials) and wave 3 decisions (5) are in #117. New operator tooling merged: ./infra approve carry-over and retry (#164, #174), ./infra secrets get (#175), vault-pr hook fix and the deploy gate's uncommitted-vault refusal (#167). Production = main (prod at the #178 promotion). Open PRs: none. Still blocked as before: #118 gate test 2, #119 history search/Trash, #120 until P8/P7/R close.

**Tracker update 2026-10-15:** rotation wave 1 is done and deployed (forgejo, headscale, ci-ephemeral-wakeup, lldap); details and the wave 2 checklist (19 provider-issued credentials) and wave 3 decisions (5) are in #117. New operator tooling merged: `./infra approve` carry-over and retry (#164, #174), `./infra secrets get` (#175), `vault-pr` hook fix and the deploy gate's uncommitted-vault refusal (#167). Production = main (`prod` at the #178 promotion). Open PRs: none. Still blocked as before: #118 gate test 2, #119 history search/Trash, #120 until P8/P7/R close.
Author
Owner

Roadmap status 2026-10-17. Completed this cycle: OVH rehearsal on both nodes; Terraform state snapshot into every backup (6 workspaces, both repos); DR-relevant firewall fix; Traefik->web over the tailnet; tailnet ACL admin group + Authelia login policy + Alertmanager behind Authelia (tracking #251, deployed and verified); manage CLI: user roles/access/grant/revoke, health fixes, restore snapshot validation, node list on macOS; cleanup items A, B, D and the safe part of C (#204). In progress: OVH cutover prep (#187: firewall tag on ovh-services, stale Headscale nodes, name swap and backup flip decisions); #220 (dedicated state bucket, OVH copy, encryption); #242 (Headscale DNS names). Not started: DR drill, tag:ci split (#251), coach-site backup unit fix (#240, other repo), rest of cleanup C (waits on #120).

Roadmap status 2026-10-17. Completed this cycle: OVH rehearsal on both nodes; Terraform state snapshot into every backup (6 workspaces, both repos); DR-relevant firewall fix; Traefik->web over the tailnet; tailnet ACL admin group + Authelia login policy + Alertmanager behind Authelia (tracking #251, deployed and verified); manage CLI: user roles/access/grant/revoke, health fixes, restore snapshot validation, node list on macOS; cleanup items A, B, D and the safe part of C (#204). In progress: OVH cutover prep (#187: firewall tag on ovh-services, stale Headscale nodes, name swap and backup flip decisions); #220 (dedicated state bucket, OVH copy, encryption); #242 (Headscale DNS names). Not started: DR drill, tag:ci split (#251), coach-site backup unit fix (#240, other repo), rest of cleanup C (waits on #120).
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
jfraeys/infra#122
No description provided.