Next: immediate steps (merge, deploy, verify) after the 2026-10-13 session #116
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Status 2026-10-13 ~15:35 EDT. CI is green and the queue is empty.
origin/main=221396d; the operator checkout is synced to it. Staging run #573 passed onfd26dc3and #77 is validated at that commit (the later221396donly changedsecrets/vault.sops.yml, a non-staged path, so the gate accepts it). Operator progress (~18:30 EDT): everything merged is now deployed and verified; no alert is firing. production is deployed and verified for everything merged so far.origin/main=e405bf1; promotion #77 and #130 merged (prodcontainsmain); Terraform clean (./infra plan: no changes, no drift); rotated secrets deployed; provisioner, metrics/cAdvisor and Alertmanager push deployed; both alert paths verified with real delivery (phone push, and email fromalerts@). Open PRs: #138 (promotion to production, approval only), #101/#102 (green, merge or close), #106 (redundant, close). #137 merged. Merged today: #136 (trivy report), #135 (promotion, journald cap deployed), #127, #131, #132, #133, #134. Branches cleaned up (45 remote, 46 local merged branches deleted; kept: main, prod, #101/#102/#106 branches, local backup/main-before-rebase). #131 and #132 (CI tools) merged and verified on the runner.Pending deploy: trivy report fix (#136 merged
4e70c3b): staging is green now, promotion PR #138 is open. The first staging run for #136 (API 604, UI #413) failed on one probe,tailnet Postfix relay (ACL tag:web -> services:25) -> filtered, a single TCP sample right after the services deploy (the HTTPS/LLDAP probes after it retry 6 x 10 s). #137 merged (a1250c6): the open-expecting TCP probes now retry 6 x 10 s. The retest passed: staging run API 607 (UI #416), 882 s, validate + promote green; promotion PR #138 is validated ata1250c6(= main head), open and mergeable. That supports a startup-timing flake as the cause (still not proven: the run did not say whether the probe needed its retries).prodis stilla6d7219. To finish: (1)./infra plan(expect exit 0; nothing here touches Terraform); (2) merge #138; (3)git pull; (4)./infra deploy --tags hardening --limit services; (5)sudo systemctl start trivy-scan.serviceonce (about a minute of CPU on this host); (6)sudo journalctl -u trivy-scan.service -o catfor the CVE lists (the per-image JSON is under/var/log/trivy-scans/trivy-scan-<ts>/); then triage per #121.Remaining operator steps (short list)
validate-hardening.ymlcheck ran before the hardening role is applied; #134 gated it to the post-deploy pass; proving run API 599 (UI #408) passed in 910 s; promotion PR #135 merged (prod=a6d7219, contains main);./infra deploy --tags hardening --limit servicesrun. On the host: drop-inSystemMaxUse=500Mpresent and effective; journal 2.4 GB to 493 MB;/82% to 73% used (free 4.2 GB to 6.2 GB); live/free 25.8% against the 20% threshold;HostDiskSpaceWarningresolved (ruleinactive, no alerts firing). The UI numbers runs by a per-repo index (UI 405 = API id 596).POSTFIX_SENDER_ALERTS_APIKEY, vault only) merged. In your checkout:git restore secrets/vault.sops.ymlthengit pullif not done yet (backup:~/.local/state/infra/vault.sops.yml.local-alerts-key-2026-10-13).jfraeys/feat/infra-lifecycle-metricsis fully merged; deleting it closes the PR).merged: falseafter being reported done). Check what the page shows after clicking merge (error banner, a scheduled "merge when checks succeed", a different button); agent-side merges worked each time.VAULTWARDEN_ADMIN_TOKENworks at/admin; Bitwarden re-save with the fixed script; live rotation-lock test (see #117).~/.local/state/infra/vault.sops.yml.local-*backups once you are satisfied the deployed values work (they are encrypted copies of older vault states).Resolved today (operator-confirmed unless noted)
ci_ephemeral_provisionerdeploy: the assert failed becauseCI_EPHEMERAL_TAILSCALE_AUTHKEYis environment-only and onlydeploy-services.shmints it (a directwith-vault.sh ansible-playbookskips that). Re-run through./infra deploy --tags ci_ephemeral_provisioner --limit services: done. Verified on the host: the deployed provisioner contains the #111 fix (TASK_NOT_LISTED), the timer is active, queue empty.oauth2: "invalid_client": Authelia and Forgejo ran different client secrets (one side deployed without the other; Authelia loggedclient secret did not matchat 15:46:47). A group deploy (authelia, forgejo, ...) fixed it: operator confirms the Forgejo login works.Still to verify after the secrets deploy
grafana_enabled: falseininventory/group_vars/all.yml, no container exists, so nothing to log in to. Its rotated secret is in the vault for when it is enabled./identity/sso/prevalidate200, image 1.37.4, SSO_ONLY set).VAULTWARDEN_ADMIN_TOKENworks at/admin(not yet confirmed; the SSO login does not exercise it)../infra deploy --tags authelia,forgejo,grafana,vaultwarden --limit services); deploying one side alone breaks that service's SSO.Done
--forceafter its own confirmation; refresh task usesenv bash, macOS/bin/bash3.2 has nolocal -n)--save-to-bitwardenstored the description instead of the secret and never verified; nowbw_store_verified)plan-prod.sh unlock <ID>, and lock errors now print Who/Created).Do now (operator machine, in this order)
git restore,git pull, fast-forward). Done. Backup of the pre-merge local vault remains at~/.local/state/infra/vault.sops.yml.local-rotation-2026-10-13; delete it once the services are deployed and verified.linodeworkspace released (stale lockc3f00f95-..., holder the operator machine, ~5h40m old, nothing running).scripts/plan-prod.shrun, 2026-10-13 ~15:45 EDT, exit 2 (changes found; not an error).0 to add, 0 to change, 3 to destroy, all three intended by merged PRs:module.dns[0].cloudflare_record.web_a["mail"]andweb_aaaa["mail"](#90, unusedmail.jfraeys.comreturned Cloudflare 526) andmodule.security_settings[0].cloudflare_page_rule.security_level_vpn[0](#89, ineffective vpn Page Rule). No Linode, volume or other record is touched. The "would DESTROY or REPLACE" warning fires for any destroy and is expected here.services_aapp/auth/demo/services/vault/vpn,services_aaaa["vpn"],services_wildcard_a[0],unproxied_agit and*.git. The plan (which refreshes) proposes no change to any of them, so live matches the code and only the stored state is stale; an apply refreshes it. The changed attribute was not inspected. To see it:PLAN_PROD_VERBOSE=1 scripts/plan-prod.sh drift | sed -n '/services_a\["app"\] has changed/,/^ *$/p'(DNS data is not secret; do not paste the whole verbose output).[WARN] promotion PR was validated at fd26dc3, but main is now 221396dis expected:221396d(#125) changed onlysecrets/vault.sops.yml, a non-staged path.fd26dc3.c3ef211../infra plan(plan-prod.sh both):[OK] plan (linode): no changes,[OK] drift (linode): no changes. So themailA/AAAA records (#90) and the vpn Page Rule (#89) are gone, and the 10 records reported as drifted are refreshed in state.rotate-secret.shprinted per service../scripts/rotate-secret.sh vaultwarden --save-to-bitwarden(this rotates again unless--dry-run; decide if another rotation is acceptable). Items saved by the old code hold the description, not the secret.--tags infra_metrics,prometheus,exporters --limit services, #105/#107/#108) done and verified on the host 2026-10-13 ~16:05 EDT: cAdvisor at 19.3 MiB (was ~93 MiB) with the trimmed flags; available memory 254 MB (was 95 MB), swap 556 MB;HostMemoryPressureWarningno longer firing; thePostfixDownseries (container_last_seen{name="postfix-postfix-1"}) still exists; timers present (infra-exposure,infra-backup,infra-tool-latest); textfile metrics written (infra_exposure,infra_static,infra_tool_latest,ci_ephemeral_authkey); first exposure check: 0 unexpected public ports.CI_EPHEMERAL_TAILSCALE_AUTHKEYrefreshed by today's provisioner deploy (it mints a fresh 168h key; expiry about 2026-10-20 afternoon). I did not read the expiry itself: confirm theCIEphemeralAuthKeyExpiringSoonmetric shows the new date, and re-run the deploy before it lapses.Housekeeping
Status 2026-10-13 ~20:30 EDT: trivy fix (#136) deployed; first real scan done and acted on. #139 (cadvisor/node-exporter/fail2ban bumps), #140 (watchtower -> nicholas-fedor fork), #141 (hugo-docs removed) merged, staging green on
a97294a, promoted (#142) and deployed to services; hugo leftovers removed. Failing images 8 -> 3 (postfix, lldap, vaultwarden; accepted, details in #121). Open PR: #143 (manifestrotated_atfor the 2026-10-13 rotations; expiry report unknown 48 -> 44). The R history search was not run (auto mode refused decrypting the vault for it); it needs you to run it.Status 2026-10-14 ~12:00 EDT. Merged and promoted: #139/#140/#141 (image bumps, watchtower fork, hugo removed; deployed, trivy 8 -> 3 failing images, details in #121), #143 (manifest rotations), #101/#102, #144/#146/#147 (P7 wizard, then retired), #148 (ephemeral runner registration leak fix; promotion #145 merged,
./infra deploy --tags ci_ephemeral_provisioner --limit servicesdone; the new script is installed). forge-agent #4/#5/#6 merged (exec-env docs, integration timeout, its own SOPS keys). #106 closed. Open: nothing on the PR list.Verification pending: the runner sweep runs after the current idle VM is destroyed (~12:22). Expect the 5 stale registrations (514, 516, 518, 424, 106) to be deleted; if they are not, the
status: "offline"assumption in the sweep is wrong and needs a fix. Result goes in #121.Still yours (unchanged): #101/#102/#106 triage is done. Remaining from the list above: new
VAULTWARDEN_ADMIN_TOKENat /admin; one real mail per other Postfix sender key; Bitwarden re-save; delete~/.local/state/infra/vault.sops.yml.local-*backups; move the forge-agent recovery key's offline copy from~/Documents/recovery_codesto real offline media; Forgejo UI merges that do not take (#129, #133, #134).Runner-sweep verification result (see #121): #148 works in production. The five stale ephemeral runner registrations were deleted at 12:17:35 EDT after the idle VM was destroyed, and a following cycle that reused its VM three times left no orphans. Nothing pending from the ephemeral-runner cleanup. Still pending: #149 (CI running), and the local Vaultwarden rotation (vault change uncommitted, nothing deployed).
Vaultwarden /admin token: fixed and verified 2026-10-14 (operator-confirmed). The new admin token works at /admin. Root cause was not the token: the compose template passed the Argon2 hash unescaped, Docker Compose expanded every
$..., and the container received=19=65536,t=2,p=4, which Vaultwarden treats as a plaintext password (so no real token ever worked, and /admin accepted that predictable string; /admin is behind Authelia, so exposure needed an SSO login first). Fixed in #157 (escape$as$$, staging asserts the running token equals the vault's, Bitwarden note is the bare token, argon2-only hashing). Promoted (#158) and deployed; on services the running ADMIN_TOKEN is now an Argon2 PHC string (5$sections, length 118; shape checked, value not printed). Vaultwarden rotation (#150) is deployed with it. This ticks: "New VAULTWARDEN_ADMIN_TOKEN works at /admin".Superseded for the current cycle: see #122 (roadmap status) and #251 (access rollout, done) and #187 (OVH migration).