jfraeys/infra
Watch
1
0
Fork
You've already forked infra
 0
Code Issues 21 Pull requests 4 Projects Releases Packages Wiki Activity Actions

Infra backlog: capacity, CI verification, alerts, older loose ends #121

New issue
Open
opened 2026-10-13 14:25:44 -04:00 by jfraeys · 4 comments
jfraeys commented 2026-10-13 14:25:44 -04:00
Owner
Copy link

Status 2026-10-13 updates. Done today: #110 verified live (and its flaw found and fixed in #123), #111 and #113 merged, #108 merged (cAdvisor trim; measure after deploy). New: the production state lock (see #116), plan-prod.sh unlock (#124).

Open threads that are not part of the secrets plan.

Found 2026-10-13 after the metrics deploy (both pre-date today's deploys)

  • PostfixQueueGrowing is firing and correct. 77 messages queued, all from alerts@jfraeys.com (so Alertmanager email is not being delivered), some about 35 h old. Findings 2026-10-13 ~16:08 EDT: the relay mail.zedmail.com (144.217.242.50) refused connections at first probe (all four ports) but port 2587 then answered 6/6 consecutive probes, so the refusals are intermittent and external (same signature as the 2026-09 incident in docs/incidents.md); Postfix is holding the destination in "delivery temporarily suspended" backoff, which is why the old connect ... Connection refused entries persist. 9 messages instead failed SASL 535 Authentication failed for the alerts@ sender (vault key POSTFIX_SENDER_ALERTS_APIKEY, sender-dependent auth): check the Zedmail dashboard for that key (revoked/rotated/rate-limited after earlier failed attempts). To do: (1) decide whether 35 h-old alert mails are worth delivering (flushing sends them all) or should be dropped (postsuper -d ALL deferred inside the postfix container); (2) docker exec postfix-postfix-1 postqueue -f to retry now, then watch the log for status=sent; (3) if 535 persists, fix or regenerate the alerts@ API key in Zedmail and redeploy --tags postfix; (4) confirm another alert channel is live while email is down (Alertmanager receiver config was not readable from the host without sudo output; verify the Discord/Slack webhook receivers).
  • Phone push for critical/warning alerts: deployed and verified end to end (2026-10-13 ~17:10 EDT). #128 (ntfy push + Alertmanager scrape + AlertNotificationsFailing), #129 (ALERTMANAGER_NTFY_TOPIC) and promotion #130 merged; ./infra deploy --tags alertmanager,prometheus --limit services run. Host checks: /opt/alertmanager/secrets/ntfy_url 0400 nobody:nogroup; the topic is not in the world-readable alertmanager.yml; running config has primary-push (critical and warning route to it, info stays on primary); Prometheus alertmanager target up; AlertNotificationsFailing loaded. A synthetic critical alert (NtfyDeliveryTest, auto-resolving after 5 min) was posted into Alertmanager: webhook notifications 1, failures 0, and the operator received it on the phone. Direct test message to the topic also reached the phone earlier. Not covered: Alertmanager or the whole host being down (needs an external heartbeat).
  • Email delivery works again for the alerts@ sender (2026-10-13 17:10 EDT). After the Postfix container was recreated at 17:10:29, the deferred synthetic-alert email went out: status=sent (250 OK 1 message(s) queued), queue empty, Alertmanager email failures 0; the operator received the email in the inbox. Before that Zedmail answered 535 Authentication failed for alerts@ even though TLS and connectivity were fine. Cause confirmed by the operator: they generated a new alerts@ key in Zedmail, stored it and redeployed Postfix. The old POSTFIX_SENDER_ALERTS_APIKEY was a copy of ZEDMAIL_API_KEY (identical values, checked by equality) while NOREPLY/CONTACT/BOOKING are distinct; Zedmail ties one key to one From address (docs/incidents.md), so a copied general key is the likely reason it was rejected. The new key was only in the operator's local vault until PR #133 (encrypted, only that key differs from main).
  • Verify the other sender keys (NOREPLY, CONTACT, BOOKING): only alerts@ is proven. Authelia, Forgejo and Vaultwarden mail uses them; trigger one real mail from each (e.g. a password-reset email) and look for status=sent.
  • Merge #133 so git matches production; consider alerting on repeated 535s in the Postfix log so a rejected key is noticed before the queue grows. Also check the other three sender keys (above) are distinct and bound to their own From addresses.
  • Prove email works again. The queue was emptied (77 deleted) and is empty, but nothing has proven Zedmail now accepts alerts@. Send a test from the postfix container to an address of yours (printf 'Subject: postfix test\n\ntest\n' | sendmail -f alerts@jfraeys.com YOU@example inside postfix-postfix-1) and look for status=sent. New lifecycle alerts (#105/#107: ToolMinorBehind x5, ToolPatchBehind, ToolUpstreamArchived, SecretsExpiryUnknown) are pending and will start emailing, so a still-broken relay will refill the queue.
  • HostDiskSpaceWarning: fixed 2026-10-13 ~18:23 EDT. The systemd journal was 2.4 GB of the 2.7 GB in /var/log and nothing capped it. The hardening role's SystemMaxUse=500M drop-in (#127, with the staging-check fix #134) is deployed: journal 493 MB, / 73% used (6.2 GB free), alert resolved. /var/lib/docker is 8.2 GB: still to review with docker system df.
  • Failed units on the services host (found 2026-10-13, neither looked into yet):
    • backup-restore-test "failures": investigated 2026-10-13; backups are restorable. The scheduled weekly run on 2026-10-12 01:00 EDT passed completely (report /var/log/backup-restore-test/test-20261012_010004.log): opt/vaultwarden/data, opt/traefik, opt/forgejo restored and verified, Vaultwarden SQLite integrity ok, Forgejo dump (zip) and Authelia dump (sqlite) valid, and the secondary (OVH) restore also SUCCESS. The two red units (run-u4261, run-u4287, 2026-10-13 12:00:57 and 12:05:11) were transient systemd-run units: they lack the service's EnvironmentFile=/etc/infra-backup.env, HOME and XDG_CACHE_HOME; with an empty environment the script stops at RESTIC_REPOSITORY: unbound variable (reproduced on the host). They created no report and did not touch the metric (last written 2026-10-12 01:00:45), so they died in the first lines; their output never reached the journal (probably piped to the launcher), and who started them is unknown, so the exact error was not seen. Run it by hand with the unit: sudo systemctl start backup-restore-test.service then journalctl -u backup-restore-test -f. Confirmed by a fresh run 2026-10-13 18:29-18:30 EDT (sudo systemctl start backup-restore-test.service, exit 0 in 43 s, report test-20261013_182924.log, metric file refreshed 18:30:07): primary restore SUCCESS (snapshot 01dca5d0: all three paths, Vaultwarden SQLite integrity, Forgejo dump forgejo-dump-20261013T040007Z.zip and Authelia dump authelia-db-20261013T040007Z.sqlite3 valid) and the OVH secondary restore SUCCESS. The stale run-u* failed units were cleared with reset-failed; trivy-scan.service is now the only failed unit. Next scheduled run Mon 2026-10-19 01:00 EDT. Note for manual runs: journalctl needs sudo (the ansible user is not in adm/systemd-journal), otherwise it shows nothing.
    • trivy-scan.service (weekly, Sunday 03:00): 15 images scanned 2026-10-11, 9 failed, 119 CRITICAL in total. Counts (CRITICAL/HIGH) from the journal: klakegg/hugo:ext-alpine 55/746, boky/postfix:5.1.0 36/360, vaultwarden/server:1.37.1 7/129, gcr.io/cadvisor/cadvisor:v0.49.1 5/66, containrrr/watchtower:1.7.1 5/51, lldap/lldap:stable 4/57, authelia/authelia:4.39.2 3/45, crazymax/fail2ban:1.1.0 3/31, prom/node-exporter:v1.10.2 1/43; clean (0 CRITICAL): crowdsec v1.8.0, forgejo 16.0.3, headscale 0.29.2, alertmanager v0, prometheus v3, traefik v3. The saved report never contained the failing images (bug: scan_image returned before appending; the report held only the 6 clean ones), so the CVE details were never kept. Fix: PR #136, merged (4e70c3b) but not deployed (keeps every image's JSON, writes one file per image, prints the top CRITICAL CVEs with fixed versions into the journal; 10 tests with a fake trivy; staging run API 604 / UI #413 in progress when written, the promotion PR follows it). The scan ran BEFORE the 2026-10-13 patch updates (#103: Forgejo 16.0.6, Vaultwarden 1.37.4, Headscale 0.29.5, Authelia), so vaultwarden/authelia/forgejo/headscale may already be better.
      • To do: promotion PR #138 (staging green, API 607) must be merged first (see #116); ./infra deploy --tags hardening --limit services, then run sudo systemctl start trivy-scan.service once (about 1 min CPU; brief memory pressure on this host) and read the per-image files / journal (sudo journalctl -u trivy-scan.service -o cat), or wait for Sunday 03:00. Then triage each image: fixable by a version bump (docs/tool-versions.yml, docs/update-and-patch-policy.md) / accept (exposure) / replace.
      • Likely decisions (to confirm with the real CVE list): klakegg/hugo:ext-alpine (55 CRITICAL, floating tag, serves the forgejo-hugo-docs container) looks like an unmaintained image: replace with a maintained one or drop that container; containrrr/watchtower is upstream-archived (a maintained fork exists); boky/postfix:5.1.0 is the latest upstream release per the tool survey, so its CRITICALs are base-image packages (port 25 is tailnet-only: low exposure); cAdvisor 0.49.1 update to a newer release is a deferred decision; lldap:stable and crazymax/fail2ban can be re-pulled/bumped.

CI gaps found 2026-10-13 (lint job log)

  • amtool check in CI: #131 merged.
  • Alert-rule and key-isolation tests in CI: #132 merged (pinned promtool 3.15.0 / amtool 0.34.1 / sops 3.13.3 / age 1.3.2, SHA-256 verified). Verified on the real runner (lint job 981): 4 checksums OK, 37 rules found, 3 promtool test files SUCCESS, key isolation 8/8, amtool accepts both configs, no SKIP lines, and no promtool crash on native amd64 (supports the theory that the concurrent map writes crash seen under QEMU on an Apple Silicon Mac was an emulation artifact). The lint job went from ~90 s to ~160 s.

Capacity / platform

  • Services host has 960 MB RAM and 1 vCPU (was ~95 MB available; after the cAdvisor trim, 254 MB available at ~16:05 on 2026-10-13 and HostMemoryPressureWarning cleared. Resizing to 2 GB at the OVH move is still the real fix). #108 trims cAdvisor; measure after deploy. Resize to 2 GB (Terraform instance type, --confirm-prod, short reboot) when moving to OVH, as decided.
  • Prometheus instance label shows the container name (node-exporter:9100); optional relabel to show services.
  • OVH migration: not started in the repo; the plan, cost and downtime are the open decision.

CI

  • Verify #110 live: done 2026-10-13; it skipped a run wrongly (a later commit touching no staged path), fixed in #123. #565 now skips in 6 s and #568 runs in full.
  • Deploy #111 (see #116), then verify it live (provisioner no longer stalls on a task Forgejo does not list); wire pytest roles/ci_ephemeral_provisioner/tests into lint (it is not in CI today, nor is the existing test_breaker.py).
  • Avoid cancelling a run that is mid-job: it orphaned a runner cycle for ~55 min on 2026-10-13. Cancel only runs in waiting.
  • ansible-lint fails syntax-check on playbooks/rotate-secret.yml standalone (needs -e secret_name=); give secret_name a lint-safe default or exclude it.
  • The runner for the manual-only staging-reboot-check.yml is now ephemeral-large (#113, merged); run it once: the reboot check is built but has never been run.

Alerts and metrics (new, #105-#107)

  • After deploy, confirm each new alert renders and the weekly upstream check has run; triage docs/tool-versions.yml entries flagged behind/archived.
  • Confirm PublicPortUnexpected is quiet (allowlist infra_public_ports_allowed: 80/443, 3478/udp, 8443).

Older loose ends

  • mail.jfraeys.com returned 526: the record is removed (#90 applied 2026-10-13, plan clean); coaching-site repo still claims the apex; old PRs #15, #25, #26; PR #77 (promotion) review.
  • Deferred items needing your input before implementing: app_ssh_access register/deregister scripts; forgejo_runner_macos dynamic token.
  • Sequential audit: Phase 0 baseline, L1 and demo readiness are still open (see the earlier audit notes).
  • Restic: services/web were rebuilt from a stale backup on 2026-08-20; confirm old infra-new is deleted and a restore drill has run.
**Status 2026-10-13 updates.** Done today: #110 verified live (and its flaw found and fixed in #123), #111 and #113 merged, #108 merged (cAdvisor trim; measure after deploy). New: the production state lock (see #116), `plan-prod.sh unlock` (#124). Open threads that are not part of the secrets plan. **Found 2026-10-13 after the metrics deploy (both pre-date today's deploys)** - [ ] `PostfixQueueGrowing` is firing and correct. 77 messages queued, all from `alerts@jfraeys.com` (so Alertmanager email is not being delivered), some about 35 h old. Findings 2026-10-13 ~16:08 EDT: the relay `mail.zedmail.com` (144.217.242.50) refused connections at first probe (all four ports) but port 2587 then answered 6/6 consecutive probes, so the refusals are intermittent and external (same signature as the 2026-09 incident in `docs/incidents.md`); Postfix is holding the destination in "delivery temporarily suspended" backoff, which is why the old `connect ... Connection refused` entries persist. 9 messages instead failed SASL `535 Authentication failed` for the `alerts@` sender (vault key `POSTFIX_SENDER_ALERTS_APIKEY`, sender-dependent auth): check the Zedmail dashboard for that key (revoked/rotated/rate-limited after earlier failed attempts). **To do:** (1) decide whether 35 h-old alert mails are worth delivering (flushing sends them all) or should be dropped (`postsuper -d ALL deferred` inside the postfix container); (2) `docker exec postfix-postfix-1 postqueue -f` to retry now, then watch the log for `status=sent`; (3) if 535 persists, fix or regenerate the `alerts@` API key in Zedmail and redeploy `--tags postfix`; (4) confirm another alert channel is live while email is down (Alertmanager receiver config was not readable from the host without sudo output; verify the Discord/Slack webhook receivers). - [x] **Phone push for critical/warning alerts: deployed and verified end to end (2026-10-13 ~17:10 EDT).** #128 (ntfy push + Alertmanager scrape + `AlertNotificationsFailing`), #129 (`ALERTMANAGER_NTFY_TOPIC`) and promotion #130 merged; `./infra deploy --tags alertmanager,prometheus --limit services` run. Host checks: `/opt/alertmanager/secrets/ntfy_url` 0400 nobody:nogroup; the topic is not in the world-readable `alertmanager.yml`; running config has `primary-push` (critical and warning route to it, info stays on `primary`); Prometheus `alertmanager` target up; `AlertNotificationsFailing` loaded. A synthetic critical alert (`NtfyDeliveryTest`, auto-resolving after 5 min) was posted into Alertmanager: webhook notifications 1, failures 0, and **the operator received it on the phone.** Direct test message to the topic also reached the phone earlier. Not covered: Alertmanager or the whole host being down (needs an external heartbeat). - [x] **Email delivery works again for the `alerts@` sender (2026-10-13 17:10 EDT).** After the Postfix container was recreated at 17:10:29, the deferred synthetic-alert email went out: `status=sent (250 OK 1 message(s) queued)`, queue empty, Alertmanager email failures 0; **the operator received the email in the inbox.** Before that Zedmail answered `535 Authentication failed` for `alerts@` even though TLS and connectivity were fine. **Cause confirmed by the operator:** they generated a new `alerts@` key in Zedmail, stored it and redeployed Postfix. The old `POSTFIX_SENDER_ALERTS_APIKEY` was a copy of `ZEDMAIL_API_KEY` (identical values, checked by equality) while NOREPLY/CONTACT/BOOKING are distinct; Zedmail ties one key to one From address (`docs/incidents.md`), so a copied general key is the likely reason it was rejected. The new key was only in the operator's local vault until **PR #133** (encrypted, only that key differs from main). - [ ] **Verify the other sender keys** (`NOREPLY`, `CONTACT`, `BOOKING`): only `alerts@` is proven. Authelia, Forgejo and Vaultwarden mail uses them; trigger one real mail from each (e.g. a password-reset email) and look for `status=sent`. - [ ] Merge #133 so git matches production; consider alerting on repeated `535`s in the Postfix log so a rejected key is noticed before the queue grows. Also check the other three sender keys (above) are distinct and bound to their own From addresses. - [ ] **Prove email works again.** The queue was emptied (77 deleted) and is empty, but nothing has proven Zedmail now accepts `alerts@`. Send a test from the postfix container to an address of yours (`printf 'Subject: postfix test\n\ntest\n' | sendmail -f alerts@jfraeys.com YOU@example` inside `postfix-postfix-1`) and look for `status=sent`. New lifecycle alerts (#105/#107: `ToolMinorBehind` x5, `ToolPatchBehind`, `ToolUpstreamArchived`, `SecretsExpiryUnknown`) are `pending` and will start emailing, so a still-broken relay will refill the queue. - [x] **`HostDiskSpaceWarning`: fixed 2026-10-13 ~18:23 EDT.** The systemd journal was 2.4 GB of the 2.7 GB in `/var/log` and nothing capped it. The `hardening` role's `SystemMaxUse=500M` drop-in (#127, with the staging-check fix #134) is deployed: journal 493 MB, `/` 73% used (6.2 GB free), alert resolved. `/var/lib/docker` is 8.2 GB: still to review with `docker system df`. - [ ] **Failed units on the services host (found 2026-10-13, neither looked into yet):** - [x] `backup-restore-test` "failures": **investigated 2026-10-13; backups are restorable.** The scheduled weekly run on 2026-10-12 01:00 EDT passed completely (report `/var/log/backup-restore-test/test-20261012_010004.log`): `opt/vaultwarden/data`, `opt/traefik`, `opt/forgejo` restored and verified, Vaultwarden SQLite integrity ok, Forgejo dump (zip) and Authelia dump (sqlite) valid, and the **secondary (OVH) restore also SUCCESS**. The two red units (`run-u4261`, `run-u4287`, 2026-10-13 12:00:57 and 12:05:11) were transient `systemd-run` units: they lack the service's `EnvironmentFile=/etc/infra-backup.env`, `HOME` and `XDG_CACHE_HOME`; with an empty environment the script stops at `RESTIC_REPOSITORY: unbound variable` (reproduced on the host). They created no report and did not touch the metric (last written 2026-10-12 01:00:45), so they died in the first lines; their output never reached the journal (probably piped to the launcher), and who started them is unknown, so the exact error was not seen. **Run it by hand with the unit:** `sudo systemctl start backup-restore-test.service` then `journalctl -u backup-restore-test -f`. **Confirmed by a fresh run 2026-10-13 18:29-18:30 EDT** (`sudo systemctl start backup-restore-test.service`, exit 0 in 43 s, report `test-20261013_182924.log`, metric file refreshed 18:30:07): primary restore SUCCESS (snapshot `01dca5d0`: all three paths, Vaultwarden SQLite integrity, Forgejo dump `forgejo-dump-20261013T040007Z.zip` and Authelia dump `authelia-db-20261013T040007Z.sqlite3` valid) and the OVH secondary restore SUCCESS. The stale `run-u*` failed units were cleared with `reset-failed`; `trivy-scan.service` is now the only failed unit. Next scheduled run Mon 2026-10-19 01:00 EDT. Note for manual runs: `journalctl` needs `sudo` (the `ansible` user is not in `adm`/`systemd-journal`), otherwise it shows nothing. - [ ] **`trivy-scan.service` (weekly, Sunday 03:00): 15 images scanned 2026-10-11, 9 failed, 119 CRITICAL in total.** Counts (CRITICAL/HIGH) from the journal: `klakegg/hugo:ext-alpine` 55/746, `boky/postfix:5.1.0` 36/360, `vaultwarden/server:1.37.1` 7/129, `gcr.io/cadvisor/cadvisor:v0.49.1` 5/66, `containrrr/watchtower:1.7.1` 5/51, `lldap/lldap:stable` 4/57, `authelia/authelia:4.39.2` 3/45, `crazymax/fail2ban:1.1.0` 3/31, `prom/node-exporter:v1.10.2` 1/43; clean (0 CRITICAL): crowdsec v1.8.0, forgejo 16.0.3, headscale 0.29.2, alertmanager v0, prometheus v3, traefik v3. **The saved report never contained the failing images** (bug: `scan_image` returned before appending; the report held only the 6 clean ones), so the CVE details were never kept. **Fix: PR #136, merged (`4e70c3b`) but not deployed** (keeps every image's JSON, writes one file per image, prints the top CRITICAL CVEs with fixed versions into the journal; 10 tests with a fake trivy; staging run API 604 / UI #413 in progress when written, the promotion PR follows it). The scan ran BEFORE the 2026-10-13 patch updates (#103: Forgejo 16.0.6, Vaultwarden 1.37.4, Headscale 0.29.5, Authelia), so vaultwarden/authelia/forgejo/headscale may already be better. - To do: promotion PR #138 (staging green, API 607) must be merged first (see #116); `./infra deploy --tags hardening --limit services`, then run `sudo systemctl start trivy-scan.service` once (about 1 min CPU; brief memory pressure on this host) and read the per-image files / journal (`sudo journalctl -u trivy-scan.service -o cat`), or wait for Sunday 03:00. Then triage each image: fixable by a version bump (`docs/tool-versions.yml`, `docs/update-and-patch-policy.md`) / accept (exposure) / replace. - Likely decisions (to confirm with the real CVE list): `klakegg/hugo:ext-alpine` (55 CRITICAL, floating tag, serves the forgejo-hugo-docs container) looks like an unmaintained image: replace with a maintained one or drop that container; `containrrr/watchtower` is upstream-archived (a maintained fork exists); `boky/postfix:5.1.0` is the latest upstream release per the tool survey, so its CRITICALs are base-image packages (port 25 is tailnet-only: low exposure); cAdvisor 0.49.1 update to a newer release is a deferred decision; `lldap:stable` and `crazymax/fail2ban` can be re-pulled/bumped. **CI gaps found 2026-10-13 (lint job log)** - [x] amtool check in CI: **#131 merged**. - [x] Alert-rule and key-isolation tests in CI: **#132 merged** (pinned promtool 3.15.0 / amtool 0.34.1 / sops 3.13.3 / age 1.3.2, SHA-256 verified). **Verified on the real runner** (lint job 981): 4 checksums OK, `37 rules found`, 3 promtool test files SUCCESS, key isolation 8/8, amtool accepts both configs, no SKIP lines, and no promtool crash on native amd64 (supports the theory that the `concurrent map writes` crash seen under QEMU on an Apple Silicon Mac was an emulation artifact). The lint job went from ~90 s to ~160 s. **Capacity / platform** - [ ] Services host has 960 MB RAM and 1 vCPU (was ~95 MB available; after the cAdvisor trim, 254 MB available at ~16:05 on 2026-10-13 and `HostMemoryPressureWarning` cleared. Resizing to 2 GB at the OVH move is still the real fix). #108 trims cAdvisor; measure after deploy. Resize to 2 GB (Terraform instance type, `--confirm-prod`, short reboot) when moving to OVH, as decided. - [ ] Prometheus `instance` label shows the container name (`node-exporter:9100`); optional relabel to show `services`. - [ ] OVH migration: not started in the repo; the plan, cost and downtime are the open decision. **CI** - [x] Verify #110 live: done 2026-10-13; it skipped a run wrongly (a later commit touching no staged path), fixed in #123. #565 now skips in 6 s and #568 runs in full. - [ ] Deploy #111 (see #116), then verify it live (provisioner no longer stalls on a task Forgejo does not list); wire `pytest roles/ci_ephemeral_provisioner/tests` into lint (it is not in CI today, nor is the existing `test_breaker.py`). - [ ] Avoid cancelling a run that is mid-job: it orphaned a runner cycle for ~55 min on 2026-10-13. Cancel only runs in `waiting`. - [ ] `ansible-lint` fails `syntax-check` on `playbooks/rotate-secret.yml` standalone (needs `-e secret_name=`); give `secret_name` a lint-safe default or exclude it. - [ ] The runner for the manual-only `staging-reboot-check.yml` is now `ephemeral-large` (#113, merged); run it once: the reboot check is built but has never been run. **Alerts and metrics (new, #105-#107)** - [ ] After deploy, confirm each new alert renders and the weekly upstream check has run; triage `docs/tool-versions.yml` entries flagged behind/archived. - [ ] Confirm `PublicPortUnexpected` is quiet (allowlist `infra_public_ports_allowed`: 80/443, 3478/udp, 8443). **Older loose ends** - [ ] `mail.jfraeys.com` returned 526: the record is removed (#90 applied 2026-10-13, plan clean); coaching-site repo still claims the apex; old PRs #15, #25, #26; PR #77 (promotion) review. - [ ] Deferred items needing your input before implementing: `app_ssh_access` register/deregister scripts; `forgejo_runner_macos` dynamic token. - [ ] Sequential audit: Phase 0 baseline, L1 and demo readiness are still open (see the earlier audit notes). - [ ] Restic: services/web were rebuilt from a stale backup on 2026-08-20; confirm old `infra-new` is deleted and a restore drill has run.
jfraeys commented 2026-10-13 20:19:05 -04:00
Author
Owner
Copy link

trivy-scan triage, 2026-10-13 (first run with the #136 fix, services, report trivy-scan-20261013_192619)

15 images scanned, 8 failed on CRITICAL. Status:

  • prom/node-exporter v1.10.2 -> v1.12.1, cadvisor v0.49.1 -> ghcr.io/google/cadvisor:v0.60.6 (gcr.io has no newer tag), crazymax/fail2ban 1.1.0 -> 1.1.1: merged in #139. trivy image on the new tags: 0 CRITICAL. Not yet re-scanned on the host.
  • containrrr/watchtower 1.7.1 (5 CRITICAL; archived upstream) -> ghcr.io/nicholas-fedor/watchtower:1.23.1: merged in #140. 0 CRITICAL; a one-shot monitor-only run on services scanned 6 containers, failed=0.
  • klakegg/hugo:ext-alpine (55 CRITICAL; abandoned): the hugo-docs service removed in #141. It only ever served Hugo's placeholder (its jfraeys/docs repo never existed). Pending after deploy: remove the orphaned container and /opt/forgejo/hugo-entrypoint.sh by hand.
  • boky/postfix:5.1.0 (36 CRITICAL, internet-facing relay): no newer upstream release (5.1.0 is latest); the CVEs are in the Debian base. Re-check upstream monthly; the tool-versions drift report already tracks it.
  • lldap/lldap:stable (4 CRITICAL, all Go stdlib 1.18.2): floating tag, latest release v0.6.3. Check whether a pinned tag has a newer Go before accepting.
  • vaultwarden/server:1.37.4 (4 CRITICAL, libmariadb3 / mariadb-common, "no fix yet" in Debian): 1.37.4 is the latest release; we run SQLite (roles/backups snapshots it), not MariaDB, so the vulnerable client libs are unused. Accepted until Debian ships a fix.

After the deploy, re-run trivy-scan.service; expected failing set: postfix, lldap, vaultwarden.

**trivy-scan triage, 2026-10-13 (first run with the #136 fix, services, report `trivy-scan-20261013_192619`)** 15 images scanned, 8 failed on CRITICAL. Status: - [x] `prom/node-exporter` v1.10.2 -> v1.12.1, `cadvisor` v0.49.1 -> `ghcr.io/google/cadvisor:v0.60.6` (gcr.io has no newer tag), `crazymax/fail2ban` 1.1.0 -> 1.1.1: merged in #139. `trivy image` on the new tags: 0 CRITICAL. Not yet re-scanned on the host. - [x] `containrrr/watchtower` 1.7.1 (5 CRITICAL; archived upstream) -> `ghcr.io/nicholas-fedor/watchtower:1.23.1`: merged in #140. 0 CRITICAL; a one-shot monitor-only run on services scanned 6 containers, failed=0. - [x] `klakegg/hugo:ext-alpine` (55 CRITICAL; abandoned): the `hugo-docs` service removed in #141. It only ever served Hugo's placeholder (its `jfraeys/docs` repo never existed). Pending after deploy: remove the orphaned container and `/opt/forgejo/hugo-entrypoint.sh` by hand. - [ ] `boky/postfix:5.1.0` (36 CRITICAL, internet-facing relay): no newer upstream release (5.1.0 is latest); the CVEs are in the Debian base. Re-check upstream monthly; the tool-versions drift report already tracks it. - [ ] `lldap/lldap:stable` (4 CRITICAL, all Go stdlib 1.18.2): floating tag, latest release v0.6.3. Check whether a pinned tag has a newer Go before accepting. - [ ] `vaultwarden/server:1.37.4` (4 CRITICAL, libmariadb3 / mariadb-common, "no fix yet" in Debian): 1.37.4 is the latest release; we run SQLite (roles/backups snapshots it), not MariaDB, so the vulnerable client libs are unused. Accepted until Debian ships a fix. After the deploy, re-run `trivy-scan.service`; expected failing set: postfix, lldap, vaultwarden.
jfraeys commented 2026-10-13 21:02:21 -04:00
Author
Owner
Copy link

trivy-scan result after the deploy, 2026-10-13 ~20:30 EDT (services, 14 images, 3 failed; was 15 / 8)

Deployed: #139, #140, #141 (promotion #142, prod = a97294a). Re-scan CRITICAL counts: fail2ban 1.1.1 0, cadvisor v0.60.6 0, node-exporter v1.12.1 0, watchtower (fork) 1.23.1 0; hugo removed. The hugo container and /opt/forgejo/hugo-entrypoint.sh are removed from services. Also verified live: all Prometheus targets up, container_last_seen still reports postfix (so PostfixDown has data), HostDiskSpaceWarning cleared after removing the six superseded/test images (77% used, ~21.6% free vs the 20% threshold; tight).

Still failing, all accepted for now:

  • boky/postfix:5.1.0 36 CRITICAL: 5.1.0 is the latest upstream release; the CVEs are in the Debian base. Re-check upstream each month.
  • lldap/lldap:stable 4 CRITICAL (Go stdlib: CVE-2023-24538, CVE-2023-24540, CVE-2024-24790, CVE-2025-68121): checked the newest dated tag 2026-09-22 (and its -alpine / -debian variants) with trivy: the same 4. No tag fixes it, so this is an upstream build issue. Not reachable from outside (admin UI is tailnet-only).
  • vaultwarden/server:1.37.4 4 CRITICAL (libmariadb3 / mariadb-common, "no fix yet"): 1.37.4 is the latest release and we run SQLite, so the MariaDB client libs are unused.

Next scheduled scan: Mon 2026-10-19 01:00 EDT; expect exactly these 3 to fail.

**trivy-scan result after the deploy, 2026-10-13 ~20:30 EDT (services, 14 images, 3 failed; was 15 / 8)** Deployed: #139, #140, #141 (promotion #142, prod = a97294a). Re-scan CRITICAL counts: fail2ban 1.1.1 **0**, cadvisor v0.60.6 **0**, node-exporter v1.12.1 **0**, watchtower (fork) 1.23.1 **0**; hugo removed. The hugo container and `/opt/forgejo/hugo-entrypoint.sh` are removed from services. Also verified live: all Prometheus targets up, `container_last_seen` still reports postfix (so `PostfixDown` has data), HostDiskSpaceWarning cleared after removing the six superseded/test images (77% used, ~21.6% free vs the 20% threshold; tight). Still failing, all accepted for now: - [ ] `boky/postfix:5.1.0` 36 CRITICAL: 5.1.0 is the latest upstream release; the CVEs are in the Debian base. Re-check upstream each month. - [ ] `lldap/lldap:stable` 4 CRITICAL (Go stdlib: CVE-2023-24538, CVE-2023-24540, CVE-2024-24790, CVE-2025-68121): checked the newest dated tag `2026-09-22` (and its `-alpine` / `-debian` variants) with trivy: the same 4. No tag fixes it, so this is an upstream build issue. Not reachable from outside (admin UI is tailnet-only). - [ ] `vaultwarden/server:1.37.4` 4 CRITICAL (libmariadb3 / mariadb-common, "no fix yet"): 1.37.4 is the latest release and we run SQLite, so the MariaDB client libs are unused. Next scheduled scan: Mon 2026-10-19 01:00 EDT; expect exactly these 3 to fail.
jfraeys commented 2026-10-14 12:00:53 -04:00
Author
Owner
Copy link

2026-10-14 additions to the backlog:

  • Ephemeral runner registration leak (found while checking CI cleanup): _reuse_vm_for_job reset runner_id to None, so an unconsumed registration from an earlier job on a reused VM was never deleted; 5 stale offline ephemeral-* registrations (514, 516, 518, 424, 106; oldest ~11 days) were found. Fixed in #148 (delete the previous registration on reuse, sweep offline ephemeral-* after a VM destroy), deployed. To verify: after the idle VM is destroyed (~12:22 EDT 2026-10-14) the five should be gone; if not, the sweep's status: "offline" assumption is wrong. VMs themselves converge correctly (10-job-cap VM destroyed, tofu destroy clean).
  • scripts/backup-terraform.sh removed (#147): state is in the S3 backend, the script backed up nonexistent local files and exited 0. docs/runbooks/node-failure.md now says so and shows a manual tofu state pull. Gap: the repo has no separate backup of the state object itself (no versioning or replica evidenced); worth a decision.
  • services is a 960 MB node with ~200 MB available and active swap (575 MB used, 33% I/O wait seen during the forge-agent integration run); the forge-agent integration tests also run against the production Forgejo (jfraeys/forge-agent-it) despite their header saying disposable. Options: resize (runbook resize-node), cut memory, or a throwaway Forgejo for the tests.
**2026-10-14 additions to the backlog:** - [x] **Ephemeral runner registration leak** (found while checking CI cleanup): `_reuse_vm_for_job` reset `runner_id` to None, so an unconsumed registration from an earlier job on a reused VM was never deleted; 5 stale offline `ephemeral-*` registrations (514, 516, 518, 424, 106; oldest ~11 days) were found. Fixed in #148 (delete the previous registration on reuse, sweep offline `ephemeral-*` after a VM destroy), deployed. **To verify:** after the idle VM is destroyed (~12:22 EDT 2026-10-14) the five should be gone; if not, the sweep's `status: "offline"` assumption is wrong. VMs themselves converge correctly (10-job-cap VM destroyed, `tofu destroy` clean). - [x] `scripts/backup-terraform.sh` removed (#147): state is in the S3 backend, the script backed up nonexistent local files and exited 0. `docs/runbooks/node-failure.md` now says so and shows a manual `tofu state pull`. **Gap:** the repo has no separate backup of the state object itself (no versioning or replica evidenced); worth a decision. - [ ] `services` is a 960 MB node with ~200 MB available and active swap (575 MB used, 33% I/O wait seen during the forge-agent integration run); the forge-agent integration tests also run against the production Forgejo (`jfraeys/forge-agent-it`) despite their header saying disposable. Options: resize (runbook resize-node), cut memory, or a throwaway Forgejo for the tests.
jfraeys commented 2026-10-14 12:26:39 -04:00
Author
Owner
Copy link

Runner registration leak: verified fixed in production (2026-10-14). #148 deployed 11:57 EDT. The idle VM hit its 3600 s lifetime cap at 12:16:47 and was destroyed (tofu destroy clean, Linode instance gone); at 12:17:35 the new sweep deleted all five stale ephemeral-* registrations (ids 518, 516, 514, 424, 106) and the log says cycle 1791990954 converged: no VM, no registration. The status: "offline" assumption in the sweep held. A new cycle then started on a fresh VM and reused it three times (12:22, 12:23, 12:24); the runner table showed exactly one registration (the current runner) throughout, so reuse no longer leaks. Closing this item; if a stale registration ever shows up again, the end-of-cycle sweep collects it.

**Runner registration leak: verified fixed in production (2026-10-14).** #148 deployed 11:57 EDT. The idle VM hit its 3600 s lifetime cap at 12:16:47 and was destroyed (`tofu destroy` clean, Linode instance gone); at 12:17:35 the new sweep deleted all five stale `ephemeral-*` registrations (ids 518, 516, 514, 424, 106) and the log says `cycle 1791990954 converged: no VM, no registration`. The `status: "offline"` assumption in the sweep held. A new cycle then started on a fresh VM and reused it three times (12:22, 12:23, 12:24); the runner table showed exactly one registration (the current runner) throughout, so reuse no longer leaks. Closing this item; if a stale registration ever shows up again, the end-of-cycle sweep collects it.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
jfraeys/fix/runner-ansible-core
jfraeys/fix/report-status-token
jfraeys/fix/app-check-installs-app-roles
prod
ci/report-staging-status
jfraeys/feat/ci-cache-weekly-reset
jfraeys/fix/forgejo-actions-rate-limit
jfraeys/fix/runner-forgejo-over-tailnet
jfraeys/feat/vpn-status-tags
jfraeys/fix/setup-mints-ci-authkey
jfraeys/feat/web-ssh-dns
jfraeys/feat/tag-ci-acl
jfraeys/fix/backup-status-full-snapshot
jfraeys/fix/staging-ntfy-topic
jfraeys/refactor/linode-object-storage-names
jfraeys/feat/backups-repo-credentials
jfraeys/ci/cancel-superseded-pr-runs
jfraeys/feat/scripts-util-lib
jfraeys/fix/health-remote-service
jfraeys/refactor/container-memory-vars
jfraeys/fix/linode-prod-keep-nanodes
jfraeys/fix/vpn-revoke-ambiguity
jfraeys/docs/ovh-runbook-ci-window
jfraeys/fix/stable-baseline-sizing
jfraeys/fix/fail2ban-ignore-tailnet
jfraeys/fix/user-access-reach-wording
jfraeys/feat/manage-user-access
jfraeys/fix/headscale-acl-admin-group
jfraeys/fix/alertmanager-behind-authelia
jfraeys/fix/health-compose-names
jfraeys/fix/headscale-oidc-groups
jfraeys/feat/webapps-upstream-over-tailnet
jfraeys/fix/fail2ban-native-needs-authlog
jfraeys/docs/incident-233-fixed
jfraeys/docs/node-naming-primary
jfraeys/fix/firewall-return-last-and-web-open
jfraeys/docs/incident-docker-user-return-order
jfraeys/docs/ovh-key-scope
jfraeys/chore/ovh-inventory-template
jfraeys/fix/firewall-legacy-rule-cleanup
jfraeys/fix/restore-validate-snapshot-first
jfraeys/docs/tailscale-authkey-flow
jfraeys/feat/with-vault-env-flag
jfraeys/fix/manage-no-timeout-binary
jfraeys/fix/state-snapshot-curl-7-88
jfraeys/feat/state-snapshot-backup
jfraeys/docs/roadmap-state-and-backups
jfraeys/docs/ovh-monthly-billing
jfraeys/docs/ovh-migration-runbook
jfraeys/fix/cloud-init-history-cmd
jfraeys/fix/cloud-init-dpkg-lock
jfraeys/fix/plan-prod-desec-retry
jfraeys/style/format-scripts-lib
jfraeys/feat/forgejo-auto-theme
jfraeys/docs/fold-rollback-prune-future
jfraeys/chore/remove-ansible-profile-cfg
jfraeys/refactor/shared-test-log-libs
jfraeys/docs/stale-references
jfraeys/refactor/infra-lib
chore/rotate-remove-tf-defaults-20261015-183007
jfraeys/docs/workspace-tfvars-and-sizes
jfraeys/feat/linode-node-types
jfraeys/fix/workspace-default-var-file
jfraeys/chore/ovh-prod-tfvars-phase1
jfraeys/docs/dr-ovh-to-linode
jfraeys/fix/runner-stop-before-reuse
jfraeys/feat/linode-dr-workspace
jfraeys/fix/scripts-use-repo-venv
jfraeys/feat/ovh-per-node-flavor
jfraeys/docs/secrets-sync-apply
chore/rotate-linode-token-20261015-125422
chore/rotate-wave3-20261015-124048
jfraeys/fix/vault-pr-key-names
jfraeys/chore/wave3-policy
chore/rotate-lldap-20261015-104606
jfraeys/feat/secrets-get
jfraeys/feat/approve-retry
chore/rotate-ci-ephemeral-wakeup-20261015-095624
chore/rotate-headscale-20261015-093925
chore/rotate-forgejo-20261014-185806
jfraeys/fix/vault-pr-hook-venv
jfraeys/docs/approve-everywhere
jfraeys/feat/approve-carry-over
jfraeys/fix/rotate-forgejo-headscale
chore/rotate-forgejo-20261014-173427
chore/rotate-forgejo-20261014-173353
jfraeys/chore/secrets-rotation-policy
jfraeys/chore/scripts-layout
jfraeys/feat/infra-approve
jfraeys/fix/vaultwarden-admin-token-dollar
jfraeys/feat/infra-one-front-door
jfraeys/feat/vault-change-pr
chore/rotate-vaultwarden-2026-10-14
jfraeys/fix/rotate-playbook-replay-vars
jfraeys/fix/ephemeral-runner-registration-leak
jfraeys/chore/retire-p7-wizard-and-stale-backup
jfraeys/fix/wizard-offline-dir-prompt
jfraeys/feat/project-keys-wizard
jfraeys/chore/manifest-rotated-2026-10-13
jfraeys/chore/remove-hugo-docs
jfraeys/fix/watchtower-maintained-fork
jfraeys/fix/bump-critical-cve-images
jfraeys/fix/connectivity-probe-retries
jfraeys/fix/trivy-report-keeps-failures
jfraeys/fix/journald-check-post-deploy
jfraeys/feat/infra-lifecycle-metrics
jfraeys/docs/wakeup-webhook-events
jfraeys/fix/staging-diag-traefik-acme
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
jfraeys/infra#121
No description provided.