feat(ops): tool version registry, monthly upstream survey, and the update/security runbook (stacked on #100) #105

Merged
jfraeys merged 2 commits from jfraeys/feat/tool-version-procedure into main 2026-10-13 13:03:39 -04:00
Owner

Merge #98, #99, #100 first (stacked: this reuses #100's issue helper and lint step).

There was no repeatable way to learn that a pinned tool had fallen behind. The existing safety net, Watchtower, cannot see exact-pinned images and tells nobody (no notification URL on either instance): Vaultwarden fell three releases behind including a High advisory, Authelia 26 patch releases. Checked on production 2026-10-13: both instances run hourly without failures, the notify instance logged 'Found new traefik:v3 image' every hour for ~4 days with nowhere to send it, and upstream containrrr/watchtower was archived on 2025-12-17 (fork nicholas-fedor/watchtower is maintained).

  • docs/tool-versions.yml: registry of 16 pinned tools (file + regex for each pin, upstream, tier, stateful, checksum flag, notes) plus the floating tags.
  • scripts/check-tool-versions.py: reads every pin from the repo, queries the upstream release APIs, reports latest-in-my-major-line vs latest-overall, the gap (patch/minor/major), and upstreams that are archived. Exit 1 if a pin regex stops matching exactly one version or a tool's pins disagree (the registry must follow the code), 3 if an upstream cannot be queried; outdated is reported, not failed. Real run today: watchtower archived; anubis, cadvisor, forgejo-runner, node-exporter, opentofu minor; authelia, crowdsec, fail2ban, forgejo, headscale, vaultwarden patch; sops, gitleaks, docker-bench, postfix current.
  • .forgejo/workflows/tool-versions.yml: runs weekly (Mondays 14:00 UTC), keeps one issue per month 'Tool versions report YYYY-MM' refreshed each week; closing it means seen, and the next run opens a fresh one.
  • docs/runbooks/updating-tool-versions.md: the protocol: inputs, severity deadlines (High/Critical on an exposed service 48h, Medium 14d, patches monthly, minors quarterly, majors planned, archived within a quarter), the 10-step procedure, what staging does and does not prove (a fresh install cannot test an in-place data migration), backup before stateful services, one-service-at-a-time order, rollback (revert the pin, or restore the snapshot after a migration), special cases (OpenTofu, the runner, ansible-core, Debian, Watchtower) and the known gaps.
  • docs/update-and-patch-policy.md: links the procedure and records the Watchtower findings.
  • 8 tests (scripts/tests/test_tool_versions.py) in the lint workflow, including one that checks the real registry against the real repo so a changed pin format fails CI instead of silently dropping the tool from the report. The tests caught a real bug in my first classifier (a pin ahead of the latest list was reported as a minor gap); fixed.

Companion PRs: #103 (container patch bumps: Vaultwarden 1.37.4 with a High advisory, Forgejo 16.0.6, Headscale 0.29.5, Authelia 4.39.28, CrowdSec 1.8.1) and #104 (OpenTofu 1.12.7).

Not done here, needs your decision: whether to configure a Watchtower notification URL (e.g. mail through the internal Postfix) or retire the notify instance and rely on this report, and whether to move to the maintained fork.

🤖 Generated with Claude Code

**Merge #98, #99, #100 first** (stacked: this reuses #100's issue helper and lint step). There was no repeatable way to learn that a pinned tool had fallen behind. The existing safety net, Watchtower, cannot see exact-pinned images and tells nobody (no notification URL on either instance): Vaultwarden fell three releases behind including a High advisory, Authelia 26 patch releases. Checked on production 2026-10-13: both instances run hourly without failures, the notify instance logged 'Found new traefik:v3 image' every hour for ~4 days with nowhere to send it, and upstream containrrr/watchtower was archived on 2025-12-17 (fork nicholas-fedor/watchtower is maintained). - `docs/tool-versions.yml`: registry of 16 pinned tools (file + regex for each pin, upstream, tier, stateful, checksum flag, notes) plus the floating tags. - `scripts/check-tool-versions.py`: reads every pin from the repo, queries the upstream release APIs, reports latest-in-my-major-line vs latest-overall, the gap (patch/minor/major), and upstreams that are archived. Exit 1 if a pin regex stops matching exactly one version or a tool's pins disagree (the registry must follow the code), 3 if an upstream cannot be queried; outdated is reported, not failed. Real run today: watchtower archived; anubis, cadvisor, forgejo-runner, node-exporter, opentofu minor; authelia, crowdsec, fail2ban, forgejo, headscale, vaultwarden patch; sops, gitleaks, docker-bench, postfix current. - `.forgejo/workflows/tool-versions.yml`: runs weekly (Mondays 14:00 UTC), keeps one issue per month 'Tool versions report YYYY-MM' refreshed each week; closing it means seen, and the next run opens a fresh one. - `docs/runbooks/updating-tool-versions.md`: the protocol: inputs, severity deadlines (High/Critical on an exposed service 48h, Medium 14d, patches monthly, minors quarterly, majors planned, archived within a quarter), the 10-step procedure, what staging does and does not prove (a fresh install cannot test an in-place data migration), backup before stateful services, one-service-at-a-time order, rollback (revert the pin, or restore the snapshot after a migration), special cases (OpenTofu, the runner, ansible-core, Debian, Watchtower) and the known gaps. - `docs/update-and-patch-policy.md`: links the procedure and records the Watchtower findings. - 8 tests (scripts/tests/test_tool_versions.py) in the lint workflow, including one that checks the real registry against the real repo so a changed pin format fails CI instead of silently dropping the tool from the report. The tests caught a real bug in my first classifier (a pin ahead of the latest list was reported as a minor gap); fixed. Companion PRs: #103 (container patch bumps: Vaultwarden 1.37.4 with a High advisory, Forgejo 16.0.6, Headscale 0.29.5, Authelia 4.39.28, CrowdSec 1.8.1) and #104 (OpenTofu 1.12.7). Not done here, needs your decision: whether to configure a Watchtower notification URL (e.g. mail through the internal Postfix) or retire the notify instance and rely on this report, and whether to move to the maintained fork. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
feat(secrets): one expiry classifier and a manifest-only expiry report
All checks were successful
Lint / lint (pull_request) Successful in 1m13s
Secret scan / scan (pull_request) Successful in 14s
44ba182f43
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
feat(secrets): rotation lock for vault read-modify-write cycles
All checks were successful
Lint / lint (pull_request) Successful in 1m58s
Secret scan / scan (pull_request) Successful in 12s
b35100b7dc
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
feat(secrets): monthly expiry report workflow and the manifest facts the repo knows
All checks were successful
Lint / lint (pull_request) Successful in 1m42s
Secret scan / scan (pull_request) Successful in 24s
d05d8e0b96
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
feat(ops): tool version registry, monthly upstream survey, and the update/security runbook
Some checks failed
Lint / lint (pull_request) Has been cancelled
Secret scan / scan (pull_request) Has been cancelled
86986b754f
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
ops: tool versions report runs weekly (Mondays), one issue per month refreshed each week
All checks were successful
Lint / lint (pull_request) Successful in 1m47s
Secret scan / scan (pull_request) Successful in 17s
1e4d538f6d
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
jfraeys/infra!105
No description provided.