jfraeys/infra
Watch
1
0
Fork
You've already forked infra
 0
Code Issues 21 Pull requests 4 Projects Releases Packages Wiki Activity Actions

feat(ops): secret expiry and tool version drift as Prometheus metrics and alerts (stacked on #105) #106

Closed
jfraeys wants to merge 0 commits from jfraeys/feat/infra-lifecycle-metrics into main
pull from: jfraeys/feat/infra-lifecycle-metrics
merge into: jfraeys:main
Conversation 0 Commits - Files changed -
jfraeys commented 2026-10-13 12:45:48 -04:00
Owner
Copy link

Merge #98, #99, #100, #105 first (stacked: reuses the expiry classifier, the tool registry and the survey script).

Expiry and version drift only ever reached a monthly/weekly issue. This puts them where Prometheus and Alertmanager already are, so they page through your existing routes (critical/warning/info to email).

Data (new role roles/infra_metrics, wired into services.yml after prometheus):

  • Static metrics generated at deploy time by scripts/export-infra-metrics.py (run by scripts/deploy-services.sh for the production inventory; a failure only warns): when each secret expires or is due for rotation (manifest only, no values; 13 today have computable dates), unknown-count, the pinned version of every registry tool, last-deploy time and commit. Pins and rotation dates only change through git and a deploy, so a file written at deploy time is current by construction.
  • A weekly systemd timer on the services host asks each upstream for its latest stable release (public APIs, no credentials) and writes latest / latest-in-pinned-major / archived metrics; an unreachable upstream is counted, never stored as a stale value. Pre-release tags (-rc, -beta) are never 'latest' (found and fixed while testing).
  • Versions are encoded major1000000+minor1000+patch so PromQL compares numbers.

Alerts (group infra_lifecycle, 11 rules): SecretExpired (critical), SecretRotationOverdue, SecretExpiresSoon (warning), SecretRotationDueSoon, SecretsExpiryUnknown (info), ToolPatchBehind (warning after 3d), ToolMinorBehind (info, 14d), ToolMajorBehind (info, 30d), ToolUpstreamArchived (warning), InfraToolCheckBroken, InfraNotDeployedRecently. docs/runbooks/updating-tool-versions.md section 8 lists each alert and what to do.

Tests: promtool (Prometheus's own binary, via Docker) check passes on all 34 rules, and 8 simulated-time scenarios verify firing times, for-durations, labels and the negative cases (a patch gap does not trigger the minor alert, a current pin never alerts, past-due does not also fire 'expires soon'). Mutation-checked: breaking one expectation makes the run fail. 14 Python tests for the exporter and the host script (encoding, ordering, in-line vs overall, pre-releases, unreachable upstream, stale registry refuses to export, real inputs). shellcheck -S error clean; ansible-lint clean on the role and services.yml. The alert rule test skips itself where Docker is unavailable (CI runners), so it runs locally.

Not exercised on a real node: the role (timer, ownership, first run) and the live Prometheus scrape. After merging and deploying (scripts/deploy-services.sh --tags infra_metrics,prometheus --limit services, gated), check on the host: systemctl list-timers infra-tool-latest.timer, ls /var/lib/node_exporter/textfile_collector/infra_*, and query infra_tool_pinned_version in Prometheus. Staging will run the role on a fresh node (without the static file).

Next PR in this line: an hourly check for unexpected public ports (the LLDAP-class exposure), which needs the same role.

🤖 Generated with Claude Code

**Merge #98, #99, #100, #105 first** (stacked: reuses the expiry classifier, the tool registry and the survey script). Expiry and version drift only ever reached a monthly/weekly issue. This puts them where Prometheus and Alertmanager already are, so they page through your existing routes (critical/warning/info to email). Data (new role roles/infra_metrics, wired into services.yml after prometheus): - Static metrics generated at deploy time by scripts/export-infra-metrics.py (run by scripts/deploy-services.sh for the production inventory; a failure only warns): when each secret expires or is due for rotation (manifest only, no values; 13 today have computable dates), unknown-count, the pinned version of every registry tool, last-deploy time and commit. Pins and rotation dates only change through git and a deploy, so a file written at deploy time is current by construction. - A weekly systemd timer on the services host asks each upstream for its latest stable release (public APIs, no credentials) and writes latest / latest-in-pinned-major / archived metrics; an unreachable upstream is counted, never stored as a stale value. Pre-release tags (-rc, -beta) are never 'latest' (found and fixed while testing). - Versions are encoded major*1000000+minor*1000+patch so PromQL compares numbers. Alerts (group infra_lifecycle, 11 rules): SecretExpired (critical), SecretRotationOverdue, SecretExpiresSoon (warning), SecretRotationDueSoon, SecretsExpiryUnknown (info), ToolPatchBehind (warning after 3d), ToolMinorBehind (info, 14d), ToolMajorBehind (info, 30d), ToolUpstreamArchived (warning), InfraToolCheckBroken, InfraNotDeployedRecently. docs/runbooks/updating-tool-versions.md section 8 lists each alert and what to do. Tests: promtool (Prometheus's own binary, via Docker) check passes on all 34 rules, and 8 simulated-time scenarios verify firing times, for-durations, labels and the negative cases (a patch gap does not trigger the minor alert, a current pin never alerts, past-due does not also fire 'expires soon'). Mutation-checked: breaking one expectation makes the run fail. 14 Python tests for the exporter and the host script (encoding, ordering, in-line vs overall, pre-releases, unreachable upstream, stale registry refuses to export, real inputs). shellcheck -S error clean; ansible-lint clean on the role and services.yml. The alert rule test skips itself where Docker is unavailable (CI runners), so it runs locally. Not exercised on a real node: the role (timer, ownership, first run) and the live Prometheus scrape. After merging and deploying (`scripts/deploy-services.sh --tags infra_metrics,prometheus --limit services`, gated), check on the host: `systemctl list-timers infra-tool-latest.timer`, `ls /var/lib/node_exporter/textfile_collector/infra_*`, and query infra_tool_pinned_version in Prometheus. Staging will run the role on a fresh node (without the static file). Next PR in this line: an hourly check for unexpected public ports (the LLDAP-class exposure), which needs the same role. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
feat(secrets): one expiry classifier and a manifest-only expiry report
All checks were successful
Lint / lint (pull_request) Successful in 1m13s
Details
Secret scan / scan (pull_request) Successful in 14s
Details
44ba182f43
 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
feat(secrets): rotation lock for vault read-modify-write cycles
All checks were successful
Lint / lint (pull_request) Successful in 1m58s
Details
Secret scan / scan (pull_request) Successful in 12s
Details
b35100b7dc
 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
feat(secrets): monthly expiry report workflow and the manifest facts the repo knows
All checks were successful
Lint / lint (pull_request) Successful in 1m42s
Details
Secret scan / scan (pull_request) Successful in 24s
Details
d05d8e0b96
 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
feat(ops): tool version registry, monthly upstream survey, and the update/security runbook
Some checks failed
Lint / lint (pull_request) Has been cancelled
Details
Secret scan / scan (pull_request) Has been cancelled
Details
86986b754f
 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
ops: tool versions report runs weekly (Mondays), one issue per month refreshed each week
All checks were successful
Lint / lint (pull_request) Successful in 1m47s
Details
Secret scan / scan (pull_request) Successful in 17s
Details
1e4d538f6d
 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
feat(ops): secret expiry and tool version drift as Prometheus metrics and alerts
All checks were successful
Lint / lint (pull_request) Successful in 1m35s
Details
Secret scan / scan (pull_request) Successful in 15s
Details
168d0f4b02
 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
jfraeys closed this pull request 2026-10-13 21:04:41 -04:00
All checks were successful
Lint / lint (pull_request) Successful in 1m35s
Details
Secret scan / scan (pull_request) Successful in 15s
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
jfraeys/infra!106
No description provided.