feat(ci): alert when CI jobs wait too long or the provisioner stops running #85

Merged

jfraeys merged 1 commit from jfraeys/feat/provisioner-queue-stall-alert into main 2026-10-12 23:15:03 -04:00

Conversation 0 Commits 1 Files changed 3 +63 -3

jfraeys commented 2026-10-12 22:48:11 -04:00

Owner

Copy link

Today's 25-minute CI stall (recycled-IP host-key failure) produced no signal: runner_service_active deliberately returns 'alive' when it cannot check, so the cycle stayed 'running' and neither the breaker nor any alert fired.

• Provisioner: each tick writes ci_ephemeral_provisioner_oldest_queued_job_age_seconds (first-seen times kept in queue-seen.json; an API error such as a 429 keeps the previous ages rather than resetting them) and ci_ephemeral_provisioner_last_pass_timestamp_seconds into the existing textfile. Never fatal.
• alerts.yml.j2: CIEphemeralJobsWaitingTooLong (>40 min for 5m, warning; PR checks can legitimately wait ~30 min behind a staging run) and CIEphemeralProvisionerNotRunning (>10 min since the last pass, critical).
• The 'failure to check = assume alive' rule is unchanged (runaway-registration incident).

Verified with a mocked Forgejo API: fresh job 0, 20 min old 1200, 429 keeps 1200, drained queue 0; alert template renders as valid YAML (promtool is not installed locally, so expression syntax mirrors the existing rules).

Takes effect after the provisioner role and Prometheus alert rules are deployed.

Plan: L5 of the loose-ends track.

🤖 Generated with Claude Code

Today's 25-minute CI stall (recycled-IP host-key failure) produced no signal: runner_service_active deliberately returns 'alive' when it cannot check, so the cycle stayed 'running' and neither the breaker nor any alert fired. - Provisioner: each tick writes ci_ephemeral_provisioner_oldest_queued_job_age_seconds (first-seen times kept in queue-seen.json; an API error such as a 429 keeps the previous ages rather than resetting them) and ci_ephemeral_provisioner_last_pass_timestamp_seconds into the existing textfile. Never fatal. - alerts.yml.j2: CIEphemeralJobsWaitingTooLong (>40 min for 5m, warning; PR checks can legitimately wait ~30 min behind a staging run) and CIEphemeralProvisionerNotRunning (>10 min since the last pass, critical). - The 'failure to check = assume alive' rule is unchanged (runaway-registration incident). Verified with a mocked Forgejo API: fresh job 0, 20 min old 1200, 429 keeps 1200, drained queue 0; alert template renders as valid YAML (promtool is not installed locally, so expression syntax mirrors the existing rules). Takes effect after the provisioner role and Prometheus alert rules are deployed. Plan: L5 of the loose-ends track. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

jfraeys added 1 commit 2026-10-12 22:48:12 -04:00

feat(ci): alert when CI jobs wait too long or the provisioner stops running ... 6e606e9036

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

jfraeys merged commit faa798c50b into main 2026-10-12 23:15:03 -04:00

jfraeys referenced this pull request from a commit 2026-10-12 23:15:06 -04:00

Merge pull request 'feat(ci): alert when CI jobs wait too long or the provisioner stops running' (#85) from jfraeys/feat/provisioner-queue-stall-alert into main

jfraeys referenced this pull request 2026-10-13 09:21:39 -04:00

fix(ci): queue-age metric resets when the queue drains (Forgejo returns null) #92

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

jfraeys/infra!85

No description provided.