fix(ci): queue-age metric resets when the queue drains (Forgejo returns null)

jfraeys commented

2026-10-13 09:21:39 -04:00

Owner

Bug in #85, found while verifying it on production: GET /admin/runners/jobs answers an empty queue with 200 and a JSON null, not []. queue_metrics() only updated its state for a list, so after a job queued and then drained, its first-seen time stayed in queue-seen.json forever and oldest_queued_job_age_seconds would climb until CIEphemeralJobsWaitingTooLong fired falsely after 40 minutes. Not triggered yet on production (the queue has been empty since the deploy).

Treat 200 + null as an empty queue. My original test mocked [] and missed this; the new check runs the real shape: job queued 50 min -> 3000, drained (null) -> 0, a 429 still keeps the previous state.

Takes effect on the next deploy of the ci_ephemeral_provisioner role: scripts/deploy-services.sh --tags ci_ephemeral_provisioner --limit services.

🤖 Generated with Claude Code

Bug in #85, found while verifying it on production: GET /admin/runners/jobs answers an empty queue with 200 and a JSON null, not []. queue_metrics() only updated its state for a list, so after a job queued and then drained, its first-seen time stayed in queue-seen.json forever and oldest_queued_job_age_seconds would climb until CIEphemeralJobsWaitingTooLong fired falsely after 40 minutes. Not triggered yet on production (the queue has been empty since the deploy). Treat 200 + null as an empty queue. My original test mocked [] and missed this; the new check runs the real shape: job queued 50 min -> 3000, drained (null) -> 0, a 429 still keeps the previous state. Takes effect on the next deploy of the ci_ephemeral_provisioner role: scripts/deploy-services.sh --tags ci_ephemeral_provisioner --limit services. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

jfraeys added 1 commit

2026-10-13 09:21:40 -04:00

fix(ci): queue-age metric resets when Forgejo answers an empty queue with null

Lint / lint (pull_request) Successful in 1m44s

Details

Secret scan / scan (pull_request) Successful in 13s

Details

eedede5beb

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

jfraeys merged commit d3437b2f1e into main

2026-10-13 09:25:10 -04:00

jfraeys referenced this pull request from a commit

2026-10-13 09:25:14 -04:00

Merge pull request 'fix(ci): queue-age metric resets when the queue drains (Forgejo returns null)' (#92) from jfraeys/fix/queue-metrics-null-queue into main

Rows
Columns

fix(ci): queue-age metric resets when the queue drains (Forgejo returns null) #92