fix(ci): a task Forgejo no longer lists no longer stalls the runner cycle

jfraeys commented

2026-10-13 13:33:46 -04:00

Owner

Found live 2026-10-13: cycle 1791909394 tracked job 902/task 642; the task was absent from Forgejo's tasks listing (a run cancelled mid-job), get_task_status returned None, and the cycle waited out job_deadline (~55 min) while the whole queue sat in waiting.

get_task_status now pages the listing (50 per page, capped at 20 pages) instead of reading only the newest 50.
Returns TASK_NOT_LISTED when the whole listing was read and the task is absent, None only for an API failure (the two were conflated).
reconcile_running: on TASK_NOT_LISTED it asks the VM (runner_service_active). Exited runner -> finished and cleaned up; live runner (or an unreachable check) -> keeps waiting, as before. An API error never probes the VM.
7 pytest cases (paging, exact-page boundary, API error vs absence, dead/live runner, no VM probe on API error); mutation-checked by disabling the new branch.

Deploy: scripts/deploy-services.sh --tags ci_ephemeral_provisioner --limit services, only when nothing is queued (it restarts the provisioner). Not exercised against a real stuck cycle.

🤖 Generated with Claude Code

Found live 2026-10-13: cycle 1791909394 tracked job 902/task 642; the task was absent from Forgejo's tasks listing (a run cancelled mid-job), get_task_status returned None, and the cycle waited out job_deadline (~55 min) while the whole queue sat in waiting. - get_task_status now pages the listing (50 per page, capped at 20 pages) instead of reading only the newest 50. - Returns TASK_NOT_LISTED when the whole listing was read and the task is absent, None only for an API failure (the two were conflated). - reconcile_running: on TASK_NOT_LISTED it asks the VM (runner_service_active). Exited runner -> finished and cleaned up; live runner (or an unreachable check) -> keeps waiting, as before. An API error never probes the VM. - 7 pytest cases (paging, exact-page boundary, API error vs absence, dead/live runner, no VM probe on API error); mutation-checked by disabling the new branch. Deploy: scripts/deploy-services.sh --tags ci_ephemeral_provisioner --limit services, only when nothing is queued (it restarts the provisioner). Not exercised against a real stuck cycle. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

jfraeys added 1 commit

2026-10-13 13:33:46 -04:00

fix(ci): a task Forgejo no longer lists no longer stalls the runner cycle until its deadline

Lint / lint (pull_request) Successful in 1m51s

Details

Secret scan / scan (pull_request) Successful in 13s

Details

9cf2de1f71

get_task_status read only the newest 50 tasks and returned None for both an API error and a missing task, so a cycle
whose job vanished (e.g. a run cancelled mid-job) waited ~55 min. It now pages the listing, distinguishes 'not listed'
from an error, and asks the VM whether the one-job runner is still alive.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

jfraeys merged commit bb1cf2fba1 into main

2026-10-13 13:53:33 -04:00

jfraeys referenced this pull request from a commit

2026-10-13 13:53:34 -04:00

Merge pull request 'fix(ci): a task Forgejo no longer lists no longer stalls the runner cycle' (#111) from jfraeys/fix/provisioner-missing-task into main

jfraeys referenced this pull request

2026-10-13 14:25:41 -04:00

Next: immediate steps (merge, deploy, verify) after the 2026-10-13 session #116

jfraeys referenced this pull request

2026-10-13 14:25:44 -04:00

Infra backlog: capacity, CI verification, alerts, older loose ends #121

jfraeys referenced this pull request