Cancelling a running job leaves the ephemeral runner executing it and blocks the queue #192
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Found 2026-10-15, live. Cancelling a staging-validate run from the UI while it was running on an ephemeral runner did not stop it. The cancelled task kept executing on the runner VM and held the single runner, so every queued job waited behind it.
Timeline (provisioner journal on
services, runner VMci-runner-staging):job 1191 finished with status=cancelled->try_reuse_or_cleanup->reusing VM for job id=1195._reuse_vm_for_jobdeletes the previous runner registration and the cycle moves toregistering.job 1195 still in progress (status=waiting)every tick. Job 1195 never starts.forgejo-runner one-jobprocess (PID 17173) is still alive with./setup --workspace linode-stagingandansible-playbook playbooks/services.ymlbeneath it, and the runner log repeatsReportLog/ReportState error: unauthenticated: unregistered runneronce a second.Cause (from the code and the evidence above):
try_reuse_or_cleanuptreats Forgejo's job statuscancelledas proof that the runner is idle, but a cancel only changes the server-side status; the one-job runner process keeps going. Reuse then deletes that runner's registration, so the process can no longer report or learn about the cancel (hence "unregistered runner"), and starting the one-job unit again is a no-op while the old process is alive, so the new registration is never served. The check that would catch this,runner_service_active(ip), is used for the re-run case inreconcile_runningbut not before reuse after acancelledstatus. Related earlier fixes: the comments inreconcile_running(2026-08-27, 2026-09-19) describe the same family of problem.Side effects: the workflow's
Cleanup on failurestep isif: failure(), so a cancelled staging run never destroys itsweb-staging/services-stagingnodes (they were left running, billed); the task keeps deploying to them after the cancel.Fix ideas:
try_reuse_or_cleanup, when the finished job's status iscancelled(or any timerunner_service_activeis true), stop the runner unit on the VM (systemctl stop forgejo-runner-ephemeral, killing its process tree) before deleting the registration and reusing the VM; if it cannot be stopped, destroy the VM instead of reusing it.Start ephemeral runner (one-job)task restart rather than start when the unit is already active.roles/ci_ephemeral_provisioner/tests/test_runner_cleanup.py: a cancelled job whose runner is still active must not be reused with a live process.if: failure() || cancelled()to the staging workflow's cleanup step so a cancel also destroys the staging nodes (the runner has to survive long enough to run it; the stop above would prevent that, so decide which wins).Workaround until fixed: do not cancel a running staging run; if you must, restart
forgejo-runner-ephemeral.serviceon the runner VM afterwards, then check the Linode dashboard for leftover-stagingnodes and./infra unlock <id>if the state lock is held.Related: #148 (runner cleanup), #121 (backlog).
Fixed in #194 (merged): the provisioner stops a still-running runner before reusing its VM after a cancelled job, and destroys the VM if it cannot. Not yet deployed: ./infra deploy --tags ci_ephemeral_provisioner --limit services. Still open from the fix ideas: whether the staging workflow's cleanup should also run on cancel.