jfraeys/infra
Watch
1
0
Fork
You've already forked infra
 0
Code Issues 21 Pull requests 4 Projects Releases Packages Wiki Activity Actions

fix(ci): delete every ephemeral runner registration, not just the last one per cycle #148

Merged
jfraeys merged 1 commit from jfraeys/fix/ephemeral-runner-registration-leak into main 2026-10-14 11:36:19 -04:00
Conversation 0 Commits 1 Files changed 2 +117
jfraeys commented 2026-10-14 11:07:57 -04:00
Owner
Copy link

The documented end state of every provisioner cycle is "no VM, no runner registration, no cycle.json". Checked on 2026-10-14: VMs and cycle.json converge correctly (the 10-job-cap VM was destroyed, tofu destroy clean), but 5 stale ephemeral runner registrations remain in Forgejo (ids 514, 516, 518 from one cycle, 424 from ~1 day earlier, 106 from ~11 days earlier; all ephemeral-*, offline, last_online 0; the only live one, id 526, belongs to the running cycle).

Cause. A reused VM registers one ephemeral runner per job. _reuse_vm_for_job sets cycle["runner_id"] = None, so a previous registration that never consumed itself is forgotten, and cleanup_cycle only deletes the last runner_id. Each leaked registration is a valid runner credential that was never revoked (the tokens only ever lived on VMs that are now destroyed, so the practical risk is low, but it should converge to none).

Fix.

  • _reuse_vm_for_job: delete the previous registration (if it still exists) before forgetting it. Reuse only happens after the previous runner has exited.
  • cleanup_cycle: after a successful tofu destroy, sweep every offline ephemeral-* registration (best effort, never blocks convergence, skipped while a VM may still exist). This also collects the old orphans on the first cycle after deploy.
  • 6 new tests (test_runner_cleanup.py) with an in-memory fake of the admin runners API: sweep deletes only offline ephemeral-* runners; reuse deletes an unconsumed previous registration and leaves a consumed one alone; cleanup sweeps after destroy and skips the sweep when destroy failed. pytest roles/ci_ephemeral_provisioner/tests: 23 passed.

Unverified: the sweep relies on the admin runners list returning status: "offline" for stale runners (the Forgejo ActionRunner field). I could not call that API from my token (needs read:admin). If the field is absent the sweep deletes nothing (safe default), but then the first deployed cycle should be checked.

Deploy: touches roles/ci_ephemeral_provisioner (deploy with ./infra deploy --tags ci_ephemeral_provisioner --limit services; use ./infra, not a bare ansible-playbook, since the Tailscale key is minted there). Until then, delete ids 514, 516, 518, 424, 106 by hand in Site Administration > Actions > Runners.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Ge5tovcwdEUBtPwUktsZZZ

The documented end state of every provisioner cycle is "no VM, no runner registration, no cycle.json". Checked on 2026-10-14: VMs and cycle.json converge correctly (the 10-job-cap VM was destroyed, `tofu destroy` clean), but **5 stale ephemeral runner registrations remain** in Forgejo (ids 514, 516, 518 from one cycle, 424 from ~1 day earlier, 106 from ~11 days earlier; all `ephemeral-*`, offline, `last_online` 0; the only live one, id 526, belongs to the running cycle). **Cause.** A reused VM registers one ephemeral runner per job. `_reuse_vm_for_job` sets `cycle["runner_id"] = None`, so a previous registration that never consumed itself is forgotten, and `cleanup_cycle` only deletes the last `runner_id`. Each leaked registration is a valid runner credential that was never revoked (the tokens only ever lived on VMs that are now destroyed, so the practical risk is low, but it should converge to none). **Fix.** - `_reuse_vm_for_job`: delete the previous registration (if it still exists) before forgetting it. Reuse only happens after the previous runner has exited. - `cleanup_cycle`: after a successful `tofu destroy`, sweep every offline `ephemeral-*` registration (best effort, never blocks convergence, skipped while a VM may still exist). This also collects the old orphans on the first cycle after deploy. - 6 new tests (`test_runner_cleanup.py`) with an in-memory fake of the admin runners API: sweep deletes only offline `ephemeral-*` runners; reuse deletes an unconsumed previous registration and leaves a consumed one alone; cleanup sweeps after destroy and skips the sweep when destroy failed. `pytest roles/ci_ephemeral_provisioner/tests`: 23 passed. **Unverified:** the sweep relies on the admin runners list returning `status: "offline"` for stale runners (the Forgejo `ActionRunner` field). I could not call that API from my token (needs `read:admin`). If the field is absent the sweep deletes nothing (safe default), but then the first deployed cycle should be checked. **Deploy:** touches `roles/ci_ephemeral_provisioner` (deploy with `./infra deploy --tags ci_ephemeral_provisioner --limit services`; use `./infra`, not a bare ansible-playbook, since the Tailscale key is minted there). Until then, delete ids 514, 516, 518, 424, 106 by hand in Site Administration > Actions > Runners. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Ge5tovcwdEUBtPwUktsZZZ
fix(ci): delete every ephemeral runner registration, not just the last one per cycle
All checks were successful
Lint / lint (pull_request) Successful in 2m21s
Details
Secret scan / scan (pull_request) Successful in 17s
Details
f8099ffde9
 
A reused VM registers one ephemeral runner per job, and _reuse_vm_for_job reset
cycle["runner_id"] to None, so an earlier registration that never consumed
itself was forgotten and never deleted (valid, never-revoked runner credentials;
5 were found on 2026-10-14, the oldest ~11 days). cleanup_cycle only knew the
last one.

- _reuse_vm_for_job deletes the previous registration (if it still exists)
  before forgetting it.
- cleanup_cycle sweeps every offline ephemeral-* registration after the VM is
  destroyed (best effort; the sweep is skipped while a VM may still exist).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ge5tovcwdEUBtPwUktsZZZ
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
jfraeys/infra!148
No description provided.