feat(ci-ephemeral): provisioner circuit breaker and alert

jfraeys commented

2026-10-02 18:59:21 -04:00

Owner

An expired Headscale preauth key made the provisioner create and destroy a billed VM every ~3.5 min indefinitely. Now 3 consecutive provisioning failures pause new cycles for 30 min, then allow one trial cycle; reaching running resets it. Failures count once per cycle; known causes (expired key, rejected SSH key, checksum mismatch) log a fixed hint without echoing output.

Breaker state is exported via node-exporter textfile metrics, and CIEphemeralProvisionerCircuitOpen (warning, for 2m) fires on it through the existing Alertmanager receiver. Tunables: ci_ephemeral_provisioner_breaker_threshold / _cooldown.

Tested: 10 unit tests (pytest roles/ci_ephemeral_provisioner/tests), promtool check rules on alerts.yml. Not run: a real provisioner cycle or alert delivery. Deploy needs services.yml for the provisioner and prometheus roles.

🤖 Generated with Claude Code

An expired Headscale preauth key made the provisioner create and destroy a billed VM every ~3.5 min indefinitely. Now 3 consecutive provisioning failures pause new cycles for 30 min, then allow one trial cycle; reaching running resets it. Failures count once per cycle; known causes (expired key, rejected SSH key, checksum mismatch) log a fixed hint without echoing output. Breaker state is exported via node-exporter textfile metrics, and CIEphemeralProvisionerCircuitOpen (warning, for 2m) fires on it through the existing Alertmanager receiver. Tunables: ci_ephemeral_provisioner_breaker_threshold / _cooldown. Tested: 10 unit tests (pytest roles/ci_ephemeral_provisioner/tests), promtool check rules on alerts.yml. Not run: a real provisioner cycle or alert delivery. Deploy needs services.yml for the provisioner and prometheus roles. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

jfraeys added 2 commits

2026-10-02 18:59:21 -04:00

feat(ci-ephemeral): circuit breaker for repeated provisioning failures 9c83d50b29

An expired Headscale preauth key made the provisioner create and destroy a billed
VM every ~3.5 min indefinitely. Now N consecutive provisioning failures (default 3)
pause new cycles for a cooldown (default 30 min), then allow one trial cycle; a cycle
reaching running resets it. Failures count once per cycle, and known causes (expired
key, rejected SSH key, checksum mismatch) log a fixed hint without echoing output.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(ci-ephemeral): alert when the provisioner circuit breaker opens 7ee8144f9b

The breaker exports circuit_open/consecutive_failures via node-exporter's textfile
collector; CIEphemeralProvisionerCircuitOpen (warning, for 2m) fires on it and routes
through the existing Alertmanager receiver. Metrics rewrite every tick so clearing
breaker.json clears the alert; a metrics write failure never blocks provisioning.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

jfraeys added 1 commit

2026-10-02 19:01:56 -04:00

Merge branch 'main' into feat/provisioner-circuit-breaker

Lint / lint (pull_request) Failing after 15m1s

Details

44167f4bf6

# Conflicts:
#	roles/ci_ephemeral_provisioner/tasks/main.yml

jfraeys merged commit 092af98d96 into main

2026-10-02 19:02:09 -04:00

jfraeys referenced this pull request from a commit

2026-10-02 19:02:13 -04:00

Merge pull request 'feat(ci-ephemeral): provisioner circuit breaker and alert' (#9) from feat/provisioner-circuit-breaker into main

jfraeys referenced this pull request

2026-10-02 20:12:24 -04:00

fix(lint): run on the ephemeral runner without sudo or system pip #10

Rows
Columns

feat(ci-ephemeral): provisioner circuit breaker and alert #9