jfraeys/infra
Watch
1
0
Fork
You've already forked infra
 0
Code Issues 21 Pull requests 4 Projects Releases Packages Wiki Activity Actions

fix(ci): ephemeral runners reach Forgejo's public name over the tailnet #280

Merged
jfraeys merged 2 commits from jfraeys/fix/runner-forgejo-over-tailnet into main 2026-10-18 17:49:12 -04:00
Conversation 0 Commits 2 Files changed 12 +74 -4
jfraeys commented 2026-10-18 16:00:16 -04:00
Owner
Copy link

Fixes the coaching_site `Build artifacts` failure (and every `upload-artifact` from an ephemeral runner).

Cause (from the coaching_site diagnostic run and Forgejo's own log). The runner talks to Forgejo over the tailnet (`ACTIONS_RUNTIME_URL=http://100.64.0.3:3000/...`), so the artifact-container `POST` works. Forgejo then returns an upload URL built from its public address (`ROOT_URL`, `https://git.jfraeys.com/...`). The CI VM cannot reach that name (a plain `curl` timed out after 20 s; the name has both A and AAAA records), so no chunk is ever sent and the finalize `PATCH` fails with `lstat .../actions_artifacts/tmp: no such file or directory` (500). Even a 1 KB artifact fails. Traefik never logs it; Forgejo logs only the POST and the failing PATCHes. Not memory, disk, bans, size, the cache or tag:ci (the failure predates all of them, e.g. runs 748 on Oct 15, 1021 on Oct 17, 1046 on Oct 18).

Change.

  • `roles/forgejo_runner_native/tasks/forgejo_hosts.yml` (tag `base`, ephemeral runners only): maps Forgejo's public name to the services tailnet IP in `/etc/hosts` (idempotent, replaces an old mapping, regex-escaped). The provisioner passes the name via `CI_EPHEMERAL_FORGEJO_HOSTNAME` (role default `forgejo_hostname`, same pattern as `vpn_hostname`). Empty hostname = no mapping.
  • ACL: the `tag:ci` rule gains `tag:services:443` (Traefik, which already holds the real certificate for `git.jfraeys.com`, so TLS validates normally).
  • Tests: `test-runner-forgejo-hosts.sh` (6 cases, real Ansible against a temp hosts file) and the ACL test, both in lint.
  • App repos untouched (the deployment contract keeps tailnet addresses out of them).

Not found: why the public name is unreachable from a CI VM at all (both families). This works around it for runners; run `curl -4 -sv` / `curl -6 -sv` against `https://git.jfraeys.com/api/healthz` from a runner if you want that answered.

Rollout (two deploys, in this order)

  1. merge, staging-validate green, `./infra approve`, `git pull`.
  2. `./infra deploy --tags headscale --limit services` (tag:ci gains :443; harmless alone).
  3. `scripts/release/deploy-services.sh --tags ci_ephemeral_provisioner --limit services` (env file + script + playbook copy; restart the provisioner only when nothing is queued). New VMs get the mapping; running ones don't.
  4. Dispatch the coaching_site diagnostic or `ci.yml`: `getent hosts git.jfraeys.com` on the runner shows `100.64.0.3`, all four artifacts upload.

Rollback: revert and redeploy both steps; nothing else depends on the entry.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Ge5tovcwdEUBtPwUktsZZZ

Fixes the coaching_site \`Build artifacts\` failure (and every \`upload-artifact\` from an ephemeral runner). **Cause (from the coaching_site diagnostic run and Forgejo's own log).** The runner talks to Forgejo over the tailnet (\`ACTIONS_RUNTIME_URL=http://100.64.0.3:3000/...\`), so the artifact-container \`POST\` works. Forgejo then returns an upload URL built from its public address (\`ROOT_URL\`, \`https://git.jfraeys.com/...\`). The CI VM cannot reach that name (a plain \`curl\` timed out after 20 s; the name has both A and AAAA records), so no chunk is ever sent and the finalize \`PATCH\` fails with \`lstat .../actions_artifacts/tmp<run>: no such file or directory\` (500). Even a 1 KB artifact fails. Traefik never logs it; Forgejo logs only the POST and the failing PATCHes. Not memory, disk, bans, size, the cache or tag:ci (the failure predates all of them, e.g. runs 748 on Oct 15, 1021 on Oct 17, 1046 on Oct 18). **Change.** - \`roles/forgejo_runner_native/tasks/forgejo_hosts.yml\` (tag \`base\`, ephemeral runners only): maps Forgejo's public name to the services tailnet IP in \`/etc/hosts\` (idempotent, replaces an old mapping, regex-escaped). The provisioner passes the name via \`CI_EPHEMERAL_FORGEJO_HOSTNAME\` (role default \`forgejo_hostname\`, same pattern as \`vpn_hostname\`). Empty hostname = no mapping. - ACL: the \`tag:ci\` rule gains \`tag:services:443\` (Traefik, which already holds the real certificate for \`git.jfraeys.com\`, so TLS validates normally). - Tests: \`test-runner-forgejo-hosts.sh\` (6 cases, real Ansible against a temp hosts file) and the ACL test, both in lint. - App repos untouched (the deployment contract keeps tailnet addresses out of them). **Not found:** why the public name is unreachable from a CI VM at all (both families). This works around it for runners; run \`curl -4 -sv\` / \`curl -6 -sv\` against \`https://git.jfraeys.com/api/healthz\` from a runner if you want that answered. ## Rollout (two deploys, in this order) 1. merge, staging-validate green, \`./infra approve\`, \`git pull\`. 2. \`./infra deploy --tags headscale --limit services\` (tag:ci gains :443; harmless alone). 3. \`scripts/release/deploy-services.sh --tags ci_ephemeral_provisioner --limit services\` (env file + script + playbook copy; **restart the provisioner only when nothing is queued**). New VMs get the mapping; running ones don't. 4. Dispatch the coaching_site diagnostic or \`ci.yml\`: \`getent hosts git.jfraeys.com\` on the runner shows \`100.64.0.3\`, all four artifacts upload. Rollback: revert and redeploy both steps; nothing else depends on the entry. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Ge5tovcwdEUBtPwUktsZZZ
Forgejo hands the artifact upload action a URL built from its public address; CI VMs cannot reach it, so
every artifact upload failed (0 bytes arrived, finalize 500). The runner role maps the public name to the
services tailnet IP in /etc/hosts (ephemeral only) and the tag:ci ACL rule gains services:443.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ge5tovcwdEUBtPwUktsZZZ
docs(runner): put the hosts-mapping decision in the existing decisions section
All checks were successful
Lint / lint (pull_request) Successful in 3m33s
Details
Secret scan / scan (pull_request) Successful in 15s
Details
d328c3b578
 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ge5tovcwdEUBtPwUktsZZZ
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
jfraeys/infra!280
No description provided.