fix(traefik): Forgejo's Actions API gets its own router and a higher rate limit #282

Merged

jfraeys merged 1 commit from jfraeys/fix/forgejo-actions-rate-limit into main

2026-10-18 18:35:59 -04:00

jfraeys commented

2026-10-18 18:29:13 -04:00

Owner

After #280 the artifact upload reaches Forgejo (coaching_site run 1058: 11.5 MB uploaded). One item, precompress, then failed on 429 Too Many Requests (six times, Retry-After: 1, X-Retry-In: 896ms); the action retries only five times, one second apart, and gave up. The source is Traefik's general rate-limit middleware (100/min, burst 50) on the Forgejo router: an artifact upload sends many parallel chunk requests from one source address.

New middleware rate-limit-actions (traefik_actions_rate_average: 1800, traefik_actions_rate_burst: 300, per minute), defined next to rate-limit in base.yml.j2.
New router forgejo-actions in the Forgejo compose labels: Host(...) && PathPrefix('/api/actions_pipeline/'), priority 1000, same service, entrypoint, TLS and security headers, using the new middleware. Everything else on the Forgejo host keeps the general limit.
Why this scoping: the path needs a valid per-job token, so a low limit there protects nothing; scoping by path (not by source range) does not depend on what source address Traefik sees for tailnet traffic. If it still 429s after this, the next thing to look at is Traefik's source-address strategy (all runners sharing one counter).
Test: scripts/tests/test_forgejo_actions_route.py (8 cases, in lint) renders both templates and checks the router rule, the middleware reference and definition, the higher limit, the explicit priority, the unchanged general limit, and that the rendered compose is valid YAML. Verified it fails when the middleware is removed or the limit is lowered to the general one.

Rollout

merge, staging-validate green, ./infra approve, git pull.
scripts/release/deploy-services.sh --tags traefik,forgejo --limit services. The Forgejo labels change, so Compose recreates the Forgejo container (a short outage for git and CI): do it when no job is running. Traefik picks up the new middleware from its file provider.
Dispatch the coaching_site ci.yml: all of Build artifacts should upload. Confirm in the Traefik log that the requests carry no 429: docker logs traefik-traefik-1 --since 20m 2>&1 | grep actions_pipeline | grep -c '"DownstreamStatus":429' should print 0.

Rollback: revert and redeploy traefik,forgejo.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Ge5tovcwdEUBtPwUktsZZZ

After #280 the artifact upload reaches Forgejo (coaching_site run 1058: 11.5 MB uploaded). One item, `precompress`, then failed on `429 Too Many Requests` (six times, `Retry-After: 1`, `X-Retry-In: 896ms`); the action retries only five times, one second apart, and gave up. The source is Traefik's general `rate-limit` middleware (100/min, burst 50) on the Forgejo router: an artifact upload sends many parallel chunk requests from one source address. - New middleware `rate-limit-actions` (`traefik_actions_rate_average: 1800`, `traefik_actions_rate_burst: 300`, per minute), defined next to `rate-limit` in `base.yml.j2`. - New router `forgejo-actions` in the Forgejo compose labels: `Host(...) && PathPrefix('/api/actions_pipeline/')`, priority 1000, same service, entrypoint, TLS and security headers, using the new middleware. Everything else on the Forgejo host keeps the general limit. - Why this scoping: the path needs a valid per-job token, so a low limit there protects nothing; scoping by path (not by source range) does not depend on what source address Traefik sees for tailnet traffic. If it still 429s after this, the next thing to look at is Traefik's source-address strategy (all runners sharing one counter). - Test: `scripts/tests/test_forgejo_actions_route.py` (8 cases, in lint) renders both templates and checks the router rule, the middleware reference and definition, the higher limit, the explicit priority, the unchanged general limit, and that the rendered compose is valid YAML. Verified it fails when the middleware is removed or the limit is lowered to the general one. ## Rollout 1. merge, staging-validate green, `./infra approve`, `git pull`. 2. `scripts/release/deploy-services.sh --tags traefik,forgejo --limit services`. The Forgejo labels change, so Compose **recreates the Forgejo container** (a short outage for git and CI): do it when no job is running. Traefik picks up the new middleware from its file provider. 3. Dispatch the coaching_site `ci.yml`: all of `Build artifacts` should upload. Confirm in the Traefik log that the requests carry no 429: `docker logs traefik-traefik-1 --since 20m 2>&1 | grep actions_pipeline | grep -c '"DownstreamStatus":429'` should print 0. Rollback: revert and redeploy `traefik,forgejo`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Ge5tovcwdEUBtPwUktsZZZ

jfraeys added 1 commit

2026-10-18 18:29:14 -04:00

fix(traefik): Forgejo's Actions API gets its own router and a higher rate limit

Lint / lint (pull_request) Successful in 2m39s

Details

Secret scan / scan (pull_request) Successful in 13s

Details

bc70185641

Artifact uploads from a CI job send many parallel chunk requests from one address; the general
100/min limit answered with 429 and the action gave up after 5 retries (run 1058: 11.5 MB uploaded,
precompress failed). /api/actions_pipeline/ needs a per-job token, so the limit there protects nothing.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ge5tovcwdEUBtPwUktsZZZ

jfraeys merged commit 7a49212c29 into main

2026-10-18 18:35:59 -04:00

jfraeys referenced this pull request from a commit

2026-10-18 18:36:00 -04:00

Merge pull request 'fix(traefik): Forgejo's Actions API gets its own router and a higher rate limit' (#282) from jfraeys/fix/forgejo-actions-rate-limit into main

jfraeys referenced this pull request

2026-10-18 21:35:30 -04:00

Migrate production compute from Linode to OVH (starting with services) #187