fix(fail2ban): ignore the tailnet in the nginx jails (demo/app 502 after #244)

jfraeys commented

2026-10-16 21:38:10 -04:00

Owner

REGRESSION FROM #244. Since Traefik reaches web over the tailnet, nginx sees 100.64.0.3 as the client of every proxied request. Scanner traffic to demo/app (.php and wp- probes) made fail2ban's nginx-botsearch jail ban 100.64.0.3, so Traefik got connection refused on ports 80/443 (22 stayed open: the ban is port-specific) and demo.jfraeys.com returned 502. Found by checking the route after the ACL rollout (it was 404 'no such demo app' before). The jail's ignoreip listed only the services node's public address, as the template comment warns (2026-09-19 incident).

Fix: ignoreip also lists 100.64.0.0/10 and fd7a:115c:a1e0::/48. test_fail2ban_ignoreip.py renders the template; removing the prefixes fails it. docs/incidents.md records it. I should have checked address-keyed allowlists when changing the upstream address.

IMMEDIATE (not part of this PR; needs a run on web): sudo fail2ban-client set nginx-botsearch unbanip 100.64.0.3. Then deploy this with ./infra deploy --tags fail2ban --limit web after staging and promotion, so it cannot recur. Without the deploy the next burst re-bans it.
Better long term: have nginx log the real client address from X-Forwarded-For (set_real_ip_from + real_ip_header) so the jails ban visitors instead of the proxy; that is in the app's nginx config, not this repo.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Ge5tovcwdEUBtPwUktsZZZ

REGRESSION FROM #244. Since Traefik reaches web over the tailnet, nginx sees 100.64.0.3 as the client of every proxied request. Scanner traffic to demo/app (.php and wp- probes) made fail2ban's nginx-botsearch jail ban 100.64.0.3, so Traefik got connection refused on ports 80/443 (22 stayed open: the ban is port-specific) and demo.jfraeys.com returned 502. Found by checking the route after the ACL rollout (it was 404 'no such demo app' before). The jail's ignoreip listed only the services node's public address, as the template comment warns (2026-09-19 incident). Fix: ignoreip also lists 100.64.0.0/10 and fd7a:115c:a1e0::/48. test_fail2ban_ignoreip.py renders the template; removing the prefixes fails it. docs/incidents.md records it. I should have checked address-keyed allowlists when changing the upstream address. IMMEDIATE (not part of this PR; needs a run on web): sudo fail2ban-client set nginx-botsearch unbanip 100.64.0.3. Then deploy this with ./infra deploy --tags fail2ban --limit web after staging and promotion, so it cannot recur. Without the deploy the next burst re-bans it. Better long term: have nginx log the real client address from X-Forwarded-For (set_real_ip_from + real_ip_header) so the jails ban visitors instead of the proxy; that is in the app's nginx config, not this repo. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Ge5tovcwdEUBtPwUktsZZZ

jfraeys added 1 commit

2026-10-16 21:38:10 -04:00

fix(fail2ban): ignore the tailnet prefixes in the nginx jails (Traefik now reaches web over the tailnet)

Lint / lint (pull_request) Successful in 2m57s

Details

Secret scan / scan (pull_request) Successful in 17s

Details

4bbd2afbdf

#244 moved Traefik's webapps upstream to web's tailnet address, so nginx saw 100.64.0.3 instead of the services public address for every proxied request. The
nginx-botsearch jail banned it after a burst of scanner 404s and demo/app returned 502. ignoreip only listed the services public address (2026-09-19 incident).
ignoreip now also lists 100.64.0.0/10 and fd7a:115c:a1e0::/48. Test: test_fail2ban_ignoreip.py renders the template (fails without the prefixes). Incident entry added.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ge5tovcwdEUBtPwUktsZZZ

jfraeys merged commit 70e930886f into main

2026-10-16 21:42:28 -04:00

jfraeys referenced this pull request from a commit

2026-10-16 21:42:31 -04:00

Merge pull request 'fix(fail2ban): ignore the tailnet in the nginx jails (demo/app 502 after #244)' (#255) from jfraeys/fix/fail2ban-ignore-tailnet into main

Rows
Columns

fix(fail2ban): ignore the tailnet in the nginx jails (demo/app 502 after #244) #255