firewall: 80/443 DROP unreachable on the old node (RETURN first); git DNS-only conflicts with Cloudflare-only rule (cutover blocker)

jfraeys commented

2026-10-16 15:48:25 -04:00

Owner

Found 2026-10-16 while checking the old production node after the firewall deploy (#230). Read-only checks only.

1. The 80/443 Cloudflare-only DROP is dead on the old production node

iptables -L DOCKER-USER --line-numbers: RETURN is rule 82 (268K packets), the two multiport 80,443 DROP rules are 83 and 84 with 0 packets.
Rules after RETURN are never reached. The same on IPv6 (RETURN 50 of 52).
Direct requests to the origin succeed from anywhere: curl --resolve git.jfraeys.com:443:172.105.107.95 returns 200 over IPv4 and over IPv6.
auth, vault and jfraeys.com are proxied by Cloudflare, so this bypasses Cloudflare (WAF, bot rules, rate limits) for them.
Authelia and Vaultwarden still authenticate; this removes a layer, it is not an auth bypass. The origin IP is public anyway (git's A record).
Pre-existing, at least since the node last booted (8 weeks, 0 packets on the DROPs). The 22/25/17170 rules sit before RETURN and are effective
(3041 packets dropped on port 22).
Cause: docker-user-firewall.sh.j2 appends the DROPs with -A and adds RETURN only || -A if absent. If Docker's own RETURN exists first, the DROPs land behind it.
Whether the order is right depends on whether Docker or the script created the chain first. The new OVH node got it right (DROP 83-84, RETURN 85;
a direct request to 51.79.103.235:443 times out); the old node got it wrong. My earlier old/new comparison sorted the rules and could not show order.

2. The intended rule contradicts DNS for git

Cloudflare DNS (read-only API check): git.jfraeys.com A/AAAA are proxied=false (DNS only); auth, vault, jfraeys.com are proxied=true.
With the DROP effective, git.jfraeys.com would be unreachable for everyone except Cloudflare addresses.

Consequence for the OVH cutover

The new node enforces the DROP. If git's DNS-only record points at it, HTTPS git (clone, push, API, web UI) stops working for the public at cutover.

Do not

Do not deploy a fix that makes RETURN last to the old node before this is decided: it would activate the DROP and take git.jfraeys.com down.

To decide

Why is git DNS-only (large pushes, runner traffic, Cloudflare limits)? Proxy it, or exempt it.
Then: make the script deterministic (remove existing RETURNs and append one at the end), extend test-docker-user-firewall.sh to start from a chain that already has Docker's RETURN, and add an exemption variable if git stays DNS-only.
Add the DNS/firewall alignment to the cutover checklist in docs/runbooks/provider-migration-ovh.md, and check the order of the chain on every node after a firewall run.

Found 2026-10-16 while checking the old production node after the firewall deploy (#230). Read-only checks only. ## 1. The 80/443 Cloudflare-only DROP is dead on the old production node - `iptables -L DOCKER-USER --line-numbers`: `RETURN` is rule 82 (268K packets), the two `multiport 80,443 DROP` rules are 83 and 84 with 0 packets. Rules after RETURN are never reached. The same on IPv6 (RETURN 50 of 52). - Direct requests to the origin succeed from anywhere: `curl --resolve git.jfraeys.com:443:172.105.107.95` returns 200 over IPv4 and over IPv6. `auth`, `vault` and `jfraeys.com` are proxied by Cloudflare, so this bypasses Cloudflare (WAF, bot rules, rate limits) for them. Authelia and Vaultwarden still authenticate; this removes a layer, it is not an auth bypass. The origin IP is public anyway (git's A record). - Pre-existing, at least since the node last booted (8 weeks, 0 packets on the DROPs). The 22/25/17170 rules sit before RETURN and are effective (3041 packets dropped on port 22). - Cause: `docker-user-firewall.sh.j2` appends the DROPs with `-A` and adds RETURN only `|| -A` if absent. If Docker's own RETURN exists first, the DROPs land behind it. Whether the order is right depends on whether Docker or the script created the chain first. The new OVH node got it right (DROP 83-84, RETURN 85; a direct request to 51.79.103.235:443 times out); the old node got it wrong. My earlier old/new comparison sorted the rules and could not show order. ## 2. The intended rule contradicts DNS for git Cloudflare DNS (read-only API check): `git.jfraeys.com` A/AAAA are **proxied=false** (DNS only); `auth`, `vault`, `jfraeys.com` are proxied=true. With the DROP effective, git.jfraeys.com would be unreachable for everyone except Cloudflare addresses. ## Consequence for the OVH cutover The new node enforces the DROP. If git's DNS-only record points at it, HTTPS git (clone, push, API, web UI) stops working for the public at cutover. ## Do not Do not deploy a fix that makes RETURN last to the old node before this is decided: it would activate the DROP and take git.jfraeys.com down. ## To decide 1. Why is git DNS-only (large pushes, runner traffic, Cloudflare limits)? Proxy it, or exempt it. 2. Then: make the script deterministic (remove existing RETURNs and append one at the end), extend test-docker-user-firewall.sh to start from a chain that already has Docker's RETURN, and add an exemption variable if git stays DNS-only. 3. Add the DNS/firewall alignment to the cutover checklist in docs/runbooks/provider-migration-ovh.md, and check the order of the chain on every node after a firewall run.

jfraeys referenced this issue

2026-10-16 15:49:27 -04:00

docs(incidents): Cloudflare-only 80/443 DROP dead on the old production node (#233) #234

jfraeys commented

2026-10-16 15:51:12 -04:00

Author

Owner

Options for exposing git with a working Cloudflare-only 80/443 rule (2026-10-16)

Facts these rest on: the DROP matches by port and source address, never by hostname, so it cannot treat git differently from auth/vault.
firewall_restrict_web_to_cloudflare is derived, not set per host: true whenever traefik_certresolver == cloudflare and the host is not staging.
git.jfraeys.com and vpn.jfraeys.com are DNS-only because Cloudflare's Bot Fight Mode (Free plan, zone-wide, no per-hostname exemption)
challenges non-browser clients (git, CI, API tokens); see docs/incidents.md. Nothing in terraform or roles/traefik today uses
Authenticated Origin Pulls or a Traefik IP allowlist.

#	Option	Effect	Cost / risk
A	Proxy git through Cloudflare	DROP can stay on	Bot Fight Mode would challenge git CLI, CI and API clients (403 challenge, cannot be exempted on Free). Breaks git. Rejected.
B	Second public IP for git, with its own rules	git exempt, rest restricted	Extra address on each provider, a second Traefik entrypoint, DNS and firewall per IP. Heavy.
C	Move git HTTPS to another port	git off 80/443	Every clone URL, remote and webhook changes. Disruptive.
D	Stop filtering by network; enforce "from Cloudflare" in Traefik per host	auth, vault, main restricted at L7; git open	Needs Authenticated Origin Pulls (mTLS) on those routers: enable it on the zone (terraform), load Cloudflare's origin-pull CA into Traefik, a tlsOption on the proxied routers. A plain Cloudflare IP allowlist is weaker: any Cloudflare customer can send requests to the origin from Cloudflare addresses. New code and a new failure mode to test.
E	Turn the rule off honestly (`firewall_restrict_web_to_cloudflare: false`)	what production does today in effect	No new protection, but no false assurance: the dead DROP and the Cloudflare ACCEPT list are removed by the script's existing else-branch. Origin reachable around Cloudflare for the proxied hosts (auth/vault still authenticate).

Recommendation

Before the cutover: E. Set the flag to false for production hosts explicitly (host/group var), so the new node behaves like the old one in effect and git keeps working.
It also removes the Cloudflare ACCEPT list from the chain (smaller, and the ordering bug no longer matters for 80/443). No outage risk; verify with a direct request to the new node after deploy.
Then D as its own project, if you want the origin closed to non-Cloudflare traffic for the proxied hosts: Authenticated Origin Pulls plus per-router TLS options. Test on staging (needs a proxied staging name) before production.
In any case fix the ordering in the script (RETURN last, test from a chain that already holds Docker's RETURN) and add a post-run check that the 80/443 rules, if present, sit before RETURN.

Not recommended: A, B, C.

## Options for exposing git with a working Cloudflare-only 80/443 rule (2026-10-16) Facts these rest on: the DROP matches by port and source address, never by hostname, so it cannot treat `git` differently from `auth`/`vault`. `firewall_restrict_web_to_cloudflare` is derived, not set per host: true whenever `traefik_certresolver == cloudflare` and the host is not staging. `git.jfraeys.com` and `vpn.jfraeys.com` are DNS-only because Cloudflare's Bot Fight Mode (Free plan, zone-wide, no per-hostname exemption) challenges non-browser clients (git, CI, API tokens); see docs/incidents.md. Nothing in terraform or roles/traefik today uses Authenticated Origin Pulls or a Traefik IP allowlist. | # | Option | Effect | Cost / risk | |---|---|---|---| | A | Proxy git through Cloudflare | DROP can stay on | Bot Fight Mode would challenge git CLI, CI and API clients (403 challenge, cannot be exempted on Free). Breaks git. Rejected. | | B | Second public IP for git, with its own rules | git exempt, rest restricted | Extra address on each provider, a second Traefik entrypoint, DNS and firewall per IP. Heavy. | | C | Move git HTTPS to another port | git off 80/443 | Every clone URL, remote and webhook changes. Disruptive. | | D | Stop filtering by network; enforce "from Cloudflare" in Traefik per host | auth, vault, main restricted at L7; git open | Needs Authenticated Origin Pulls (mTLS) on those routers: enable it on the zone (terraform), load Cloudflare's origin-pull CA into Traefik, a tlsOption on the proxied routers. A plain Cloudflare IP allowlist is weaker: any Cloudflare customer can send requests to the origin from Cloudflare addresses. New code and a new failure mode to test. | | E | Turn the rule off honestly (`firewall_restrict_web_to_cloudflare: false`) | what production does today in effect | No new protection, but no false assurance: the dead DROP and the Cloudflare ACCEPT list are removed by the script's existing else-branch. Origin reachable around Cloudflare for the proxied hosts (auth/vault still authenticate). | ## Recommendation 1. **Before the cutover: E.** Set the flag to false for production hosts explicitly (host/group var), so the new node behaves like the old one in effect and git keeps working. It also removes the Cloudflare ACCEPT list from the chain (smaller, and the ordering bug no longer matters for 80/443). No outage risk; verify with a direct request to the new node after deploy. 2. **Then D as its own project**, if you want the origin closed to non-Cloudflare traffic for the proxied hosts: Authenticated Origin Pulls plus per-router TLS options. Test on staging (needs a proxied staging name) before production. 3. **In any case** fix the ordering in the script (RETURN last, test from a chain that already holds Docker's RETURN) and add a post-run check that the 80/443 rules, if present, sit before RETURN. Not recommended: A, B, C.

jfraeys referenced this issue

2026-10-16 15:55:43 -04:00

fix(firewall): RETURN last, Cloudflare-only 80/443 rule off (issue #233) #235

jfraeys commented

2026-10-16 15:55:43 -04:00

Author

Owner

Decision 2026-10-16: option E now (rule off for all hosts, RETURN-last fix and check in the script), PR opened. The planned move from Cloudflare to deSEC makes E the end state: deSEC does not proxy, so no name is proxied and the Cloudflare-only rule has no purpose. Option D (Authenticated Origin Pulls with a per-router Traefik tlsOption) is NOT built; it is recorded in FUTURE_IMPROVEMENTS.md as the way to restrict the origin per host if Cloudflare proxying returns. Cutover note: with the rule off, direct 443 on the new OVH node is open, as on the old node, so git keeps working when DNS moves.

jfraeys referenced this issue from a commit

2026-10-16 16:31:04 -04:00

Merge pull request 'docs(incidents): Cloudflare-only 80/443 DROP dead on the old production node (#233)' (#234) from jfraeys/docs/incident-docker-user-return-order into main

jfraeys referenced this issue from a commit

2026-10-16 16:31:30 -04:00

Merge pull request 'fix(firewall): RETURN last, Cloudflare-only 80/443 rule off (issue #233)' (#235) from jfraeys/fix/firewall-return-last-and-web-open into main

jfraeys referenced this issue

2026-10-16 16:58:59 -04:00

docs(incidents): 80/443 DROP entry is fixed and deployed (#233, #235) #238

jfraeys commented

2026-10-16 16:59:01 -04:00

Author

Owner

Deployed to services 2026-10-16 (PR #235). Verified: DOCKER-USER 85/53 -> 22/22 rules (v4/v6), RETURN last and unique, no 80/443 or 2222 rules, 22/25/17170 DROPs and tailnet ACCEPTs intact, ports 25/17170 closed publicly, 15 containers healthy, git/auth/vault/jfraeys.com 200. Remaining for this issue: run the firewall tag on the OVH rehearsal node (it still has the old rule set; direct 443 is blocked there), and keep the cutover note that git needs direct 443 open.

jfraeys referenced this issue from a commit

2026-10-16 17:01:00 -04:00

Merge pull request 'docs(incidents): 80/443 DROP entry is fixed and deployed (#233, #235)' (#238) from jfraeys/docs/incident-233-fixed into main

jfraeys commented

2026-10-16 19:06:50 -04:00

Author

Owner

Update 2026-10-16: PR #235 (RETURN last + Cloudflare-only rule off) is merged and deployed to services. Verified: DOCKER-USER 85/53 -> 22/22 rules, RETURN last, no 80/443 or 2222 rules, 22/25/17170 protections intact, sites 200. Remaining: run --tags firewall on the OVH rehearsal node (it still has the old rule set, direct 443 blocked), then close. Related: tailnet and admin-tool access hardening is tracked in #251.

jfraeys referenced this issue

2026-10-16 19:06:50 -04:00

Migrate production compute from Linode to OVH (starting with services) #187

jfraeys referenced this issue

2026-10-16 20:42:42 -04:00

Migrate production compute from Linode to OVH (starting with services) #187

jfraeys commented

2026-10-16 21:35:04 -04:00

Author

Owner

Done 2026-10-17: the firewall tag was run on the OVH rehearsal node and verified. DOCKER-USER on ovh-services: 22/22 rules (v4/v6), RETURN last and unique, no 80/443 and no 2222 rules, 22/25/17170 DROPs and tailnet ACCEPTs intact, docker-user-firewall.service active, 15 containers healthy. Direct 443 now answers 200 on the new node as on the old one (git.jfraeys.com stays reachable at cutover); ports 17170, 25, 2222, 3000, 9093 stay closed publicly. Both nodes now have identical chains. Fixed in #235 (RETURN last, rule off), documented in docs/incidents.md. Closing.

jfraeys closed this issue

2026-10-16 21:35:04 -04:00

jfraeys referenced this issue

2026-10-16 21:35:04 -04:00

access: tailnet ACL, Headscale login and admin tools behind Authelia (tracking PRs #246-#250) #251

Rows
Columns

firewall: 80/443 DROP unreachable on the old node (RETURN first); git DNS-only conflicts with Cloudflare-only rule (cutover blocker) #233

1. The 80/443 Cloudflare-only DROP is dead on the old production node

2. The intended rule contradicts DNS for git

Consequence for the OVH cutover

Do not

To decide

Options for exposing git with a working Cloudflare-only 80/443 rule (2026-10-16)

Recommendation