fix(docker): restart on the daemon.json change before any container exists #73

Merged

jfraeys merged 1 commit from jfraeys/fix/docker-restart-before-containers into main

2026-10-12 18:41:48 -04:00

jfraeys commented

2026-10-12 17:51:01 -04:00

Owner

Staging run 495 diagnostics: the first dockerd start used containerd-snapshotter=true / overlayfs (Docker 29 default); the Restart Docker handler, deferred to the first flush_handlers (inside the crowdsec role), brought it back with containerd-snapshotter=false / overlay2 (daemon.json pins overlay2). A separate store: docker ps -a showed 0 containers, so crowdsec's verify failed with No such container for the whole retry window even though the container existed a minute earlier.

Fix: flush_handlers right after the daemon.json task in roles/docker, which runs before any container-creating role in services.yml. Steady state is unchanged (no config change, no handler, no restart). A DR rebuild of production would have hit this too. Incident recorded in docs/incidents.md. ansible-lint, yamllint, syntax-check pass. 🤖 Generated with Claude Code

Staging run 495 diagnostics: the first dockerd start used containerd-snapshotter=true / overlayfs (Docker 29 default); the `Restart Docker` handler, deferred to the first flush_handlers (inside the crowdsec role), brought it back with containerd-snapshotter=false / overlay2 (daemon.json pins overlay2). A separate store: docker ps -a showed 0 containers, so crowdsec's verify failed with `No such container` for the whole retry window even though the container existed a minute earlier. Fix: `flush_handlers` right after the daemon.json task in roles/docker, which runs before any container-creating role in services.yml. Steady state is unchanged (no config change, no handler, no restart). A DR rebuild of production would have hit this too. Incident recorded in docs/incidents.md. ansible-lint, yamllint, syntax-check pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

jfraeys added 1 commit

2026-10-12 17:51:02 -04:00

fix(docker): restart on the daemon.json change before any container exists

Lint / lint (pull_request) Successful in 1m33s

Details

Secret scan / scan (pull_request) Successful in 13s

Details

bcfd240b64

Docker's containerd-snapshotter default vs the pinned overlay2 storage-driver: a deferred restart handler switched
backends after containers existed and hid them all (staging run 495). Flush handlers right after the daemon.json
task so the restart happens first. Incident recorded.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

jfraeys merged commit ac0296e612 into main

2026-10-12 18:41:48 -04:00

jfraeys referenced this pull request from a commit

2026-10-12 18:41:49 -04:00

Merge pull request 'fix(docker): restart on the daemon.json change before any container exists' (#73) from jfraeys/fix/docker-restart-before-containers into main

jfraeys referenced this pull request

2026-10-12 19:03:36 -04:00

fix(ci): staging parity: plain prod image, explicit role skip, documented differences #74