jfraeys/infra
Watch
1
0
Fork
You've already forked infra
 0
Code Issues 21 Pull requests 4 Projects Releases Packages Wiki Activity Actions

fix(hardening): cap the persistent systemd journal at 500M #127

Merged
jfraeys merged 1 commit from jfraeys/fix/journald-size-cap into main 2026-10-13 17:44:11 -04:00
Conversation 0 Commits 1 Files changed 6 +61 -1
jfraeys commented 2026-10-13 16:08:49 -04:00
Owner
Copy link

Live 2026-10-13: the services host's / was 82% used (HostDiskSpaceWarning); the journal was 2.4 GB of the 2.7 GB in /var/log and no role or cloud-init sets any journald limit.

  • roles/hardening: drop-in /etc/systemd/journald.conf.d/50-size-cap.conf with SystemMaxUse={{ hardening_journald_max_use }} (default 500M). The role applies to both web and services nodes. Changing the value notifies two handlers on one topic: restart systemd-journald, then journalctl --vacuum-size to trim what already exceeds the cap (skipped under Molecule).
  • Verification: Molecule hardening scenario passes locally in Docker (converge clean, second run changed=0, verify asserts the cap); playbooks/validate-hardening.yml checks the effective setting on staging web and services (systemd-analyze cat-config). ansible-lint and yamllint clean.
  • Effect on first deploy: the vacuum permanently deletes journal history beyond 500M on the services host (about 2 GB). Logs shipped to Loki are unaffected.
  • The staging check hardcodes 500M; if the default changes, update it too.
  • Deploy: ./infra deploy --tags hardening --limit services (and the web host with its own playbook).

🤖 Generated with Claude Code

Live 2026-10-13: the services host's / was 82% used (HostDiskSpaceWarning); the journal was 2.4 GB of the 2.7 GB in /var/log and no role or cloud-init sets any journald limit. - roles/hardening: drop-in /etc/systemd/journald.conf.d/50-size-cap.conf with SystemMaxUse={{ hardening_journald_max_use }} (default 500M). The role applies to both web and services nodes. Changing the value notifies two handlers on one topic: restart systemd-journald, then journalctl --vacuum-size to trim what already exceeds the cap (skipped under Molecule). - Verification: Molecule hardening scenario passes locally in Docker (converge clean, second run changed=0, verify asserts the cap); playbooks/validate-hardening.yml checks the effective setting on staging web and services (systemd-analyze cat-config). ansible-lint and yamllint clean. - Effect on first deploy: the vacuum permanently deletes journal history beyond 500M on the services host (about 2 GB). Logs shipped to Loki are unaffected. - The staging check hardcodes 500M; if the default changes, update it too. - Deploy: ./infra deploy --tags hardening --limit services (and the web host with its own playbook). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
fix(hardening): cap the persistent systemd journal at 500M
All checks were successful
Lint / lint (pull_request) Successful in 1m56s
Details
Secret scan / scan (pull_request) Successful in 13s
Details
e386ef03f8
 
The journal reached 2.4 GB of the 25 GB services disk (82% used, HostDiskSpaceWarning) because nothing capped it.
A journald drop-in sets SystemMaxUse (hardening_journald_max_use); changing it restarts journald and vacuums once.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
jfraeys deleted branch jfraeys/fix/journald-size-cap 2026-10-13 17:45:02 -04:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
jfraeys/infra!127
No description provided.