jfraeys/infra
Watch
1
0
Fork
You've already forked infra
 0
Code Issues 21 Pull requests 4 Projects Releases Packages Wiki Activity Actions

feat(ci_cache_host): weekly reset of the CI cache data #284

Open
jfraeys wants to merge 1 commit from jfraeys/feat/ci-cache-weekly-reset into main
pull from: jfraeys/feat/ci-cache-weekly-reset
merge into: jfraeys:main
Conversation 0 Commits 1 Files changed 9 +162 -1
jfraeys commented 2026-10-18 21:31:58 -04:00
Owner
Copy link

The services disk hit 100% on 2026-10-18. The largest single item was the CI cache server's data, `/opt/forgejo-runner-cache/data`: 4.5 GB of a 25 GB disk. The runner's own pruning (unused entries after 7 days, anything after 30) is not configurable, and ephemeral CI VMs start cold and save a large Go build cache on most runs.

  • `forgejo-runner-cache-reset.service` (oneshot): stops `forgejo-runner-cache`, runs `find -mindepth 1 -delete` (the directory itself stays), and `ExecStopPost` starts the server again even if the delete fails, so the cache is never left stopped.
  • `forgejo-runner-cache-reset.timer`: `Sun --* 04:00:00`, `RandomizedDelaySec=15min`, `Persistent=true` (a missed run happens at the next boot). `ci_cache_host_reset_enabled` (default true) and `ci_cache_host_reset_oncalendar` are role variables; turning it off removes the units.
  • `tasks/reset_guard.yml`: refuses a data dir that is not an absolute path at least three levels deep, or has `..` or whitespace, because the reset deletes as root.
  • Cost: the first CI build after each reset is slower (cold Go cache). Entries are disposable by design.
  • Tests (`scripts/tests/test_ci_cache_reset.py`, 6 cases, in lint): the rendered service and timer, and the guard run through real Ansible against accepted and refused paths (`/`, `/opt`, `/opt/x`, `/opt/a/../b`, relative, spaces, trailing slash). Verified the guard test fails when the regex is weakened.

Rollout

  1. merge, `staging-validate` green, `./infra approve`, `git pull`.
  2. `scripts/release/deploy-services.sh --tags ci_cache_host --limit services`.
  3. Check: `systemctl list-timers forgejo-runner-cache-reset.timer` shows the next Sunday run, and `systemctl cat forgejo-runner-cache-reset.service` shows the `find` on `/opt/forgejo-runner-cache/data`. Nothing is deleted by the deploy itself.

Not in this PR (separate): a single Trivy cache location (two ~1.6 GB copies), a journal size cap, and a `trap` so an interrupted remote restore does not leave `/tmp/restic-env.*`. The disk itself (25 GB) is still too small for this host.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Ge5tovcwdEUBtPwUktsZZZ

The services disk hit 100% on 2026-10-18. The largest single item was the CI cache server's data, \`/opt/forgejo-runner-cache/data\`: **4.5 GB** of a 25 GB disk. The runner's own pruning (unused entries after 7 days, anything after 30) is not configurable, and ephemeral CI VMs start cold and save a large Go build cache on most runs. - \`forgejo-runner-cache-reset.service\` (oneshot): stops \`forgejo-runner-cache\`, runs \`find <data dir> -mindepth 1 -delete\` (the directory itself stays), and \`ExecStopPost\` starts the server again **even if the delete fails**, so the cache is never left stopped. - \`forgejo-runner-cache-reset.timer\`: \`Sun *-*-* 04:00:00\`, \`RandomizedDelaySec=15min\`, \`Persistent=true\` (a missed run happens at the next boot). \`ci_cache_host_reset_enabled\` (default true) and \`ci_cache_host_reset_oncalendar\` are role variables; turning it off removes the units. - \`tasks/reset_guard.yml\`: refuses a data dir that is not an absolute path at least three levels deep, or has \`..\` or whitespace, because the reset deletes as root. - Cost: the first CI build after each reset is slower (cold Go cache). Entries are disposable by design. - Tests (\`scripts/tests/test_ci_cache_reset.py\`, 6 cases, in lint): the rendered service and timer, and the guard run through real Ansible against accepted and refused paths (\`/\`, \`/opt\`, \`/opt/x\`, \`/opt/a/../b\`, relative, spaces, trailing slash). Verified the guard test fails when the regex is weakened. ## Rollout 1. merge, \`staging-validate\` green, \`./infra approve\`, \`git pull\`. 2. \`scripts/release/deploy-services.sh --tags ci_cache_host --limit services\`. 3. Check: \`systemctl list-timers forgejo-runner-cache-reset.timer\` shows the next Sunday run, and \`systemctl cat forgejo-runner-cache-reset.service\` shows the \`find\` on \`/opt/forgejo-runner-cache/data\`. Nothing is deleted by the deploy itself. Not in this PR (separate): a single Trivy cache location (two ~1.6 GB copies), a journal size cap, and a \`trap\` so an interrupted remote restore does not leave \`/tmp/restic-env.*\`. The disk itself (25 GB) is still too small for this host. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Ge5tovcwdEUBtPwUktsZZZ
feat(ci_cache_host): weekly reset of the CI cache data
All checks were successful
Lint / lint (pull_request) Successful in 3m0s
Details
Secret scan / scan (pull_request) Successful in 17s
Details
13a6acadae
 
The runner's own pruning (unused 7 days, any 30 days) is not configurable and the cache data reached
4.5 GB on the 25 GB services host (the disk hit 100% on 2026-10-18). A weekly timer stops the server,
empties the data dir, and always starts the server again; a guard refuses an unsafe data dir.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ge5tovcwdEUBtPwUktsZZZ
All checks were successful
Lint / lint (pull_request) Successful in 3m0s
Details
Secret scan / scan (pull_request) Successful in 17s
Details
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin jfraeys/feat/ci-cache-weekly-reset:jfraeys/feat/ci-cache-weekly-reset
git switch jfraeys/feat/ci-cache-weekly-reset

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff jfraeys/feat/ci-cache-weekly-reset
git switch jfraeys/feat/ci-cache-weekly-reset
git rebase main
git switch main
git merge --ff-only jfraeys/feat/ci-cache-weekly-reset
git switch jfraeys/feat/ci-cache-weekly-reset
git rebase main
git switch main
git merge --no-ff jfraeys/feat/ci-cache-weekly-reset
git switch main
git merge --squash jfraeys/feat/ci-cache-weekly-reset
git switch main
git merge --ff-only jfraeys/feat/ci-cache-weekly-reset
git switch main
git merge jfraeys/feat/ci-cache-weekly-reset
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
jfraeys/infra!284
No description provided.