services disk filled to 100% (2026-10-18): follow-ups, incl. revisit the CI cache reset #285
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
On 2026-10-18 the
servicesroot disk (25 GB, Linode) hit 100%. The warning alert (HostDiskSpaceWarning, <20% free) had been in range since the disk passed about 80%; it was 84% earlier the same day. Whether that alert reached the operator is unconfirmed.What used the space (before cleanup, approx.)
overlay27.9 GB (about 3.4 GB were unused old images, removed withdocker image prune -a)/opt/forgejo-runner-cache/data4.5 GB (CI cache server, unbounded)/tmp/trivyand under/root/.cache)/var/lib/infra-backups1.4 GB, journal 450 MB (vacuumed 268 MB), unused Docker volumes about 620 MBDone / in flight
ci_cache_host_reset_*), with a path guard and tests.Follow-ups
forgejo-runner-cache, empty/opt/forgejo-runner-cache/data, start) until #284 is deployed.ci_cache_host_reset_enabled: false(the role removes the units). Re-check if a future forgejo-runner makes cache retention configurable, or if the cache moves offservices.SystemMaxUse), and limit how many dumps/var/lib/infra-backupskeeps (it also holds a stalemoodledirectory from Sep 11).backup.sh(/tmp/restic-env.$$, line about 573): atrapso an interrupted remote restore does not leave a credentials file behind (two old ones were found onservices).app_postgres_data,app_redis_data, many anonymous ones; about 620 MB): they may hold data, so not pruned../manage health check --host services,./manage backup status(was the 00:03 full snapshot taken?), Forgejo and Vaultwarden logs forno space left.servicesnode (g6-standard-2has 80 GB; deferred until after the migration) or move to OVHd2-4(50 GB). Decision for the operator.HostDiskSpaceWarning(<20% free for 10 min) reach the operator? Consider a higher bar (for example <30% on a small disk) while the disk is this size.Related: #187 (OVH migration), #220.