web: coach-site backup.service has never produced a backup (systemd % specifiers); no failed-unit alert #240

New issue

Open

opened 2026-10-16 17:12:28 -04:00 by jfraeys · 0 comments

jfraeys commented

2026-10-16 17:12:28 -04:00

Owner

Found 2026-10-16 while comparing the new OVH web node with production web (read-only).

What is wrong

backup.service ("Coach Site Backup") on production web has failed every day since the journal begins (2026-08-23), and
/var/backups/coach-site/ is empty: no backup has ever been written. The unit's own comment says it exists to protect
saved-but-unpublished drafts, which "only ever exist on disk here". They have had no backup for at least 54 days.

Cause

systemd expands % specifiers in ExecStart= before the shell sees the line. The command uses
backup-$(date +%Y%m%d-%H%M%S).tar.gz, so %m becomes the machine id, %d the credentials directory, %H the hostname and %S
the state directory. The target file name becomes /var/backups/coach-site/backup-/etc/systemd/systemf0cb4c61.../run/credentials/backup.service-web/var/lib.tar.gz
(visible in the journal), tar cannot open it and exits 2.

Fix (belongs to the coach-site repo, not this one)

The unit is not in this repo (no role here defines it; roles here stay generic). In the unit's ExecStart= escape every percent:
$(date +%%Y%%m%%d-%%H%%M%%S). Check the rest of that line (the find ... -mtime retention part) for other %. Redeploy, then run
systemctl start backup.service once and confirm a file appears in /var/backups/coach-site/.

Gap in this repo

Nothing alerted. There is no failed-unit alert (no node_systemd_unit_state rule in roles/prometheus/templates/alerts.yml.j2) and
the infra backup metrics only cover infra-backup, not this unit. A unit that fails daily for weeks should page. Options: enable the
node exporter's systemd collector for web and alert on node_systemd_unit_state{state="failed"} == 1 for more than an hour; or have
the app role write a textfile metric.

Note

The local copy lives on the same server (the unit says so), so it is not a substitute for off-server durability; published content is
already protected by git push.

Found 2026-10-16 while comparing the new OVH web node with production web (read-only). ## What is wrong `backup.service` ("Coach Site Backup") on production `web` has **failed every day since the journal begins (2026-08-23)**, and `/var/backups/coach-site/` is **empty**: no backup has ever been written. The unit's own comment says it exists to protect saved-but-unpublished drafts, which "only ever exist on disk here". They have had no backup for at least 54 days. ## Cause systemd expands `%` specifiers in `ExecStart=` before the shell sees the line. The command uses `backup-$(date +%Y%m%d-%H%M%S).tar.gz`, so `%m` becomes the machine id, `%d` the credentials directory, `%H` the hostname and `%S` the state directory. The target file name becomes `/var/backups/coach-site/backup-/etc/systemd/systemf0cb4c61.../run/credentials/backup.service-web/var/lib.tar.gz` (visible in the journal), tar cannot open it and exits 2. ## Fix (belongs to the coach-site repo, not this one) The unit is not in this repo (no role here defines it; roles here stay generic). In the unit's `ExecStart=` escape every percent: `$(date +%%Y%%m%%d-%%H%%M%%S)`. Check the rest of that line (the `find ... -mtime` retention part) for other `%`. Redeploy, then run `systemctl start backup.service` once and confirm a file appears in `/var/backups/coach-site/`. ## Gap in this repo Nothing alerted. There is no failed-unit alert (no `node_systemd_unit_state` rule in `roles/prometheus/templates/alerts.yml.j2`) and the infra backup metrics only cover `infra-backup`, not this unit. A unit that fails daily for weeks should page. Options: enable the node exporter's systemd collector for `web` and alert on `node_systemd_unit_state{state="failed"} == 1` for more than an hour; or have the app role write a textfile metric. ## Note The local copy lives on the same server (the unit says so), so it is not a substitute for off-server durability; published content is already protected by git push.

jfraeys referenced this issue

2026-10-16 20:42:42 -04:00

Roadmap: remaining infra and secrets work (tracker) #122

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

jfraeys/infra#240

No description provided.

Rows
Columns