jfraeys/infra
Watch
1
0
Fork
You've already forked infra
 0
Code Issues 21 Pull requests 4 Projects Releases Packages Wiki Activity Actions

feat(alertmanager): phone push through ntfy, and alert when notifications fail #128

Merged
jfraeys merged 1 commit from jfraeys/feat/alertmanager-ntfy into main 2026-10-13 16:37:10 -04:00
Conversation 0 Commits 1 Files changed 12 +202 -13
jfraeys commented 2026-10-13 16:16:44 -04:00
Owner
Copy link

Found 2026-10-13: Alertmanager's only receiver was email (Alertmanager -> Postfix -> Zedmail), so while the relay refused or rejected mail for over a day no alert reached anyone, including PostfixQueueGrowing itself.

  • critical and warning alerts also push to your phone through ntfy when ALERTMANAGER_NTFY_TOPIC is set in the vault (new optional key, in the manifest). Receiver primary-push = email + webhook; info stays email-only; unset topic = unchanged email-only config. ntfy's built-in alertmanager template formats the webhook, so no relay container runs on the 960 MB host.
  • The URL (it contains the topic, the only secret on public ntfy.sh) is written to /opt/alertmanager/secrets/ntfy_url (0400, owner 65534, the container user) and read through url_file, so it never appears in the world-readable alertmanager.yml. The topic is validated (16-64 chars of A-Za-z0-9_-). Changing it restarts Alertmanager.
  • Prometheus did not scrape Alertmanager at all: added the job and AlertNotificationsFailing (warning; an integration's notifications failing for 10m), so a dead email path announces itself by push.
  • Tests: scripts/tests/test-alertmanager-config.sh (routing for both configs, topic absent from the rendered config, amtool check-config in prom/alertmanager:v0 = 0.34.1, which accepts url_file); roles/prometheus/tests/alert_notifications_test.yml (3 promtool scenarios). Mutation-checked: a weakened rule and a critical route sent to email-only both fail. yamllint and ansible-lint clean; secret hygiene passes.
  • Not exercised end to end: no real push has been sent. After merge: install the ntfy app, pick a random topic, subscribe, store it (pbpaste | scripts/vault-set.py set ALERTMANAGER_NTFY_TOPIC), then ./infra deploy --tags alertmanager,prometheus --limit services and fire a test alert. Setup steps are in roles/alertmanager/CLAUDE.md.
  • Not covered: Alertmanager or the whole host down (needs an external heartbeat).

🤖 Generated with Claude Code

Found 2026-10-13: Alertmanager's only receiver was email (Alertmanager -> Postfix -> Zedmail), so while the relay refused or rejected mail for over a day no alert reached anyone, including PostfixQueueGrowing itself. - critical and warning alerts also push to your phone through ntfy when ALERTMANAGER_NTFY_TOPIC is set in the vault (new optional key, in the manifest). Receiver primary-push = email + webhook; info stays email-only; unset topic = unchanged email-only config. ntfy's built-in alertmanager template formats the webhook, so no relay container runs on the 960 MB host. - The URL (it contains the topic, the only secret on public ntfy.sh) is written to /opt/alertmanager/secrets/ntfy_url (0400, owner 65534, the container user) and read through url_file, so it never appears in the world-readable alertmanager.yml. The topic is validated (16-64 chars of A-Za-z0-9_-). Changing it restarts Alertmanager. - Prometheus did not scrape Alertmanager at all: added the job and AlertNotificationsFailing (warning; an integration's notifications failing for 10m), so a dead email path announces itself by push. - Tests: scripts/tests/test-alertmanager-config.sh (routing for both configs, topic absent from the rendered config, amtool check-config in prom/alertmanager:v0 = 0.34.1, which accepts url_file); roles/prometheus/tests/alert_notifications_test.yml (3 promtool scenarios). Mutation-checked: a weakened rule and a critical route sent to email-only both fail. yamllint and ansible-lint clean; secret hygiene passes. - Not exercised end to end: no real push has been sent. After merge: install the ntfy app, pick a random topic, subscribe, store it (pbpaste | scripts/vault-set.py set ALERTMANAGER_NTFY_TOPIC), then ./infra deploy --tags alertmanager,prometheus --limit services and fire a test alert. Setup steps are in roles/alertmanager/CLAUDE.md. - Not covered: Alertmanager or the whole host down (needs an external heartbeat). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
feat(alertmanager): phone push through ntfy for critical and warning alerts, and alert when notifications fail
All checks were successful
Lint / lint (pull_request) Successful in 1m31s
Details
Secret scan / scan (pull_request) Successful in 13s
Details
e89885526e
 
Email was the only alert channel; with Zedmail failing, nothing reached anyone for over a day. critical/warning now also post to
an ntfy topic (ALERTMANAGER_NTFY_TOPIC, optional) via ntfy's built-in alertmanager template, the URL read from a 0400 file so
the topic never lands in the world-readable config. Prometheus now scrapes Alertmanager and warns when an integration keeps failing.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
jfraeys/infra!128
No description provided.