jfraeys/infra
Watch
1
0
Fork
You've already forked infra
 0
Code Issues 21 Pull requests 4 Projects Releases Packages Wiki Activity Actions

fix(manage): health shows Prometheus as stopped while it runs; monitor Prometheus and Alertmanager #249

Merged
jfraeys merged 2 commits from jfraeys/fix/health-compose-names into main 2026-10-16 19:10:08 -04:00
Conversation 0 Commits 2 Files changed 3 +51 -2
jfraeys commented 2026-10-16 18:50:22 -04:00
Owner
Copy link

Reported: 'prometheus in health check says stop but I can access it'.

Cause: Prometheus and Alertmanager are not in the health check's monitored table, so they fall into the generic 'known services' loop, which compares container names EXACTLY. Compose names containers -- (prometheus-prometheus-1), so 'prometheus' was never 'running'; /opt/prometheus exists, so the loop concluded 'deployed but stopped' and printed 'STOP (stopped, data preserved)'. Grafana and Loki would have shown the same once enabled.

Fix

  • _service_from_container maps a Compose container name to its service; used by both discovery functions. Explicitly named containers (vaultwarden, headscale) pass through.
  • Prometheus and Alertmanager join the monitored table with real checks: wget of /-/healthy inside the container (neither has a Docker HEALTHCHECK). Verified on the live host: Prometheus answers 'Prometheus Server is Healthy.' and Alertmanager 'OK'.

Checked: scripts/tests/test-health-service-state.sh (8 cases with a stubbed docker; three fail if the normalization is removed; in lint.yml); shellcheck -S error via the official image: clean.
Not checked: the full 'manage health check' output on services (needs this installed there). After merge and promotion: ./infra deploy --tags manage_cli --limit services, then ./manage health check --host services and confirm prometheus and alertmanager show a check mark.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Ge5tovcwdEUBtPwUktsZZZ

Reported: 'prometheus in health check says stop but I can access it'. Cause: Prometheus and Alertmanager are not in the health check's monitored table, so they fall into the generic 'known services' loop, which compares container names EXACTLY. Compose names containers <project>-<service>-<n> (prometheus-prometheus-1), so 'prometheus' was never 'running'; /opt/prometheus exists, so the loop concluded 'deployed but stopped' and printed 'STOP (stopped, data preserved)'. Grafana and Loki would have shown the same once enabled. Fix - _service_from_container maps a Compose container name to its service; used by both discovery functions. Explicitly named containers (vaultwarden, headscale) pass through. - Prometheus and Alertmanager join the monitored table with real checks: wget of /-/healthy inside the container (neither has a Docker HEALTHCHECK). Verified on the live host: Prometheus answers 'Prometheus Server is Healthy.' and Alertmanager 'OK'. Checked: scripts/tests/test-health-service-state.sh (8 cases with a stubbed docker; three fail if the normalization is removed; in lint.yml); shellcheck -S error via the official image: clean. Not checked: the full 'manage health check' output on services (needs this installed there). After merge and promotion: ./infra deploy --tags manage_cli --limit services, then ./manage health check --host services and confirm prometheus and alertmanager show a check mark. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Ge5tovcwdEUBtPwUktsZZZ
Prometheus showed 'STOP (stopped, data preserved)' while running: the generic service loop compared container names exactly, but Compose names
them <project>-<service>-<n> (prometheus-prometheus-1), and /opt/prometheus made it 'deployed but not running'. The discovery functions now map
a Compose name to its service. Prometheus and Alertmanager (no Docker HEALTHCHECK) are added to the monitored table with /-/healthy probes run
inside their containers (both verified on the live host). Grafana and Loki, when enabled, would have hit the same bug.
Test: test-health-service-state.sh (8 cases; fails without the normalization).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ge5tovcwdEUBtPwUktsZZZ
ci: run the health service-state test
All checks were successful
Lint / lint (pull_request) Successful in 2m30s
Details
Secret scan / scan (pull_request) Successful in 15s
Details
7769acddee
 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ge5tovcwdEUBtPwUktsZZZ
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
jfraeys/infra!249
No description provided.