Skip to content

Performance Monitoring Quick Start

Get started with performance monitoring in 5 minutes.

Prerequisites

  • Docker and Docker Compose
  • Go 1.21 or later
  • GitHub repository (for CI/CD integration)

1. Start Monitoring Stack

make monitoring-performance

This starts: - Grafana: http://localhost:3001 (admin/admin) - Pushgateway: http://localhost:9091 - Loki: http://localhost:3100

2. Run Benchmarks

# Run benchmarks locally
make benchmark

# Or run with detailed output
go test -bench=. -benchmem ./tests/benchmarks/...

3. CPU Profiling

HTTP Load Test Profiling

# CPU profile MediumLoad HTTP test (with rate limiting)
make profile-load

# CPU profile MediumLoad HTTP test (no rate limiting - recommended for profiling)
make profile-load-norate

This generates cpu_load.out which you can analyze with:

# View interactive profile
go tool pprof cpu_load.out

# Generate flame graph
go tool pprof -raw cpu_load.out | go-flamegraph.pl > cpu_flame.svg

# View top functions
go tool pprof -top cpu_load.out

WebSocket Queue Profiling

# CPU profile WebSocket → Redis queue → worker path
make profile-ws-queue

Generates cpu_ws.out for WebSocket performance analysis.

Profiling Tips

  • Use profile-load-norate for cleaner CPU profiles (no rate limiting delays)
  • Profiles run for 60 seconds by default
  • Requires Redis running on localhost:6379
  • Results show throughput, latency, and error rate metrics

4. View Results

Open Grafana dashboard: http://localhost:3001

Navigate to the Performance Dashboard to see: - Real-time benchmark results - Historical trends - Performance comparisons

5. Enable CI/CD Integration

Add GitHub secret: 

PROMETHEUS_PUSHGATEWAY_URL=http://your-pushgateway:9091

Now benchmarks run automatically on: - Every push to main/develop - Pull requests - Daily schedule

6. Verify Integration

  1. Push code to trigger workflow
  2. Check Pushgateway: http://localhost:9091/metrics
  3. View metrics in Grafana

7. Key Metrics

  • benchmark_time_per_op - Execution time
  • benchmark_memory_per_op - Memory usage
  • benchmark_allocs_per_op - Allocation count

8. Troubleshooting

No metrics in Grafana? 

# Check services
docker ps --filter "name=monitoring"

# Check Pushgateway
curl http://localhost:9091/metrics

Workflow failing? - Verify GitHub secret configuration - Check workflow logs in GitHub Actions

Profiling issues? 

# Flag error like "flag provided but not defined: -test.paniconexit0"
# This should be fixed now, but if it persists:
go test ./tests/load -run TestLoadProfile_Medium -count=1 -cpuprofile cpu_load.out -v -args -profile-norate

# Redis not available?
# Start Redis for profiling tests:
docker run -d -p 6379:6379 redis:alpine

# Check profile file generated
ls -la cpu_load.out

9. Next Steps

Ready in 5 minutes!