# Testing Guide

Comprehensive testing documentation for FetchML platform with integrated monitoring.

## Quick Start Testing

### 5-Minute Fast Test

```bash
# Clean environment
make self-cleanup

# Start development stack with monitoring
make dev-up

# Quick authentication test
make test-auth

# Clean up
make dev-down
```

**Expected Results**:
- Admin user: Full access, shows all jobs
- Researcher user: Own jobs only
- Analyst user: Read-only access

## Test Environment Setup

### Development Environment with Monitoring

```bash
# Start development stack with monitoring
make dev-up

# Verify all services are running
make dev-status

# Run tests against running services
make test

# Check monitoring during tests
# Grafana: http://localhost:3000 (admin/admin123)
```

### Test Environment Verification

```bash
# Verify API server
curl -f http://localhost:8080/health

# Verify monitoring services
curl -f http://localhost:3000/api/health
curl -f http://localhost:9090/api/v1/query?query=up
curl -f http://localhost:3100/ready

# Verify Redis
docker exec ml-experiments-redis redis-cli ping
```

## Test Types

### Unit Tests
```bash
make test-unit          # Go unit tests only
cd cli && zig build test  # Zig CLI tests

# Unit tests live under tests/unit/ (including tests that cover internal/ packages)
go test ./tests/unit/...
```

### Integration Tests
```bash
make test-integration   # API and database integration
```

### End-to-End Tests
```bash
make test-e2e          # Full workflow testing

# Podman E2E is opt-in because it builds/runs containers
FETCH_ML_E2E_PODMAN=1 go test ./tests/e2e/...  # Enables TestPodmanIntegration
```

### All Tests
```bash
make test              # Run complete test suite
make test-coverage     # With coverage report
```

## Deployment-Specific Testing

### Development Environment Testing

```bash
# Start dev stack
make dev-up

# Run tests with monitoring
make test

# View test results in Grafana
# Load Test Performance dashboard
# System Health dashboard
```

### Production Environment Testing

```bash
cd deployments
make prod-up

# Test production deployment
make test-auth         # Multi-user auth test
make self-cleanup      # Clean up after testing

# Verify production monitoring
curl -f https://your-domain.com/health
```

### Homelab Secure Testing

```bash
cd deployments
make homelab-up

# Test secure deployment
make test-auth
make test-ssl          # SSL/TLS testing
```

## Authentication Testing Protocol

### Multi-User Authentication

```bash
# Test admin user
cp ~/.ml/config-admin.toml ~/.ml/config.toml
./cli/zig-out/bin/ml status
# Expected: Shows admin status and all jobs

# Test researcher user  
cp ~/.ml/config-researcher.toml ~/.ml/config.toml
./cli/zig-out/bin/ml status
# Expected: Shows researcher status and own jobs only

# Test analyst user
cp ~/.ml/config-analyst.toml ~/.ml/config.toml
./cli/zig-out/bin/ml status
# Expected: Shows analyst status, read-only access
```

### API Key Validation

```bash
# Test invalid API key
echo "invalid_key" > ~/.ml/config.toml
./cli/zig-out/bin/ml status
# Expected: Authentication failed error

# Test missing API key
rm ~/.ml/config.toml
./cli/zig-out/bin/ml status
# Expected: API key not configured error
```

### Job Queueing by Role

```bash
# Admin can queue jobs
cp ~/.ml/config-admin.toml ~/.ml/config.toml
echo "admin job" | ./cli/zig-out/bin/ml queue admin-test

# Researcher can queue jobs
cp ~/.ml/config-researcher.toml ~/.ml/config.toml
echo "research job" | ./cli/zig-out/bin/ml queue research-test

# Analyst cannot queue jobs (should fail)
cp ~/.ml/config-analyst.toml ~/.ml/config.toml
echo "analysis job" | ./cli/zig-out/bin/ml queue analysis-test
# Expected: Permission denied error
```

## Performance Testing

### Load Testing with Monitoring

```bash
# Start monitoring
make dev-up

# Run load tests
make load-test

# Monitor performance in real-time
# Grafana: http://localhost:3000
# Check: Request rates, response times, error rates
```

### Benchmark Suite

```bash
# Run benchmarks with performance tracking
./scripts/track_performance.sh

# Or run directly
make benchmark-local

# View results in Grafana dashboards
```

### Performance Monitoring During Tests

**Key Metrics to Watch**:
- API response times (95th percentile)
- Error rates (should be < 1%)
- Memory usage trends
- CPU utilization
- Request throughput

## CLI Testing

### Build and Test CLI

```bash
cd cli && zig build --release=fast
./zig-out/bin/ml --help
zig build test
```

### CLI Integration Tests

```bash

```

## Docker Container Testing

### Container Startup

```bash
# Start environment
make dev-up

# Check container status
docker ps --filter "name=ml-"
# Expected: All containers running and healthy
```

### Port Accessibility

```bash
# Test API server port
curl -I http://localhost:8080/health
# Expected: 200 OK response

# Test metrics port
curl -I http://localhost:9100/metrics
# Expected: 200 OK response

# Test monitoring ports
curl -I http://localhost:3000/api/health
curl -I http://localhost:9090/api/v1/query?query=up
```

### Container Cleanup

```bash
# Test cleanup script
make self-cleanup
make dev-down
# Expected: Containers stopped and removed
```

## Monitoring During Testing

### Real-Time Monitoring

```bash
# Access Grafana during tests
http://localhost:3000 (admin/admin123)

# Key Dashboards:
# - Load Test Performance: Request metrics, response times
# - System Health: Service status, resource usage
# - Log Analysis: Error logs, service logs
```

### Test Metrics Collection

**Automatically Collected**:
- HTTP request metrics
- Response time histograms
- Error counters
- Resource utilization
- Log aggregation

**Manual Test Markers**:
```bash
# Mark test start in logs
echo "TEST_START: $(date)" | tee -a /logs/test.log

# Mark test completion
echo "TEST_END: $(date)" | tee -a /logs/test.log
```

## Test Execution Protocol

### Phase 1: Environment Setup
1. Clean up any existing containers
2. Start fresh Docker environment with monitoring
3. Verify all services are running

### Phase 2: Authentication Testing
1. Test all user roles (admin, researcher, analyst)
2. Test invalid authentication scenarios
3. Verify role-based permissions

### Phase 3: Functional Testing
1. Test CLI commands (queue, status)
2. Test API endpoints
3. Test WebSocket communication

### Phase 4: Integration Testing
1. Test complete workflows
2. Test error scenarios
3. Test cleanup procedures

### Phase 5: Performance Testing
1. Run benchmarks
2. Perform load testing
3. Validate performance metrics

## Troubleshooting Tests

### Common Issues

**Server Not Running**:
```bash
# Check service status
make dev-status

# Check container logs
docker logs ml-experiments-api
docker logs ml-experiments-grafana
```

**Authentication Failures**:
```bash
# Verify configs
ls ~/.ml/config-*.toml

# Check API health
curl -I http://localhost:8080/health

# Monitor auth logs in Grafana
# Log Analysis dashboard -> filter "auth"
```

**Performance Issues**:
```bash
# Check resource usage in Grafana
# System Health dashboard

# Check API response times
# Load Test Performance dashboard

# Identify bottlenecks
# Prometheus: http://localhost:9090
```

**Monitoring Issues**:
```bash
# Re-setup monitoring provisioning (Grafana datasources/providers)
python3 scripts/setup_monitoring.py

# Restart Grafana
docker restart ml-experiments-grafana

# Check datasource connectivity
# Grafana -> Configuration -> Data Sources
```

### Debug Mode

```bash
# Enable debug logging
export LOG_LEVEL=debug
make test

# Monitor debug logs
# Grafana Log Analysis dashboard
```

## Test Configuration

### Test Configs Location
- `~/.ml/config-admin.toml` - Admin user
- `~/.ml/config-researcher.toml` - Researcher user  
- `~/.ml/config-analyst.toml` - Analyst user

### Test Data
- `tests/fixtures/` - Test data and examples
- `tests/benchmarks/` - Performance test data

### Monitoring Configuration
- `monitoring/grafana/provisioning/` - Auto-provisioned datasources
- `monitoring/grafana/dashboards/` - Auto-provisioned dashboards
- `monitoring/prometheus/prometheus.yml` - Metrics collection

## Continuous Integration

### CI Pipeline with Monitoring

Tests run automatically on:
- **Pull requests**: Full suite + performance benchmarks
- **Main branch**: Unit + integration tests
- **Releases**: Full suite + benchmarks + security scans

### CI Monitoring

During CI runs:
- Performance metrics collected
- Test results tracked in Grafana
- Regression detection
- Automated alerts on failures

## Writing Tests

### Go Tests
- Unit tests: `*_test.go` files
- Integration tests: `tests/e2e/` directory
- Benchmark tests: `tests/benchmarks/` directory

### Test with Monitoring

```go
// Add custom metrics for tests
var testRequests = prometheus.NewCounterVec(
    prometheus.CounterOpts{
        Name: "test_requests_total",
        Help: "Total number of test requests",
    },
    []string{"method", "status"},
)

// Log test events for monitoring
log.WithFields(log.Fields{
    "test": "integration_test",
    "operation": "api_call",
    "status": "success",
}).Info("Test operation completed")
```

### Zig Tests
- CLI tests: `cli/tests/` directory
- Follow Zig testing conventions

## Test Result Analysis

### Grafana Dashboard Analysis

**Load Test Performance**:
- Request rates over time
- Response time percentiles
- Error rate trends
- Throughput metrics

**System Health**:
- Service availability
- Resource utilization
- Memory usage patterns
- CPU consumption

**Log Analysis**:
- Error patterns
- Warning frequency
- Service log aggregation
- Debug information

### Performance Regression Detection

```bash
# Track performance over time
./scripts/track_performance.sh

# Compare with baseline
# Grafana: Compare current run with historical data

# Alert on regressions
# Set up Grafana alerts for performance degradation
```

## Test Cleanup

### Automated Cleanup

```bash
# Clean up test data
make self-cleanup

# Clean up Docker resources
make clean-all

# Reset monitoring data
docker volume rm monitoring_prometheus_data
docker volume rm monitoring_grafana_data
```

### Manual Cleanup

```bash
# Stop test environment
make dev-down

# Remove test artifacts
rm -rf ~/.ml/config-*.toml
rm -rf test-results/
```

## Expected Test Results

### Admin User
```
Status retrieved for user: admin_user (admin: true)
Tasks: X total, X queued, X running, X failed, X completed
```

### Researcher User  
```
Status retrieved for user: researcher1 (admin: false)
Tasks: X total, X queued, X running, X failed, X completed
```

### Analyst User
```
Status retrieved for user: analyst1 (admin: false)
Tasks: X total, X queued, X running, X failed, X completed
```

## Common Commands Reference

```bash
make help              # Show all commands
make test-auth         # Quick auth test
make self-cleanup      # Clean environment
make test-status       # Check system status
make dev-up            # Start dev environment
make dev-down          # Stop dev environment
make dev-status        # Check dev status
```