fetch_ml/DEVELOPMENT.md

# Development Guide

This guide helps developers set up their environment and contribute effectively to FetchML.

## Quick Setup

```bash
# Clone the repository
git clone <your-repo>
cd fetch_ml

# Start development environment
make dev-up

# Run tests
make test
```

## Development Environment

### Prerequisites

- Go 1.25+
- Zig 0.15+
- Python 3.11+
- Docker & Docker Compose
- Redis

### Local Development Setup

1. **Install Go dependencies**
   ```bash
   go mod download
   go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest
   ```

2. **Install Zig tools**
   ```bash
   # Zig is required for building the CLI and running CLI tests
   zig version
   ```

3. **Setup Python environment**
   ```bash
   python -m venv venv
   source venv/bin/activate  # or venv\Scripts\activate on Windows
   # Python is optional (used for a few helper scripts)
   ```

4. **Optional: Install pre-commit hooks**
   ```bash
   # If you want automated code quality checks
   pre-commit install
   pre-commit install --hook-type commit-msg
   ```

### Development Workflow

#### Making Changes

1. Create a feature branch:
   ```bash
   git checkout -b feature/your-feature-name
   ```

2. Make your changes with live feedback:
   ```bash
   # Build Go services + Zig CLI
   make dev

   # Run specific tests
   make test-unit
   make test-integration
   ```

3. Run quality checks:
   ```bash
   # Lint and format (if you have tools configured)
   make lint

   # Full test suite
   make test
   
   # Optional: Pre-commit checks
   pre-commit run --all-files
   ```

#### Testing Strategy

We maintain comprehensive test coverage across multiple categories:

- **Unit tests**: Fast tests for individual components (`tests/unit/`)
- **Integration tests**: Test component interactions (`tests/integration/`)
- **Property-based tests**: Verify invariants using gopter (`tests/property/`)
- **Fault injection tests**: Test failure scenarios (`tests/fault/`)
- **E2E tests**: Full workflow validation (`tests/e2e/`)
- **Performance tests**: Load and stress testing (`tests/benchmarks/`)

**Test Coverage Requirements:**
- All security and reproducibility requirements must have tests (see `docs/TEST_COVERAGE_MAP.md`)
- 49/49 requirements currently covered (100% coverage)
- New features must include tests before merging
- Use `fetchml-vet` custom analyzers for compile-time checks

```bash
# Run tests by type
make test-unit          # Unit tests only
make test-integration   # Integration tests only  
make test-e2e          # End-to-end tests only
make test-property      # Property-based tests (gopter)
make benchmark         # Benchmarks
make load-test         # Load tests
make test-fault        # Fault injection tests (requires FETCH_ML_FAULT_INJECTION=1)
 
# Run with coverage
make test-coverage
 
# Run all tests
make test

# Custom lint analyzers
go run ./tools/fetchml-vet/cmd/fetchml-vet ./...  # Run custom analyzers
```

**Property-Based Testing:**
We use `gopter` for property-based testing. Run with:
```bash
go test ./tests/property/...
```

## Code Quality

### Formatting

- **Go**: `gofmt` and `goimports`
- **Zig**: Built-in formatter
- **Python**: `black`
- **YAML**: `yamllint`
- **Shell**: `shellcheck`

### Linting

- **Go**: `golangci-lint`
- **Python**: `pylint`
- **Docker**: `hadolint`
- **Shell**: `shellcheck`

### Conventional Commits

We use conventional commits for clear commit messages:

```
feat: add new feature
fix: bug fix
docs: documentation changes
style: formatting changes
refactor: code refactoring
test: add or update tests
chore: maintenance tasks
```

## Performance Monitoring

### Local Monitoring

```bash
# Start monitoring stack
make dev-up
 
# View metrics
open http://localhost:3000  # Grafana
open http://localhost:9090  # Prometheus
```

### Performance Testing

```bash
# Load test API
make load-test
 
# Performance benchmarks
make benchmark

# Memory profiling
make profile-memory
```

## Database Management

### Development Database

```bash
# Reset database
make db-reset

# Run migrations
make db-migrate

# Access database
docker compose exec redis redis-cli
```

### Schema Changes

1. Update migration files in `internal/storage/`
2. Test with `make db-test-migration`
3. Update documentation

## Configuration

### Environment Variables

Copy `.env.example` to `.env.local` and adjust:

```bash
cp .env.example .env.local
```

Key variables for development:
- `LOG_LEVEL`: Set to `debug` for verbose logging
- `API_PORT`: API server port (default: 9101)

### Configuration Files

- `configs/api/dev.yaml`: Development (Docker) API server configuration
- `configs/api/homelab-secure.yaml`: Homelab secure API server configuration
- `configs/api/prod.yaml`: Production API server configuration
- `configs/workers/docker.yaml`: Docker worker configuration
- `configs/workers/worker-prod.toml`: Production worker configuration

## IDE Setup

### Recommended Extensions

**VS Code:**
- Go (golang.go)
- Zig (ziglang.vscode-zig)
- Python (ms-python.python)
- Docker (ms-azuretools.vscode-docker)
- YAML (redhat.vscode-yaml)

**Neovim:**
- Go language server (gopls)
- Zig language server (zls)
- Python LSP (pylsp)
- Docker integration

### Configuration

Set up your preferred editor with:
- Go formatting (gofmt, goimports)
- Zig formatting (built-in)
- Python formatting (black, autopep8)
- YAML linting (yamllint)

Configure these tools according to your workflow preferences.

## Troubleshooting

### Common Issues

1. **Go modules not found**: Run `go mod download`
2. **Zig build fails**: Check Zig version with `zig version`
3. **Docker containers not starting**: Check Docker daemon
4. **Redis connection failed**: Start Redis with `docker compose up redis`

### Getting Help

- Check logs: `make logs`
- Run diagnostics: `make doctor`
- Review troubleshooting guide: [docs/src/troubleshooting.md](docs/src/troubleshooting.md)

## Contributing

1. Follow the code of conduct
2. Use conventional commits
3. Add tests for new features
4. Update documentation
5. Ensure all checks pass

See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines.