fetch_ml/DEVELOPMENT.md

# Development Guide

This guide helps developers set up their environment and contribute effectively to FetchML.

## Quick Setup

```bash
# Clone the repository
git clone <your-repo>
cd fetch_ml

# Install dependencies
make setup-dev

# Start development environment
make dev-start

# Run tests
make test
```

## Development Environment

### Prerequisites

- Go 1.25+
- Zig 0.11+
- Python 3.11+
- Docker & Docker Compose
- Redis
- Node.js (for some tools)

### Local Development Setup

1. **Install Go dependencies**
   ```bash
   go mod download
   go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest
   ```

2. **Install Zig tools**
   ```bash
   # Install Zig language server
   zig build --install zls
   ```

3. **Setup Python environment**
   ```bash
   python -m venv venv
   source venv/bin/activate  # or venv\Scripts\activate on Windows
   pip install -r requirements-dev.txt
   ```

4. **Optional: Install pre-commit hooks**
   ```bash
   # If you want automated code quality checks
   pre-commit install
   pre-commit install --hook-type commit-msg
   ```

### Development Workflow

#### Making Changes

1. Create a feature branch:
   ```bash
   git checkout -b feature/your-feature-name
   ```

2. Make your changes with live feedback:
   ```bash
   # Go development with hot reload
   make dev-go

   # Zig development with build on save
   make dev-zig

   # Run specific tests
   make test-unit
   make test-integration
   ```

3. Run quality checks:
   ```bash
   # Lint and format (if you have tools configured)
   make lint
   make format

   # Full test suite
   make test-all

   # Optional: Pre-commit checks
   pre-commit run --all-files
   ```

#### Testing Strategy

- **Unit tests**: Fast tests for individual components
- **Integration tests**: Test component interactions
- **E2E tests**: Full workflow validation
- **Performance tests**: Load and stress testing

```bash
# Run tests by type
make test-unit          # Unit tests only
make test-integration   # Integration tests only  
make test-e2e          # End-to-end tests only
make test-performance  # Performance tests only

# Run with coverage
make test-coverage

# Watch mode for development
make test-watch
```

## Code Quality

### Formatting

- **Go**: `gofmt` and `goimports`
- **Zig**: Built-in formatter
- **Python**: `black`
- **YAML**: `yamllint`
- **Shell**: `shellcheck`

### Linting

- **Go**: `golangci-lint`
- **Python**: `pylint`
- **Docker**: `hadolint`
- **Shell**: `shellcheck`

### Conventional Commits

We use conventional commits for clear commit messages:

```
feat: add new feature
fix: bug fix
docs: documentation changes
style: formatting changes
refactor: code refactoring
test: add or update tests
chore: maintenance tasks
```

## Debugging

### Go Debugging

```bash
# Debug with delve
dlv debug cmd/api-server/main.go

# Debug tests
dlv test ./internal/...

# Profile with pprof
go tool pprof http://localhost:6060/debug/pprof/profile
```

### Zig Debugging

```bash
# Debug build
zig build-exe -O Debug -fstrip=false your_file.zig

# Test with debugging
zig test --gdb your_file.zig
```

### Container Debugging

```bash
# Debug containers
docker-compose exec api-server bash
docker-compose logs -f api-server

# Inspect running processes
docker-compose exec api-server ps aux
```

## Performance Monitoring

### Local Monitoring

```bash
# Start monitoring stack
make monitoring-start

# View metrics
open http://localhost:3000  # Grafana
open http://localhost:9090  # Prometheus
```

### Performance Testing

```bash
# Load test API
make load-test-api

# Performance benchmarks
make benchmark

# Memory profiling
make profile-memory
```

## Database Management

### Development Database

```bash
# Reset database
make db-reset

# Run migrations
make db-migrate

# Access database
docker-compose exec redis redis-cli
```

### Schema Changes

1. Update migration files in `internal/storage/`
2. Test with `make db-test-migration`
3. Update documentation

## Configuration

### Environment Variables

Copy `.env.example` to `.env.local` and adjust:

```bash
cp .env.example .env.local
```

Key variables for development:
- `REDIS_URL`: Redis connection string
- `LOG_LEVEL`: Set to `debug` for verbose logging
- `API_PORT`: API server port (default: 9101)

### Configuration Files

- `configs/config-dev.yaml`: Development configuration
- `configs/config-local.yaml`: Local overrides
- `configs/config-prod.yaml`: Production settings

## IDE Setup

### Recommended Extensions

**VS Code:**
- Go (golang.go)
- Zig (ziglang.vscode-zig)
- Python (ms-python.python)
- Docker (ms-azuretools.vscode-docker)
- YAML (redhat.vscode-yaml)

**Neovim:**
- Go language server (gopls)
- Zig language server (zls)
- Python LSP (pylsp)
- Docker integration

### Configuration

Set up your preferred editor with:
- Go formatting (gofmt, goimports)
- Zig formatting (built-in)
- Python formatting (black, autopep8)
- YAML linting (yamllint)

Configure these tools according to your workflow preferences.

## Troubleshooting

### Common Issues

1. **Go modules not found**: Run `go mod download`
2. **Zig build fails**: Check Zig version with `zig version`
3. **Docker containers not starting**: Check Docker daemon
4. **Redis connection failed**: Start Redis with `docker-compose up redis`

### Getting Help

- Check logs: `make logs`
- Run diagnostics: `make doctor`
- Review troubleshooting guide: [docs/src/troubleshooting.md](docs/src/troubleshooting.md)

## Contributing

1. Follow the code of conduct
2. Use conventional commits
3. Add tests for new features
4. Update documentation
5. Ensure all checks pass

See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines.