fetch_ml/DEVELOPMENT.md

Development Guide

This guide helps developers set up their environment and contribute effectively to FetchML.

Quick Setup

# Clone the repository
git clone <your-repo>
cd fetch_ml

# Install dependencies
make setup-dev

# Start development environment
make dev-start

# Run tests
make test

Development Environment

Prerequisites

• Go 1.25+
• Zig 0.11+
• Python 3.11+
• Docker & Docker Compose
• Redis
• Node.js (for some tools)

Local Development Setup

1. Install Go dependencies

   go mod download
   go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest
   

2. Install Zig tools

   # Install Zig language server
   zig build --install zls
   

3. Setup Python environment

   python -m venv venv
   source venv/bin/activate  # or venv\Scripts\activate on Windows
   pip install -r requirements-dev.txt
   

4. Optional: Install pre-commit hooks

   # If you want automated code quality checks
   pre-commit install
   pre-commit install --hook-type commit-msg
   

Development Workflow

Making Changes

1. Create a feature branch:

   git checkout -b feature/your-feature-name
   

2. Make your changes with live feedback:

   # Go development with hot reload
   make dev-go
   
   # Zig development with build on save
   make dev-zig
   
   # Run specific tests
   make test-unit
   make test-integration
   

3. Run quality checks:

   # Lint and format (if you have tools configured)
   make lint
   make format
   
   # Full test suite
   make test-all
   
   # Optional: Pre-commit checks
   pre-commit run --all-files
   

Testing Strategy

• Unit tests: Fast tests for individual components
• Integration tests: Test component interactions
• E2E tests: Full workflow validation
• Performance tests: Load and stress testing

# Run tests by type
make test-unit          # Unit tests only
make test-integration   # Integration tests only  
make test-e2e          # End-to-end tests only
make test-performance  # Performance tests only

# Run with coverage
make test-coverage

# Watch mode for development
make test-watch

Code Quality

Formatting

• Go: gofmt and goimports
• Zig: Built-in formatter
• Python: black
• YAML: yamllint
• Shell: shellcheck

Linting

• Go: golangci-lint
• Python: pylint
• Docker: hadolint
• Shell: shellcheck

Conventional Commits

We use conventional commits for clear commit messages:

feat: add new feature
fix: bug fix
docs: documentation changes
style: formatting changes
refactor: code refactoring
test: add or update tests
chore: maintenance tasks

Debugging

Go Debugging

# Debug with delve
dlv debug cmd/api-server/main.go

# Debug tests
dlv test ./internal/...

# Profile with pprof
go tool pprof http://localhost:6060/debug/pprof/profile

Zig Debugging

# Debug build
zig build-exe -O Debug -fstrip=false your_file.zig

# Test with debugging
zig test --gdb your_file.zig

Container Debugging

# Debug containers
docker-compose exec api-server bash
docker-compose logs -f api-server

# Inspect running processes
docker-compose exec api-server ps aux

Performance Monitoring

Local Monitoring

# Start monitoring stack
make monitoring-start

# View metrics
open http://localhost:3000  # Grafana
open http://localhost:9090  # Prometheus

Performance Testing

# Load test API
make load-test-api

# Performance benchmarks
make benchmark

# Memory profiling
make profile-memory

Database Management

Development Database

# Reset database
make db-reset

# Run migrations
make db-migrate

# Access database
docker-compose exec redis redis-cli

Schema Changes

1. Update migration files in internal/storage/
2. Test with make db-test-migration
3. Update documentation

Configuration

Environment Variables

Copy .env.example to .env.local and adjust:

cp .env.example .env.local

Key variables for development:

• LOG_LEVEL: Set to debug for verbose logging
• API_PORT: API server port (default: 9101)

Configuration Files

• configs/api/dev.yaml: Development (Docker) API server configuration
• configs/api/homelab-secure.yaml: Homelab secure API server configuration
• configs/api/prod.yaml: Production API server configuration
• configs/workers/docker.yaml: Docker worker configuration
• configs/workers/worker-prod.toml: Production worker configuration

IDE Setup

Recommended Extensions

VS Code:

• Go (golang.go)
• Zig (ziglang.vscode-zig)
• Python (ms-python.python)
• Docker (ms-azuretools.vscode-docker)
• YAML (redhat.vscode-yaml)

Neovim:

• Go language server (gopls)
• Zig language server (zls)
• Python LSP (pylsp)
• Docker integration

Configuration

Set up your preferred editor with:

• Go formatting (gofmt, goimports)
• Zig formatting (built-in)
• Python formatting (black, autopep8)
• YAML linting (yamllint)

Configure these tools according to your workflow preferences.

Troubleshooting

Common Issues

1. Go modules not found: Run go mod download
2. Zig build fails: Check Zig version with zig version
3. Docker containers not starting: Check Docker daemon
4. Redis connection failed: Start Redis with docker-compose up redis

Getting Help

• Check logs: make logs
• Run diagnostics: make doctor
• Review troubleshooting guide: docs/src/troubleshooting.md

Contributing

1. Follow the code of conduct
2. Use conventional commits
3. Add tests for new features
4. Update documentation
5. Ensure all checks pass

See CONTRIBUTING.md for detailed guidelines.