docs: add comprehensive development guide with flexible tooling

- Add detailed development setup instructions
- Include debugging, testing, and performance monitoring guides
- Provide IDE recommendations without enforcing specific configurations
- Make tooling optional and developer-friendly
- Add database management and configuration guidance

Creates welcoming development environment that respects developer preferences
while providing comprehensive guidance for contributing to FetchML platform.

DEVELOPMENT.md

@@ -0,0 +1,302 @@
+# Development Guide
+
+This guide helps developers set up their environment and contribute effectively to FetchML.
+
+## Quick Setup
+
+```bash
+# Clone the repository
+git clone <your-repo>
+cd fetch_ml
+
+# Install dependencies
+make setup-dev
+
+# Start development environment
+make dev-start
+
+# Run tests
+make test
+```
+
+## Development Environment
+
+### Prerequisites
+
+- Go 1.25+
+- Zig 0.11+
+- Python 3.11+
+- Docker & Docker Compose
+- Redis
+- Node.js (for some tools)
+
+### Local Development Setup
+
+1. **Install Go dependencies**
+   ```bash
+   go mod download
+   go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest
+   ```
+
+2. **Install Zig tools**
+   ```bash
+   # Install Zig language server
+   zig build --install zls
+   ```
+
+3. **Setup Python environment**
+   ```bash
+   python -m venv venv
+   source venv/bin/activate  # or venv\Scripts\activate on Windows
+   pip install -r requirements-dev.txt
+   ```
+
+4. **Optional: Install pre-commit hooks**
+   ```bash
+   # If you want automated code quality checks
+   pre-commit install
+   pre-commit install --hook-type commit-msg
+   ```
+
+### Development Workflow
+
+#### Making Changes
+
+1. Create a feature branch:
+   ```bash
+   git checkout -b feature/your-feature-name
+   ```
+
+2. Make your changes with live feedback:
+   ```bash
+   # Go development with hot reload
+   make dev-go
+
+   # Zig development with build on save
+   make dev-zig
+
+   # Run specific tests
+   make test-unit
+   make test-integration
+   ```
+
+3. Run quality checks:
+   ```bash
+   # Lint and format (if you have tools configured)
+   make lint
+   make format
+
+   # Full test suite
+   make test-all
+
+   # Optional: Pre-commit checks
+   pre-commit run --all-files
+   ```
+
+#### Testing Strategy
+
+- **Unit tests**: Fast tests for individual components
+- **Integration tests**: Test component interactions
+- **E2E tests**: Full workflow validation
+- **Performance tests**: Load and stress testing
+
+```bash
+# Run tests by type
+make test-unit          # Unit tests only
+make test-integration   # Integration tests only  
+make test-e2e          # End-to-end tests only
+make test-performance  # Performance tests only
+
+# Run with coverage
+make test-coverage
+
+# Watch mode for development
+make test-watch
+```
+
+## Code Quality
+
+### Formatting
+
+- **Go**: `gofmt` and `goimports`
+- **Zig**: Built-in formatter
+- **Python**: `black`
+- **YAML**: `yamllint`
+- **Shell**: `shellcheck`
+
+### Linting
+
+- **Go**: `golangci-lint`
+- **Python**: `pylint`
+- **Docker**: `hadolint`
+- **Shell**: `shellcheck`
+
+### Conventional Commits
+
+We use conventional commits for clear commit messages:
+
+```
+feat: add new feature
+fix: bug fix
+docs: documentation changes
+style: formatting changes
+refactor: code refactoring
+test: add or update tests
+chore: maintenance tasks
+```
+
+## Debugging
+
+### Go Debugging
+
+```bash
+# Debug with delve
+dlv debug cmd/api-server/main.go
+
+# Debug tests
+dlv test ./internal/...
+
+# Profile with pprof
+go tool pprof http://localhost:6060/debug/pprof/profile
+```
+
+### Zig Debugging
+
+```bash
+# Debug build
+zig build-exe -O Debug -fstrip=false your_file.zig
+
+# Test with debugging
+zig test --gdb your_file.zig
+```
+
+### Container Debugging
+
+```bash
+# Debug containers
+docker-compose exec api-server bash
+docker-compose logs -f api-server
+
+# Inspect running processes
+docker-compose exec api-server ps aux
+```
+
+## Performance Monitoring
+
+### Local Monitoring
+
+```bash
+# Start monitoring stack
+make monitoring-start
+
+# View metrics
+open http://localhost:3000  # Grafana
+open http://localhost:9090  # Prometheus
+```
+
+### Performance Testing
+
+```bash
+# Load test API
+make load-test-api
+
+# Performance benchmarks
+make benchmark
+
+# Memory profiling
+make profile-memory
+```
+
+## Database Management
+
+### Development Database
+
+```bash
+# Reset database
+make db-reset
+
+# Run migrations
+make db-migrate
+
+# Access database
+docker-compose exec redis redis-cli
+```
+
+### Schema Changes
+
+1. Update migration files in `internal/storage/`
+2. Test with `make db-test-migration`
+3. Update documentation
+
+## Configuration
+
+### Environment Variables
+
+Copy `.env.example` to `.env.local` and adjust:
+
+```bash
+cp .env.example .env.local
+```
+
+Key variables for development:
+- `REDIS_URL`: Redis connection string
+- `LOG_LEVEL`: Set to `debug` for verbose logging
+- `API_PORT`: API server port (default: 9101)
+
+### Configuration Files
+
+- `configs/config-dev.yaml`: Development configuration
+- `configs/config-local.yaml`: Local overrides
+- `configs/config-prod.yaml`: Production settings
+
+## IDE Setup
+
+### Recommended Extensions
+
+**VS Code:**
+- Go (golang.go)
+- Zig (ziglang.vscode-zig)
+- Python (ms-python.python)
+- Docker (ms-azuretools.vscode-docker)
+- YAML (redhat.vscode-yaml)
+
+**Neovim:**
+- Go language server (gopls)
+- Zig language server (zls)
+- Python LSP (pylsp)
+- Docker integration
+
+### Configuration
+
+Set up your preferred editor with:
+- Go formatting (gofmt, goimports)
+- Zig formatting (built-in)
+- Python formatting (black, autopep8)
+- YAML linting (yamllint)
+
+Configure these tools according to your workflow preferences.
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Go modules not found**: Run `go mod download`
+2. **Zig build fails**: Check Zig version with `zig version`
+3. **Docker containers not starting**: Check Docker daemon
+4. **Redis connection failed**: Start Redis with `docker-compose up redis`
+
+### Getting Help
+
+- Check logs: `make logs`
+- Run diagnostics: `make doctor`
+- Review troubleshooting guide: [docs/src/troubleshooting.md](docs/src/troubleshooting.md)
+
+## Contributing
+
+1. Follow the code of conduct
+2. Use conventional commits
+3. Add tests for new features
+4. Update documentation
+5. Ensure all checks pass
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines.