fetch_ml/AGENTS.md

# AGENTS.md - Coding Agent Guidelines

This document provides essential information for agentic coding assistants working with the FetchML codebase.

## Build Commands

### Go Components
```bash
# Build all components (Go binaries + native libs + Zig CLI)
make build

# Build for development (faster compilation, no LTO)
make dev

# Build production-optimized binaries
make prod

# Build production with native C++ libraries
make prod-with-native

# Cross-platform build for Linux/macOS/Windows
make cross-platform

# Build native C++ libraries
make native-build
make native-release
make native-debug
```

### Zig CLI
```bash
# Build CLI with release-small optimization (smallest binary)
make build-cli
cd cli && make all
cd cli && zig build --release=small

# Build for production (fast, LTO)
cd cli && make prod
cd cli && zig build --release=fast

# Build for development (fast compilation)
cd cli && make dev
cd cli && zig build --release=fast

# Debug build (fastest compile, no optimizations)
cd cli && make debug
cd cli && zig build -Doptimize=Debug

# Run Zig tests
cd cli && make test
cd cli && zig build test
```

## Test Commands

### Running Tests
```bash
# Run all tests with Docker infrastructure
make test

# Run unit tests only
make test-unit

# Run integration tests only
make test-integration

# Run end-to-end tests only (Podman tests opt-in)
make test-e2e
FETCH_ML_E2E_PODMAN=1 go test ./tests/e2e/...

# Run with coverage report
make test-coverage

# Run a single test
go test -v ./path/to/package -run TestName

# Run tests with race detector
go test -race ./path/to/package/...

# Run specific test file
go test -v ./tests/unit/auth/keychain_test.go

# Run tests with logging
LOG_LEVEL=debug go test -v ./path/to/package
```

### Performance Testing
```bash
# Run performance benchmarks
make benchmark
make benchmark-local
make benchmark-native

# Run load testing suite
make load-test

# Run chaos engineering tests
make chaos-test

# Run complete technical excellence suite
make complete-suite

# Detect performance regressions
make detect-regressions
make profile-tools

# CPU profiling
make profile-load
make profile-ws-queue
```

## Lint Commands

```bash
# Run all linters and formatters
make lint

# Run Go linters with golangci-lint
golangci-lint run
golangci-lint run ./path/to/package

# Run Go formatters
gofmt -w ./cmd ./internal ./tests
goimports -w ./cmd ./internal ./tests

# Run Go vet
go vet ./...

# Format Zig code
cd cli && zig fmt .
cd cli && make dev  # includes formatting

# Validate YAML configs against schema
make configlint
make worker-configlint

# Security scans
make security-scan
make gosec
make govulncheck
make check-unsafe

# OpenAPI validation
make openapi-validate
make openapi-generate
make openapi-generate-server
```

## Code Style Guidelines

### Go Code Style

1. **Formatting**:
   - Use `gofmt` for formatting (configured in .golangci.yml)
   - Line length: 100 characters
   - Use `goimports` for import organization
   - Run `make lint` to format all code

2. **Naming Conventions**:
   - Use camelCase for variables and functions
   - Use PascalCase for exported identifiers (types, interfaces, structs)
   - Use clear, descriptive names
   - Acronyms should be all caps (e.g., HTTP, URL, API, JSON)
   - Interface names: typically end in "er" (e.g., Manager, Handler)

3. **Imports**:
   - Group imports in order: standard library, third-party, internal
   - Use meaningful import aliases when needed (e.g., `jupyter "github.com/..."`)
   - Remove unused imports (golangci-lint will catch these)
   - Internal imports: `github.com/jfraeys/fetch_ml/internal/...`

4. **Error Handling**:
   - Always handle errors explicitly
   - Use `errors.Is` and `errors.As` for error checking
   - Wrap errors with context using `fmt.Errorf("context: %w", err)`
   - Don't ignore errors with `_` unless explicitly documented
   - Return errors early when appropriate (guard clauses)

5. **Documentation**:
   - All exported functions, types, and variables must have doc comments
   - Comments should explain "why" not just "what"
   - Use godoc formatting conventions
   - Include examples for complex functions
   - Document error conditions and edge cases

6. **Testing**:
   - Use `t.Parallel()` for independent tests
   - Skip tests conditionally with `t.Skip()` rather than failing
   - Test files: `*_test.go` in same package directory
   - Unit tests: `tests/unit/`
   - Integration tests: `tests/integration/`
   - E2E tests: `tests/e2e/`
   - Benchmarks: `tests/benchmarks/`

### Zig Code Style

1. **Formatting**:
   - Use built-in Zig formatter (`zig fmt .` in cli/)
   - Consistent indentation with 4 spaces
   - Run `make lint` to format all Zig code

2. **Naming Conventions**:
   - Use camelCase for variables and functions
   - Use PascalCase for types (structs, enums, unions)
   - Use snake_case for file names (e.g., `utils.zig`, `colors.zig`)
   - Public functions: `pub fn`
   - Internal functions: `fn`

3. **Error Handling**:
   - Use Zig's error union types (`!T` or `Error!T`)
   - Handle errors with `try`, `catch`, or `if` expressions
   - Return errors early when appropriate
   - Use `catch unreachable` only when error is truly impossible
   - Document error returns in function docs

4. **Imports**:
   - Use `@import("module")` for modules
   - Organize imports logically
   - Remove unused imports

5. **Testing**:
   - Tests in `cli/tests/` directory
   - Use `@import("std").testing` for test utilities
   - Run with `cd cli && zig build test`

### Python Code Style

1. **Formatting**:
   - Use Black formatter with 80 character line limit
   - Use isort for import organization (Google style)
   - Run `black .` and `isort .` in project root

2. **Type Checking**:
   - Use mypy with strict settings
   - All functions should have type annotations
   - Use `from __future__ import annotations` for forward references

3. **Testing**:
   - Use pytest framework
   - Follow naming conventions: `test_*` for functions, `*_test.py` for files

### YAML/Configuration Style

1. **Formatting**:
   - Use 2 spaces for indentation
   - No trailing whitespace
   - Use yamllint for validation

2. **Validation**:
   - All configs must pass schema validation
   - Use `make configlint` and `make worker-configlint`

## Development Guidelines

### Testing Requirements (from .windsurf/rules/test-new-features.md)
- MANDATORY: Every new feature MUST include corresponding tests
- Write tests BEFORE implementing complex features (TDD approach)
- Test coverage for new code should be >80%
- Include both unit tests and integration tests where applicable
- Test edge cases, error paths, and boundary conditions

### Code Organization
- CRITICAL: Clean up as you go - no orphaned files or dead code
- Remove commented-out code blocks (use git history instead)
- Delete unused imports, functions, and variables immediately
- Consolidate duplicate code into reusable functions
- Move TODO items from loose files into:
  - Code comments with `// TODO(context):` for implementation tasks
  - GitHub Issues for larger features
  - NEVER create standalone .md files for tracking

### Documentation Standards
- Update relevant documentation IN THE SAME COMMIT as code changes
- Documentation locations:
  - README.md: User-facing features, installation, quick start
  - CHANGELOG.md: All changes, following Keep a Changelog format
  - Code comments: Complex logic, non-obvious decisions, API contracts
  - Function/struct docs: Public APIs must have doc comments
- Use concrete examples in documentation
- Keep docs concise but complete

### When Making Changes
For EVERY significant change, complete ALL of these:

1. Write/update tests
2. Update documentation (README, CHANGELOG, code comments)
3. Update build scripts if dependencies/build process changed
4. Remove any temporary/debug code added during development
5. Delete unused files created during exploration
6. Verify no dead code remains (unused functions, imports, variables)

### Cleanup Checklist (Run BEFORE committing)
- [ ] Removed all debug print statements
- [ ] Deleted temporary test files
- [ ] Removed commented-out code
- [ ] Cleaned up unused imports
- [ ] Deleted exploratory/spike code
- [ ] Consolidated duplicate logic
- [ ] Removed obsolete scripts/configs

### Communication Style
- Report what you've done: "Added feature X with tests in test/x_test.go"
- Highlight what needs attention: "WARNING: Manual testing needed for edge case Y"
- Ask questions directly: "Should we support Z? Trade-offs are..."
- NEVER say "I'll track this in a markdown file" - use code comments or tell me directly

### Script/Build System Updates
- Update Makefile/build.zig when adding new targets or commands
- Modify CI/CD configs (.github/workflows) if build/test process changes
- Update package.json/Cargo.toml/go.mod when dependencies change
- Document new scripts in README under "Development" section

## Anti-Patterns to AVOID
- Creating notes.md, todo.md, tasks.md, ideas.md files
- Leaving commented-out code "for reference"
- Keeping old implementation files with .old or .backup suffixes
- Adding features without tests
- Updating code without updating docs
- Leaving TODO comments without context or assignee

## Preferred Patterns
- Inline TODO comments: `// TODO(user): Add caching layer for better performance`
- Self-documenting code with clear names
- Tests that serve as usage examples
- Incremental, complete commits (code + tests + docs)
- Direct communication about tasks and priorities

## Definition of Done
A task is complete ONLY when:
1. Code is written and working
2. Tests are written and passing
3. Documentation is updated
4. All temporary/dead code is removed
5. Build scripts are updated if needed
6. Changes are committed with clear message

### Commit Message Convention
Use conventional commits:
- `feat:` for new features
- `fix:` for bug fixes
- `docs:` for documentation changes
- `style:` for formatting changes
- `refactor:` for code refactoring
- `test:` for adding/updating tests
- `chore:` for maintenance tasks

## Project Structure

- `cmd/` - Main applications
- `internal/` - Internal packages
- `cli/` - Zig command-line interface
- `tests/` - Test files organized by type
- `configs/` - Configuration files and schemas
- `docs/` - Documentation
- `scripts/` - Utility scripts

## Dependencies

- Go 1.25+
- Zig 0.15+
- Python 3.11+ (for some tools)
- Redis (for integration tests)
- Docker/Podman (for container-based tests)

## Useful Make Targets

```bash
# Development workflow
make dev-start     # Start development environment
make test          # Run all tests
make lint          # Run linters
make ci-local      # Run local CI dry-run

# Documentation
make docs          # Start documentation server
make docs-build    # Build static documentation

# Cleanup
make clean         # Remove build artifacts
make clean-docs    # Remove documentation artifacts
```

## Additional Resources

- **README.md**: Project overview and quick start guide
- **DEVELOPMENT.md**: Detailed development setup and workflow
- **docs/src/testing.md**: Comprehensive testing documentation
- **cli/README.md**: Zig CLI documentation