fetch_ml/AGENTS.md

AGENTS.md - Coding Agent Guidelines

This document provides essential information for agentic coding assistants working with the FetchML codebase.

Build Commands

Go Components

# Build all components (Go binaries + native libs + Zig CLI)
make build

# Build for development (faster compilation, no LTO)
make dev

# Build production-optimized binaries
make prod

# Build production with native C++ libraries
make prod-with-native

# Cross-platform build for Linux/macOS/Windows
make cross-platform

# Build native C++ libraries
make native-build
make native-release
make native-debug

Zig CLI

# Build CLI with release-small optimization (smallest binary)
make build-cli
cd cli && make all
cd cli && zig build --release=small

# Build for production (fast, LTO)
cd cli && make prod
cd cli && zig build --release=fast

# Build for development (fast compilation)
cd cli && make dev
cd cli && zig build --release=fast

# Debug build (fastest compile, no optimizations)
cd cli && make debug
cd cli && zig build -Doptimize=Debug

# Run Zig tests
cd cli && make test
cd cli && zig build test

Test Commands

Running Tests

# Run all tests with Docker infrastructure
make test

# Run unit tests only
make test-unit

# Run integration tests only
make test-integration

# Run end-to-end tests only (Podman tests opt-in)
make test-e2e
FETCH_ML_E2E_PODMAN=1 go test ./tests/e2e/...

# Run with coverage report
make test-coverage

# Run a single test
go test -v ./path/to/package -run TestName

# Run tests with race detector
go test -race ./path/to/package/...

# Run specific test file
go test -v ./tests/unit/auth/keychain_test.go

# Run tests with logging
LOG_LEVEL=debug go test -v ./path/to/package

Performance Testing

# Run performance benchmarks
make benchmark
make benchmark-local
make benchmark-native

# Run load testing suite
make load-test

# Run chaos engineering tests
make chaos-test

# Run complete technical excellence suite
make complete-suite

# Detect performance regressions
make detect-regressions
make profile-tools

# CPU profiling
make profile-load
make profile-ws-queue

Lint Commands

# Run all linters and formatters
make lint

# Run Go linters with golangci-lint
golangci-lint run
golangci-lint run ./path/to/package

# Run Go formatters
gofmt -w ./cmd ./internal ./tests
goimports -w ./cmd ./internal ./tests

# Run Go vet
go vet ./...

# Format Zig code
cd cli && zig fmt .
cd cli && make dev  # includes formatting

# Validate YAML configs against schema
make configlint
make worker-configlint

# Security scans
make security-scan
make gosec
make govulncheck
make check-unsafe

# OpenAPI validation
make openapi-validate
make openapi-generate
make openapi-generate-server

Code Style Guidelines

Go Code Style

1. Formatting:

   • Use gofmt for formatting (configured in .golangci.yml)
   • Line length: 100 characters
   • Use goimports for import organization
   • Run make lint to format all code

2. Naming Conventions:

   • Use camelCase for variables and functions
   • Use PascalCase for exported identifiers (types, interfaces, structs)
   • Use clear, descriptive names
   • Acronyms should be all caps (e.g., HTTP, URL, API, JSON)
   • Interface names: typically end in "er" (e.g., Manager, Handler)

3. Imports:

   • Group imports in order: standard library, third-party, internal
   • Use meaningful import aliases when needed (e.g., jupyter "github.com/...")
   • Remove unused imports (golangci-lint will catch these)
   • Internal imports: github.com/jfraeys/fetch_ml/internal/...

4. Error Handling:

   • Always handle errors explicitly
   • Use errors.Is and errors.As for error checking
   • Wrap errors with context using fmt.Errorf("context: %w", err)
   • Don't ignore errors with _ unless explicitly documented
   • Return errors early when appropriate (guard clauses)

5. Documentation:

   • All exported functions, types, and variables must have doc comments
   • Comments should explain "why" not just "what"
   • Use godoc formatting conventions
   • Include examples for complex functions
   • Document error conditions and edge cases

6. Testing:

   • Use t.Parallel() for independent tests
   • Skip tests conditionally with t.Skip() rather than failing
   • Test files: *_test.go in same package directory
   • Unit tests: tests/unit/
   • Integration tests: tests/integration/
   • E2E tests: tests/e2e/
   • Benchmarks: tests/benchmarks/

Zig Code Style

1. Formatting:

   • Use built-in Zig formatter (zig fmt . in cli/)
   • Consistent indentation with 4 spaces
   • Run make lint to format all Zig code

2. Naming Conventions:

   • Use camelCase for variables and functions
   • Use PascalCase for types (structs, enums, unions)
   • Use snake_case for file names (e.g., utils.zig, colors.zig)
   • Public functions: pub fn
   • Internal functions: fn

3. Error Handling:

   • Use Zig's error union types (!T or Error!T)
   • Handle errors with try, catch, or if expressions
   • Return errors early when appropriate
   • Use catch unreachable only when error is truly impossible
   • Document error returns in function docs

4. Imports:

   • Use @import("module") for modules
   • Organize imports logically
   • Remove unused imports

5. Testing:

   • Tests in cli/tests/ directory
   • Use @import("std").testing for test utilities
   • Run with cd cli && zig build test

Python Code Style

1. Formatting:

   • Use Black formatter with 80 character line limit
   • Use isort for import organization (Google style)
   • Run black . and isort . in project root

2. Type Checking:

   • Use mypy with strict settings
   • All functions should have type annotations
   • Use from __future__ import annotations for forward references

3. Testing:

   • Use pytest framework
   • Follow naming conventions: test_* for functions, *_test.py for files

YAML/Configuration Style

1. Formatting:

   • Use 2 spaces for indentation
   • No trailing whitespace
   • Use yamllint for validation

2. Validation:

   • All configs must pass schema validation
   • Use make configlint and make worker-configlint

Development Guidelines

Testing Requirements (from .windsurf/rules/test-new-features.md)

• MANDATORY: Every new feature MUST include corresponding tests
• Write tests BEFORE implementing complex features (TDD approach)
• Test coverage for new code should be >80%
• Include both unit tests and integration tests where applicable
• Test edge cases, error paths, and boundary conditions

Code Organization

• CRITICAL: Clean up as you go - no orphaned files or dead code
• Remove commented-out code blocks (use git history instead)
• Delete unused imports, functions, and variables immediately
• Consolidate duplicate code into reusable functions
• Move TODO items from loose files into:
  • Code comments with // TODO(context): for implementation tasks
  • GitHub Issues for larger features
  • NEVER create standalone .md files for tracking

Documentation Standards

• Update relevant documentation IN THE SAME COMMIT as code changes
• Documentation locations:
  • README.md: User-facing features, installation, quick start
  • CHANGELOG.md: All changes, following Keep a Changelog format
  • Code comments: Complex logic, non-obvious decisions, API contracts
  • Function/struct docs: Public APIs must have doc comments
• Use concrete examples in documentation
• Keep docs concise but complete

When Making Changes

For EVERY significant change, complete ALL of these:

1. Write/update tests
2. Update documentation (README, CHANGELOG, code comments)
3. Update build scripts if dependencies/build process changed
4. Remove any temporary/debug code added during development
5. Delete unused files created during exploration
6. Verify no dead code remains (unused functions, imports, variables)

Cleanup Checklist (Run BEFORE committing)

• Removed all debug print statements
• Deleted temporary test files
• Removed commented-out code
• Cleaned up unused imports
• Deleted exploratory/spike code
• Consolidated duplicate logic
• Removed obsolete scripts/configs

Communication Style

• Report what you've done: "Added feature X with tests in test/x_test.go"
• Highlight what needs attention: "WARNING: Manual testing needed for edge case Y"
• Ask questions directly: "Should we support Z? Trade-offs are..."
• NEVER say "I'll track this in a markdown file" - use code comments or tell me directly

Script/Build System Updates

• Update Makefile/build.zig when adding new targets or commands
• Modify CI/CD configs (.github/workflows) if build/test process changes
• Update package.json/Cargo.toml/go.mod when dependencies change
• Document new scripts in README under "Development" section

Anti-Patterns to AVOID

• Creating notes.md, todo.md, tasks.md, ideas.md files
• Leaving commented-out code "for reference"
• Keeping old implementation files with .old or .backup suffixes
• Adding features without tests
• Updating code without updating docs
• Leaving TODO comments without context or assignee

Preferred Patterns

• Inline TODO comments: // TODO(user): Add caching layer for better performance
• Self-documenting code with clear names
• Tests that serve as usage examples
• Incremental, complete commits (code + tests + docs)
• Direct communication about tasks and priorities

Definition of Done

A task is complete ONLY when:

1. Code is written and working
2. Tests are written and passing
3. Documentation is updated
4. All temporary/dead code is removed
5. Build scripts are updated if needed
6. Changes are committed with clear message

Commit Message Convention

Use conventional commits:

• feat: for new features
• fix: for bug fixes
• docs: for documentation changes
• style: for formatting changes
• refactor: for code refactoring
• test: for adding/updating tests
• chore: for maintenance tasks

Project Structure

• cmd/ - Main applications
• internal/ - Internal packages
• cli/ - Zig command-line interface
• tests/ - Test files organized by type
• configs/ - Configuration files and schemas
• docs/ - Documentation
• scripts/ - Utility scripts

Dependencies

• Go 1.25+
• Zig 0.15+
• Python 3.11+ (for some tools)
• Redis (for integration tests)
• Docker/Podman (for container-based tests)

Useful Make Targets

# Development workflow
make dev-start     # Start development environment
make test          # Run all tests
make lint          # Run linters
make ci-local      # Run local CI dry-run

# Documentation
make docs          # Start documentation server
make docs-build    # Build static documentation

# Cleanup
make clean         # Remove build artifacts
make clean-docs    # Remove documentation artifacts

Additional Resources

• README.md: Project overview and quick start guide
• DEVELOPMENT.md: Detailed development setup and workflow
• docs/src/testing.md: Comprehensive testing documentation
• cli/README.md: Zig CLI documentation