fetch_ml/docs/src/smart-defaults.md

Smart Defaults

This document describes Fetch ML's smart defaults system, which automatically adapts configuration based on the runtime environment.

Overview

Smart defaults eliminate the need for manual configuration tweaks when running in different environments:

• Local Development: Optimized for developer machines with sensible paths and localhost services
• Container Environments: Uses container-friendly hostnames and paths
• CI/CD: Optimized for automated testing with fast polling and minimal resource usage
• Production: Uses production-ready defaults with proper security and scaling

Environment Detection

The system automatically detects the environment based on:

1. CI Detection: Checks for CI, GITHUB_ACTIONS, GITLAB_CI environment variables
2. Container Detection: Looks for /.dockerenv, KUBERNETES_SERVICE_HOST, or CONTAINER variables
3. Production Detection: Checks FETCH_ML_ENV=production or ENV=production
4. Default: Falls back to local development

Default Values by Environment

Host Configuration

• Local: localhost
• Container/CI: host.docker.internal (Docker Desktop/Colima)
• Production: 0.0.0.0

Base Paths

• Local: ~/ml-experiments
• Container/CI: /workspace/ml-experiments
• Production: /var/lib/fetch_ml/experiments

Data Directory

• Local: ~/ml-data
• Container/CI: /workspace/data
• Production: /var/lib/fetch_ml/data

Redis Address

• Local: localhost:6379
• Container/CI: redis:6379 (service name)
• Production: redis:6379

SSH Configuration

• Local: ~/.ssh/id_rsa and ~/.ssh/known_hosts
• Container/CI: /workspace/.ssh/id_rsa and /workspace/.ssh/known_hosts
• Production: /etc/fetch_ml/ssh/id_rsa and /etc/fetch_ml/ssh/known_hosts

Worker Configuration

• Local: 2 workers, 5-second poll interval
• CI: 1 worker, 1-second poll interval (fast testing)
• Production: CPU core count workers, 10-second poll interval

Log Levels

• Local: info
• CI: debug (verbose for debugging)
• Production: info

Usage

In Configuration Loaders

// Get smart defaults for current environment
smart := config.GetSmartDefaults()

// Use smart defaults
if cfg.Host == "" {
    cfg.Host = smart.Host()
}
if cfg.BasePath == "" {
    cfg.BasePath = smart.BasePath()
}

Environment Overrides

Smart defaults can be overridden with environment variables:

• FETCH_ML_HOST - Override host
• FETCH_ML_BASE_PATH - Override base path
• FETCH_ML_REDIS_ADDR - Override Redis address
• FETCH_ML_ENV - Force environment profile

Manual Environment Selection

You can force a specific environment:

# Force production mode
export FETCH_ML_ENV=production

# Force container mode
export CONTAINER=true

Implementation Details

The smart defaults system is implemented in internal/config/smart_defaults.go:

• DetectEnvironment() - Determines current environment profile
• SmartDefaults struct - Provides environment-aware defaults
• Helper methods for each configuration value

Migration Guide

For Users

No changes required - existing configurations continue to work. Smart defaults only apply when values are not explicitly set.

For Developers

When adding new configuration options:

1. Add a method to SmartDefaults struct
2. Use the smart default in config loaders
3. Document the environment-specific values

Example:

// Add to SmartDefaults struct
func (s *SmartDefaults) NewFeature() string {
    switch s.Profile {
    case ProfileContainer, ProfileCI:
        return "/workspace/new-feature"
    case ProfileProduction:
        return "/var/lib/fetch_ml/new-feature"
    default:
        return "./new-feature"
    }
}

// Use in config loader
if cfg.NewFeature == "" {
    cfg.NewFeature = smart.NewFeature()
}

Testing

To test different environments:

# Test local defaults (default)
./bin/worker

# Test container defaults
export CONTAINER=true
./bin/worker

# Test CI defaults
export CI=true
./bin/worker

# Test production defaults
export FETCH_ML_ENV=production
./bin/worker

Troubleshooting

Wrong Environment Detection

Check environment variables:

echo "CI: $CI"
echo "CONTAINER: $CONTAINER"
echo "FETCH_ML_ENV: $FETCH_ML_ENV"

Path Issues

Smart defaults expand ~ and environment variables automatically. If paths don't work as expected:

1. Check the detected environment: config.GetSmartDefaults().GetEnvironmentDescription()
2. Verify the path exists in the target environment
3. Override with environment variable if needed

Container Networking

For container environments, ensure:

• Redis service is named redis in docker-compose
• Host networking is configured properly
• host.docker.internal resolves (Docker Desktop/Colima)