fetch_ml/docs/src/configuration-reference.md

Configuration Reference

Overview

This document provides a comprehensive reference for all configuration options in the FetchML project.

Environment Configurations

Local Development

File: configs/environments/config-local.yaml

auth:
  enabled: true
  apikeys:
    dev_user:
      hash: "2baf1f40105d9501fe319a8ec463fdf4325a2a5df445adf3f572f626253678c9"
      admin: true
      roles: ["admin"]
      permissions:
        "*": true

server:
  address: ":9101"
  tls:
    enabled: false

security:
  rate_limit:
    enabled: false
  ip_whitelist:
    - "127.0.0.1"
    - "::1"
    - "localhost"

Multi-User Setup

File: configs/environments/config-multi-user.yaml

auth:
  enabled: true
  apikeys:
    admin_user:
      hash: "5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8"
      admin: true
      roles: ["user", "admin"]
      permissions:
        read: true
        write: true
        delete: true
    
    researcher1:
      hash: "ef92b778ba7a6c8f2150019a5678047b6a9a2b95cef8189518f9b35c54d2e3ae"
      admin: false
      roles: ["user", "researcher"]
      permissions:
        jobs:read: true
        jobs:create: true
        jobs:update: true
        jobs:delete: false
    
    analyst1:
      hash: "a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3"
      admin: false
      roles: ["user", "analyst"]
      permissions:
        jobs:read: true
        jobs:create: false
        jobs:update: false
        jobs:delete: false

Production

File: configs/environments/config-prod.yaml

auth:
  enabled: true
  apikeys:
    # Production users configured here

server:
  address: ":9101"
  tls:
    enabled: true
    cert_file: "/app/ssl/cert.pem"
    key_file: "/app/ssl/key.pem"

security:
  rate_limit:
    enabled: true
    requests_per_minute: 30
  ip_whitelist:
    - "127.0.0.1"
    - "::1"
    - "192.168.0.0/16"
    - "10.0.0.0/8"

redis:
  url: "redis://redis:6379"
  max_connections: 10

logging:
  level: "info"
  file: "/app/logs/app.log"
  audit_file: "/app/logs/audit.log"

Worker Configurations

Production Worker

File: configs/workers/worker-prod.toml

[worker]
name = "production-worker"
id = "worker-prod-1"

[server]
host = "api-server"
port = 9101
api_key = "your-api-key-here"

[execution]
max_concurrent_jobs = 2
timeout_minutes = 60
retry_attempts = 3

[resources]
memory_limit = "4Gi"
cpu_limit = "2"
gpu_enabled = false

[storage]
work_dir = "/tmp/fetchml-jobs"
cleanup_interval_minutes = 30

Docker Worker

File: configs/workers/worker-docker.yaml

worker:
  name: "docker-worker"
  id: "worker-docker-1"

server:
  host: "api-server"
  port: 9101
  api_key: "your-api-key-here"

execution:
  max_concurrent_jobs: 1
  timeout_minutes: 30
  retry_attempts: 3

resources:
  memory_limit: "2Gi"
  cpu_limit: "1"
  gpu_enabled: false

docker:
  enabled: true
  image: "fetchml/worker:latest"
  volume_mounts:
    - "/tmp:/tmp"
    - "/var/run/docker.sock:/var/run/docker.sock"

CLI Configuration

User Config File

Location: ~/.ml/config.toml

[server]
worker_host = "localhost"
worker_user = "appuser"
worker_base = "/app"
worker_port = 22

[auth]
api_key = "your-hashed-api-key"

[cli]
default_timeout = 30
verbose = false

Multi-User CLI Configs

Admin Config: ~/.ml/config-admin.toml

[server]
worker_host = "localhost"
worker_user = "appuser"
worker_base = "/app"
worker_port = 22

[auth]
api_key = "5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8"

Researcher Config: ~/.ml/config-researcher.toml

[server]
worker_host = "localhost"
worker_user = "appuser"
worker_base = "/app"
worker_port = 22

[auth]
api_key = "ef92b778ba7a6c8f2150019a5678047b6a9a2b95cef8189518f9b35c54d2e3ae"

Analyst Config: ~/.ml/config-analyst.toml

[server]
worker_host = "localhost"
worker_user = "appuser"
worker_base = "/app"
worker_port = 22

[auth]
api_key = "a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3"

Configuration Options

Authentication

Option	Type	Default	Description
auth.enabled	bool	false	Enable authentication
auth.apikeys	map	{}	API key configurations
auth.apikeys.[user].hash	string	-	SHA256 hash of API key
auth.apikeys.[user].admin	bool	false	Admin privileges
auth.apikeys.[user].roles	array	[]	User roles
auth.apikeys.[user].permissions	map	{}	User permissions

Server

Option	Type	Default	Description
server.address	string	":9101"	Server bind address
server.tls.enabled	bool	false	Enable TLS
server.tls.cert_file	string	-	TLS certificate file
server.tls.key_file	string	-	TLS private key file

Security

Option	Type	Default	Description
security.rate_limit.enabled	bool	true	Enable rate limiting
security.rate_limit.requests_per_minute	int	60	Rate limit
security.ip_whitelist	array	[]	Allowed IP addresses

Redis

Option	Type	Default	Description
redis.url	string	"redis://localhost:6379"	Redis connection URL
redis.max_connections	int	10	Max Redis connections

Logging

Option	Type	Default	Description
logging.level	string	"info"	Log level
logging.file	string	-	Log file path
logging.audit_file	string	-	Audit log path

Permission System

Permission Keys

Permission	Description
jobs:read	Read job information
jobs:create	Create new jobs
jobs:update	Update existing jobs
jobs:delete	Delete jobs
*	All permissions (admin only)

Role-Based Permissions

Role	Default Permissions
admin	All permissions
researcher	jobs:read, jobs:create, jobs:update
analyst	jobs:read
user	No default permissions

Environment Variables

Variable	Default	Description
FETCHML_CONFIG	-	Path to config file
FETCHML_LOG_LEVEL	"info"	Override log level
FETCHML_REDIS_URL	-	Override Redis URL
CLI_CONFIG	-	Path to CLI config file

Troubleshooting

Common Configuration Issues

1. Authentication Failures

   • Check API key hashes are correct SHA256
   • Verify YAML syntax
   • Ensure auth.enabled: true

2. Connection Issues

   • Verify server address and ports
   • Check firewall settings
   • Validate network connectivity

3. Permission Issues

   • Check user roles and permissions
   • Verify permission key format
   • Ensure admin users have "*": true

Configuration Validation

# Validate server configuration
go run cmd/api-server/main.go --config configs/environments/config-local.yaml --validate

# Test CLI configuration
./cli/zig-out/bin/ml status --debug