History

Jeremie Fraeys 86f9ae5a7e docs(config): reorganize configuration structure and add documentation Restructure configuration files for better organization: - Add scheduler configuration examples (scheduler.yaml.example) - Reorganize worker configs into subdirectories: - distributed/ - Multi-node cluster configurations - standalone/ - Single-node deployment configs - Add environment-specific configs: - dev-local.yaml, docker-dev.yaml, docker-prod.yaml - homelab-secure.yaml, worker-prod.toml - Add deployment configs for different security modes: - docker-standard.yaml, docker-hipaa.yaml, docker-dev.yaml Add documentation: - configs/README.md with configuration guidelines - configs/SECURITY.md with security configuration best practices		2026-02-26 12:04:11 -05:00
..
api	feat: update CLI, TUI, and security documentation	2026-02-19 15:35:05 -05:00
examples	chore(ops): reorganize deployments/monitoring and remove legacy scripts	2026-01-05 12:31:26 -05:00
scheduler	docs(config): reorganize configuration structure and add documentation	2026-02-26 12:04:11 -05:00
schema	chore(config): update configurations and deployment scripts	2026-02-12 12:05:37 -05:00
seccomp	feat(security): implement comprehensive security hardening phases 1-5,7	2026-02-23 18:00:33 -05:00
worker	docs(config): reorganize configuration structure and add documentation	2026-02-26 12:04:11 -05:00
README.md	docs(config): reorganize configuration structure and add documentation	2026-02-26 12:04:11 -05:00
SECURITY.md	docs(config): reorganize configuration structure and add documentation	2026-02-26 12:04:11 -05:00

README.md

fetch_ml Configuration Guide

Quick Start

Standalone Mode (Existing Behavior)

# Single worker, direct queue access
go run ./cmd/worker -config configs/worker/standalone/worker.yaml

Distributed Mode

# Terminal 1: Start scheduler
go run ./cmd/scheduler -config configs/scheduler/scheduler.yaml

# Terminal 2: Start worker
go run ./cmd/worker -config configs/worker/distributed/worker.yaml

Single-Node Mode (Zero Config)

# Both scheduler and worker in one process
go run ./cmd/fetch_ml -config configs/multi-node/single-node.yaml

Config Structure

configs/
├── scheduler/
│   └── scheduler.yaml       # Central scheduler configuration
├── worker/
│   ├── standalone/
│   │   └── worker.yaml      # Direct queue access (Redis/SQLite)
│   └── distributed/
│       └── worker.yaml      # WebSocket to scheduler
└── multi-node/
    └── single-node.yaml     # Combined scheduler+worker

Key Configuration Modes

Mode	Use Case	Backend
`standalone`	Single machine, existing behavior	Redis/SQLite/Filesystem
`distributed`	Multiple workers, central scheduler	WebSocket to scheduler
`both`	Quick testing, single process	In-process scheduler

Worker Mode Selection

Set worker.mode to switch between implementations:

worker:
  mode: "standalone"    # Uses Redis/SQLite queue.Backend
  # OR
  mode: "distributed"   # Uses SchedulerBackend over WebSocket

The worker code is unchanged — only the backend implementation changes.