86f9ae5a7e
Restructure configuration files for better organization: - Add scheduler configuration examples (scheduler.yaml.example) - Reorganize worker configs into subdirectories: - distributed/ - Multi-node cluster configurations - standalone/ - Single-node deployment configs - Add environment-specific configs: - dev-local.yaml, docker-dev.yaml, docker-prod.yaml - homelab-secure.yaml, worker-prod.toml - Add deployment configs for different security modes: - docker-standard.yaml, docker-hipaa.yaml, docker-dev.yaml Add documentation: - configs/README.md with configuration guidelines - configs/SECURITY.md with security configuration best practices
1.6 KiB
1.6 KiB
fetch_ml Configuration Guide
Quick Start
Standalone Mode (Existing Behavior)
# Single worker, direct queue access
go run ./cmd/worker -config configs/worker/standalone/worker.yaml
Distributed Mode
# Terminal 1: Start scheduler
go run ./cmd/scheduler -config configs/scheduler/scheduler.yaml
# Terminal 2: Start worker
go run ./cmd/worker -config configs/worker/distributed/worker.yaml
Single-Node Mode (Zero Config)
# Both scheduler and worker in one process
go run ./cmd/fetch_ml -config configs/multi-node/single-node.yaml
Config Structure
configs/
├── scheduler/
│ └── scheduler.yaml # Central scheduler configuration
├── worker/
│ ├── standalone/
│ │ └── worker.yaml # Direct queue access (Redis/SQLite)
│ └── distributed/
│ └── worker.yaml # WebSocket to scheduler
└── multi-node/
└── single-node.yaml # Combined scheduler+worker
Key Configuration Modes
| Mode | Use Case | Backend |
|---|---|---|
standalone |
Single machine, existing behavior | Redis/SQLite/Filesystem |
distributed |
Multiple workers, central scheduler | WebSocket to scheduler |
both |
Quick testing, single process | In-process scheduler |
Worker Mode Selection
Set worker.mode to switch between implementations:
worker:
mode: "standalone" # Uses Redis/SQLite queue.Backend
# OR
mode: "distributed" # Uses SchedulerBackend over WebSocket
The worker code is unchanged — only the backend implementation changes.