86f9ae5a7e
Restructure configuration files for better organization: - Add scheduler configuration examples (scheduler.yaml.example) - Reorganize worker configs into subdirectories: - distributed/ - Multi-node cluster configurations - standalone/ - Single-node deployment configs - Add environment-specific configs: - dev-local.yaml, docker-dev.yaml, docker-prod.yaml - homelab-secure.yaml, worker-prod.toml - Add deployment configs for different security modes: - docker-standard.yaml, docker-hipaa.yaml, docker-dev.yaml Add documentation: - configs/README.md with configuration guidelines - configs/SECURITY.md with security configuration best practices
60 lines
1.6 KiB
Markdown
60 lines
1.6 KiB
Markdown
# fetch_ml Configuration Guide
|
|
|
|
## Quick Start
|
|
|
|
### Standalone Mode (Existing Behavior)
|
|
```bash
|
|
# Single worker, direct queue access
|
|
go run ./cmd/worker -config configs/worker/standalone/worker.yaml
|
|
```
|
|
|
|
### Distributed Mode
|
|
```bash
|
|
# Terminal 1: Start scheduler
|
|
go run ./cmd/scheduler -config configs/scheduler/scheduler.yaml
|
|
|
|
# Terminal 2: Start worker
|
|
go run ./cmd/worker -config configs/worker/distributed/worker.yaml
|
|
```
|
|
|
|
### Single-Node Mode (Zero Config)
|
|
```bash
|
|
# Both scheduler and worker in one process
|
|
go run ./cmd/fetch_ml -config configs/multi-node/single-node.yaml
|
|
```
|
|
|
|
## Config Structure
|
|
|
|
```
|
|
configs/
|
|
├── scheduler/
|
|
│ └── scheduler.yaml # Central scheduler configuration
|
|
├── worker/
|
|
│ ├── standalone/
|
|
│ │ └── worker.yaml # Direct queue access (Redis/SQLite)
|
|
│ └── distributed/
|
|
│ └── worker.yaml # WebSocket to scheduler
|
|
└── multi-node/
|
|
└── single-node.yaml # Combined scheduler+worker
|
|
```
|
|
|
|
## Key Configuration Modes
|
|
|
|
| Mode | Use Case | Backend |
|
|
|------|----------|---------|
|
|
| `standalone` | Single machine, existing behavior | Redis/SQLite/Filesystem |
|
|
| `distributed` | Multiple workers, central scheduler | WebSocket to scheduler |
|
|
| `both` | Quick testing, single process | In-process scheduler |
|
|
|
|
## Worker Mode Selection
|
|
|
|
Set `worker.mode` to switch between implementations:
|
|
|
|
```yaml
|
|
worker:
|
|
mode: "standalone" # Uses Redis/SQLite queue.Backend
|
|
# OR
|
|
mode: "distributed" # Uses SchedulerBackend over WebSocket
|
|
```
|
|
|
|
The worker code is unchanged — only the backend implementation changes.
|