History

Jeremie Fraeys b3a0c78903 config: add Plugin GPU Quota, plugins, and audit logging to configs - Add Plugin GPU Quota config section to scheduler.yaml.example - Add audit logging config to homelab-secure.yaml (HIPAA-compliant) - Add Jupyter and vLLM plugin configs to all worker configs: - Security settings (passwords, trusted channels, blocked packages) - Resource limits (GPU, memory, CPU) - Model cache paths and quantization options for vLLM - Disable plugins in HIPAA deployment mode for compliance - Update deployments README with plugin services and GPU quotas		2026-02-26 14:34:42 -05:00
..
api	config: add Plugin GPU Quota, plugins, and audit logging to configs	2026-02-26 14:34:42 -05:00
examples	chore(ops): reorganize deployments/monitoring and remove legacy scripts	2026-01-05 12:31:26 -05:00
scheduler	config: add Plugin GPU Quota, plugins, and audit logging to configs	2026-02-26 14:34:42 -05:00
schema	chore(config): update configurations and deployment scripts	2026-02-12 12:05:37 -05:00
seccomp	feat(security): implement comprehensive security hardening phases 1-5,7	2026-02-23 18:00:33 -05:00
worker	config: add Plugin GPU Quota, plugins, and audit logging to configs	2026-02-26 14:34:42 -05:00
README.md	docs(config): reorganize configuration structure and add documentation	2026-02-26 12:04:11 -05:00
SECURITY.md	docs(config): reorganize configuration structure and add documentation	2026-02-26 12:04:11 -05:00

README.md

fetch_ml Configuration Guide

Quick Start

Standalone Mode (Existing Behavior)

# Single worker, direct queue access
go run ./cmd/worker -config configs/worker/standalone/worker.yaml

Distributed Mode

# Terminal 1: Start scheduler
go run ./cmd/scheduler -config configs/scheduler/scheduler.yaml

# Terminal 2: Start worker
go run ./cmd/worker -config configs/worker/distributed/worker.yaml

Single-Node Mode (Zero Config)

# Both scheduler and worker in one process
go run ./cmd/fetch_ml -config configs/multi-node/single-node.yaml

Config Structure

configs/
├── scheduler/
│   └── scheduler.yaml       # Central scheduler configuration
├── worker/
│   ├── standalone/
│   │   └── worker.yaml      # Direct queue access (Redis/SQLite)
│   └── distributed/
│       └── worker.yaml      # WebSocket to scheduler
└── multi-node/
    └── single-node.yaml     # Combined scheduler+worker

Key Configuration Modes

Mode	Use Case	Backend
`standalone`	Single machine, existing behavior	Redis/SQLite/Filesystem
`distributed`	Multiple workers, central scheduler	WebSocket to scheduler
`both`	Quick testing, single process	In-process scheduler

Worker Mode Selection

Set worker.mode to switch between implementations:

worker:
  mode: "standalone"    # Uses Redis/SQLite queue.Backend
  # OR
  mode: "distributed"   # Uses SchedulerBackend over WebSocket

The worker code is unchanged — only the backend implementation changes.