docs: add comprehensive documentation with MkDocs site

- Add complete API documentation and architecture guides
- Include quick start, installation, and deployment guides
- Add troubleshooting and security documentation
- Include CLI reference and configuration schema docs
- Add production monitoring and operations guides
- Implement MkDocs configuration with search functionality
- Include comprehensive user and developer documentation

Provides complete documentation for users and developers
covering all aspects of the FetchML platform.

docs/_config.yml

@@ -0,0 +1,88 @@
+# GitHub Pages configuration
+
+# Site settings
+title: "Fetch ML Documentation"
+description: "Secure Machine Learning Platform"
+baseurl: "/fetch_ml"
+url: "https://fetch-ml.github.io"
+
+# Build settings
+markdown: kramdown
+highlighter: rouge
+theme: minima
+plugins:
+  - jekyll-sitemap
+  - jekyll-feed
+  - jekyll-optional-front-matter
+  - jekyll-readme-index
+  - jekyll-titles-from-headings
+  - jekyll-seo-tag
+
+# Versioning
+version: "1.0.0"
+versions:
+  - "1.0.0"
+  - "0.9.0"
+latest_version: "1.0.0"
+
+# Navigation
+nav:
+  - title: "Getting Started"
+    subnav:
+      - title: "Quick Start"
+        url: "/quick-start/"
+  - title: "Guides"
+    subnav:
+      - title: "CLI Reference"
+        url: "/cli-reference/"
+      - title: "Architecture"
+        url: "/architecture/"
+      - title: "Server Setup"
+        url: "/server-setup/"
+  - title: "Development"
+    subnav:
+      - title: "Contributing"
+        url: "/contributing/"
+      - title: "API Reference"
+        url: "/api/"
+
+# Collections
+collections:
+  docs:
+    output: true
+    permalink: /:collection/:name/
+  api:
+    output: true
+    permalink: /api/:name/
+
+# Exclude files from processing
+exclude:
+  - Gemfile
+  - Gemfile.lock
+  - node_modules
+  - vendor
+  - .gitignore
+  - README.md
+  - Makefile
+
+# Include files
+include:
+  - _pages
+
+# SEO
+author: "Fetch ML Team"
+twitter:
+  username: "fetch_ml"
+  card: "summary"
+
+# Google Analytics (optional)
+google_analytics: ""
+
+# Mermaid diagrams for architecture
+mermaid:
+  enabled: true
+
+# Code highlighting
+kramdown:
+  input: GFM
+  syntax_highlighter: rouge

docs/_pages/architecture.md

@@ -0,0 +1,738 @@
+---
+layout: page
+title: "Homelab Architecture"
+permalink: /architecture/
+nav_order: 1
+---
+
+# Homelab Architecture
+
+Simple, secure architecture for ML experiments in your homelab.
+
+## Components Overview
+
+```mermaid
+graph TB
+    subgraph "Homelab Stack"
+        CLI[Zig CLI]
+        API[HTTPS API]
+        REDIS[Redis Cache]
+        FS[Local Storage]
+    end
+    
+    CLI --> API
+    API --> REDIS
+    API --> FS
+```
+
+## Core Services
+
+### API Server
+- **Purpose**: Secure HTTPS API for ML experiments
+- **Port**: 9101 (HTTPS only)
+- **Auth**: API key authentication
+- **Security**: Rate limiting, IP whitelisting
+
+### Redis
+- **Purpose**: Caching and job queuing
+- **Port**: 6379 (localhost only)
+- **Storage**: Temporary data only
+- **Persistence**: Local volume
+
+### Zig CLI
+- **Purpose**: High-performance experiment management
+- **Language**: Zig for maximum speed and efficiency
+- **Features**: 
+  - Content-addressed storage with deduplication
+  - SHA256-based commit ID generation
+  - WebSocket communication for real-time updates
+  - Rsync-based incremental file transfers
+  - Multi-threaded operations
+  - Secure API key authentication
+  - Auto-sync monitoring with file system watching
+  - Priority-based job queuing
+  - Memory-efficient operations with arena allocators
+
+## Security Architecture
+
+```mermaid
+graph LR
+    USER[User] --> AUTH[API Key Auth]
+    AUTH --> RATE[Rate Limiting]
+    RATE --> WHITELIST[IP Whitelist]
+    WHITELIST --> API[Secure API]
+    API --> AUDIT[Audit Logging]
+```
+
+### Security Layers
+1. **API Key Authentication** - Hashed keys with roles
+2. **Rate Limiting** - 30 requests/minute
+3. **IP Whitelisting** - Local networks only
+4. **Fail2Ban** - Automatic IP blocking
+5. **HTTPS/TLS** - Encrypted communication
+6. **Audit Logging** - Complete action tracking
+
+## Data Flow
+
+```mermaid
+sequenceDiagram
+    participant CLI
+    participant API
+    participant Redis
+    participant Storage
+    
+    CLI->>API: HTTPS Request
+    API->>API: Validate Auth
+    API->>Redis: Cache/Queue
+    API->>Storage: Experiment Data
+    Storage->>API: Results
+    API->>CLI: Response
+```
+
+## Deployment Options
+
+### Docker Compose (Recommended)
+```yaml
+services:
+  redis:
+    image: redis:7-alpine
+    ports: ["6379:6379"]
+    volumes: [redis_data:/data]
+    
+  api-server:
+    build: .
+    ports: ["9101:9101"]
+    depends_on: [redis]
+```
+
+### Local Setup
+```bash
+./setup.sh && ./manage.sh start
+```
+
+## Network Architecture
+
+- **Private Network**: Docker internal network
+- **Localhost Access**: Redis only on localhost
+- **HTTPS API**: Port 9101, TLS encrypted
+- **No External Dependencies**: Everything runs locally
+
+## Storage Architecture
+
+```
+data/
+├── experiments/     # ML experiment results
+├── cache/          # Temporary cache files
+└── backups/        # Local backups
+
+logs/
+├── app.log         # Application logs
+├── audit.log       # Security events
+└── access.log      # API access logs
+```
+
+## Monitoring Architecture
+
+Simple, lightweight monitoring:
+- **Health Checks**: Service availability
+- **Log Files**: Structured logging
+- **Basic Metrics**: Request counts, error rates
+- **Security Events**: Failed auth, rate limits
+
+## Homelab Benefits
+
+- ✅ **Simple Setup**: One-command installation
+- ✅ **Local Only**: No external dependencies
+- ✅ **Secure by Default**: HTTPS, auth, rate limiting
+- ✅ **Low Resource**: Minimal CPU/memory usage
+- ✅ **Easy Backup**: Local file system
+- ✅ **Privacy**: Everything stays on your network
+
+## High-Level Architecture
+
+```mermaid
+graph TB
+    subgraph "Client Layer"
+        CLI[CLI Tools]
+        TUI[Terminal UI]
+        API[REST API]
+    end
+    
+    subgraph "Authentication Layer"
+        Auth[Authentication Service]
+        RBAC[Role-Based Access Control]
+        Perm[Permission Manager]
+    end
+    
+    subgraph "Core Services"
+        Worker[ML Worker Service]
+        DataMgr[Data Manager Service]
+        Queue[Job Queue]
+    end
+    
+    subgraph "Storage Layer"
+        Redis[(Redis Cache)]
+        DB[(SQLite/PostgreSQL)]
+        Files[File Storage]
+    end
+    
+    subgraph "Container Runtime"
+        Podman[Podman/Docker]
+        Containers[ML Containers]
+    end
+    
+    CLI --> Auth
+    TUI --> Auth
+    API --> Auth
+    
+    Auth --> RBAC
+    RBAC --> Perm
+    
+    Worker --> Queue
+    Worker --> DataMgr
+    Worker --> Podman
+    
+    DataMgr --> DB
+    DataMgr --> Files
+    
+    Queue --> Redis
+    
+    Podman --> Containers
+```
+
+## Zig CLI Architecture
+
+### Component Structure
+
+```mermaid
+graph TB
+    subgraph "Zig CLI Components"
+        Main[main.zig] --> Commands[commands/]
+        Commands --> Config[config.zig]
+        Commands --> Utils[utils/]
+        Commands --> Net[net/]
+        Commands --> Errors[errors.zig]
+        
+        subgraph "Commands"
+            Init[init.zig]
+            Sync[sync.zig]
+            Queue[queue.zig]
+            Watch[watch.zig]
+            Status[status.zig]
+            Monitor[monitor.zig]
+            Cancel[cancel.zig]
+            Prune[prune.zig]
+        end
+        
+        subgraph "Utils"
+            Crypto[crypto.zig]
+            Storage[storage.zig]
+            Rsync[rsync.zig]
+        end
+        
+        subgraph "Network"
+            WS[ws.zig]
+        end
+    end
+```
+
+### Performance Optimizations
+
+#### Content-Addressed Storage
+- **Deduplication**: Files stored by SHA256 hash
+- **Space Efficiency**: Shared files across experiments
+- **Fast Lookup**: Hash-based file retrieval
+
+#### Memory Management
+- **Arena Allocators**: Efficient bulk allocation
+- **Zero-Copy Operations**: Minimized memory copying
+- **Automatic Cleanup**: Resource deallocation
+
+#### Network Communication
+- **WebSocket Protocol**: Real-time bidirectional communication
+- **Connection Pooling**: Reused connections
+- **Binary Messaging**: Efficient data transfer
+
+### Security Implementation
+
+```mermaid
+graph LR
+    subgraph "CLI Security"
+        Config[Config File] --> Hash[SHA256 Hashing]
+        Hash --> Auth[API Authentication]
+        Auth --> SSH[SSH Transfer]
+        SSH --> WS[WebSocket Security]
+    end
+```
+
+## Core Components
+
+### 1. Authentication & Authorization
+
+```mermaid
+graph LR
+    subgraph "Auth Flow"
+        Client[Client] --> APIKey[API Key]
+        APIKey --> Hash[Hash Validation]
+        Hash --> Roles[Role Resolution]
+        Roles --> Perms[Permission Check]
+        Perms --> Access[Grant/Deny Access]
+    end
+    
+    subgraph "Permission Sources"
+        YAML[YAML Config]
+        Inline[Inline Fallback]
+        Roles --> YAML
+        Roles --> Inline
+    end
+```
+
+**Features:**
+- API key-based authentication
+- Role-based access control (RBAC)
+- YAML-based permission configuration
+- Fallback to inline permissions
+- Admin wildcard permissions
+
+### 2. Worker Service
+
+```mermaid
+graph TB
+    subgraph "Worker Architecture"
+        API[HTTP API] --> Router[Request Router]
+        Router --> Auth[Auth Middleware]
+        Auth --> Queue[Job Queue]
+        Queue --> Processor[Job Processor]
+        Processor --> Runtime[Container Runtime]
+        Runtime --> Storage[Result Storage]
+        
+        subgraph "Job Lifecycle"
+            Submit[Submit Job] --> Queue
+            Queue --> Execute[Execute]
+            Execute --> Monitor[Monitor]
+            Monitor --> Complete[Complete]
+            Complete --> Store[Store Results]
+        end
+    end
+```
+
+**Responsibilities:**
+- HTTP API for job submission
+- Job queue management
+- Container orchestration
+- Result collection and storage
+- Metrics and monitoring
+
+### 3. Data Manager Service
+
+```mermaid
+graph TB
+    subgraph "Data Management"
+        API[Data API] --> Storage[Storage Layer]
+        Storage --> Metadata[Metadata DB]
+        Storage --> Files[File System]
+        Storage --> Cache[Redis Cache]
+        
+        subgraph "Data Operations"
+            Upload[Upload Data] --> Validate[Validate]
+            Validate --> Store[Store]
+            Store --> Index[Index]
+            Index --> Catalog[Catalog]
+        end
+    end
+```
+
+**Features:**
+- Data upload and validation
+- Metadata management
+- File system abstraction
+- Caching layer
+- Data catalog
+
+### 4. Terminal UI (TUI)
+
+```mermaid
+graph TB
+    subgraph "TUI Architecture"
+        UI[UI Components] --> Model[Data Model]
+        Model --> Update[Update Loop]
+        Update --> Render[Render]
+        
+        subgraph "UI Panels"
+            Jobs[Job List]
+            Details[Job Details]
+            Logs[Log Viewer]
+            Status[Status Bar]
+        end
+        
+        UI --> Jobs
+        UI --> Details
+        UI --> Logs
+        UI --> Status
+    end
+```
+
+**Components:**
+- Bubble Tea framework
+- Component-based architecture
+- Real-time updates
+- Keyboard navigation
+- Theme support
+
+## Data Flow
+
+### Job Execution Flow
+
+```mermaid
+sequenceDiagram
+    participant Client
+    participant Auth
+    participant Worker
+    participant Queue
+    participant Container
+    participant Storage
+    
+    Client->>Auth: Submit job with API key
+    Auth->>Client: Validate and return job ID
+    
+    Client->>Worker: Execute job request
+    Worker->>Queue: Queue job
+    Queue->>Worker: Job ready
+    Worker->>Container: Start ML container
+    Container->>Worker: Execute experiment
+    Worker->>Storage: Store results
+    Worker->>Client: Return results
+```
+
+### Authentication Flow
+
+```mermaid
+sequenceDiagram
+    participant Client
+    participant Auth
+    participant PermMgr
+    participant Config
+    
+    Client->>Auth: Request with API key
+    Auth->>Auth: Validate key hash
+    Auth->>PermMgr: Get user permissions
+    PermMgr->>Config: Load YAML permissions
+    Config->>PermMgr: Return permissions
+    PermMgr->>Auth: Return resolved permissions
+    Auth->>Client: Grant/deny access
+```
+
+## Security Architecture
+
+### Defense in Depth
+
+```mermaid
+graph TB
+    subgraph "Security Layers"
+        Network[Network Security]
+        Auth[Authentication]
+        AuthZ[Authorization]
+        Container[Container Security]
+        Data[Data Protection]
+        Audit[Audit Logging]
+    end
+    
+    Network --> Auth
+    Auth --> AuthZ
+    AuthZ --> Container
+    Container --> Data
+    Data --> Audit
+```
+
+**Security Features:**
+- API key authentication
+- Role-based permissions
+- Container isolation
+- File system sandboxing
+- Comprehensive audit logs
+- Input validation and sanitization
+
+### Container Security
+
+```mermaid
+graph TB
+    subgraph "Container Isolation"
+        Host[Host System]
+        Podman[Podman Runtime]
+        Network[Network Isolation]
+        FS[File System Isolation]
+        User[User Namespaces]
+        ML[ML Container]
+        
+        Host --> Podman
+        Podman --> Network
+        Podman --> FS
+        Podman --> User
+        User --> ML
+    end
+```
+
+**Isolation Features:**
+- Rootless containers
+- Network isolation
+- File system sandboxing
+- User namespace mapping
+- Resource limits
+
+## Configuration Architecture
+
+### Configuration Hierarchy
+
+```mermaid
+graph TB
+    subgraph "Config Sources"
+        Env[Environment Variables]
+        File[Config Files]
+        CLI[CLI Flags]
+        Defaults[Default Values]
+    end
+    
+    subgraph "Config Processing"
+        Merge[Config Merger]
+        Validate[Schema Validator]
+        Apply[Config Applier]
+    end
+    
+    Env --> Merge
+    File --> Merge
+    CLI --> Merge
+    Defaults --> Merge
+    
+    Merge --> Validate
+    Validate --> Apply
+```
+
+**Configuration Priority:**
+1. CLI flags (highest)
+2. Environment variables
+3. Configuration files
+4. Default values (lowest)
+
+## Scalability Architecture
+
+### Horizontal Scaling
+
+```mermaid
+graph TB
+    subgraph "Scaled Architecture"
+        LB[Load Balancer]
+        W1[Worker 1]
+        W2[Worker 2]
+        W3[Worker N]
+        Redis[Redis Cluster]
+        Storage[Shared Storage]
+        
+        LB --> W1
+        LB --> W2
+        LB --> W3
+        
+        W1 --> Redis
+        W2 --> Redis
+        W3 --> Redis
+        
+        W1 --> Storage
+        W2 --> Storage
+        W3 --> Storage
+    end
+```
+
+**Scaling Features:**
+- Stateless worker services
+- Shared job queue (Redis)
+- Distributed storage
+- Load balancer ready
+- Health checks and monitoring
+
+## Technology Stack
+
+### Backend Technologies
+
+| Component | Technology | Purpose |
+|-----------|------------|---------|
+| **Language** | Go 1.25+ | Core application |
+| **Web Framework** | Standard library | HTTP server |
+| **Authentication** | Custom | API key + RBAC |
+| **Database** | SQLite/PostgreSQL | Metadata storage |
+| **Cache** | Redis | Job queue & caching |
+| **Containers** | Podman/Docker | Job isolation |
+| **UI Framework** | Bubble Tea | Terminal UI |
+
+### Dependencies
+
+```go
+// Core dependencies
+require (
+    github.com/charmbracelet/bubbletea v1.3.10  // TUI framework
+    github.com/go-redis/redis/v8 v8.11.5        // Redis client
+    github.com/google/uuid v1.6.0               // UUID generation
+    github.com/mattn/go-sqlite3 v1.14.32        // SQLite driver
+    golang.org/x/crypto v0.45.0                 // Crypto utilities
+    gopkg.in/yaml.v3 v3.0.1                     // YAML parsing
+)
+```
+
+## Development Architecture
+
+### Project Structure
+
+```
+fetch_ml/
+├── cmd/                    # CLI applications
+│   ├── worker/            # ML worker service
+│   ├── tui/               # Terminal UI
+│   ├── data_manager/      # Data management
+│   └── user_manager/      # User management
+├── internal/              # Internal packages
+│   ├── auth/              # Authentication system
+│   ├── config/            # Configuration management
+│   ├── container/         # Container operations
+│   ├── database/          # Database operations
+│   ├── logging/           # Logging utilities
+│   ├── metrics/           # Metrics collection
+│   └── network/           # Network utilities
+├── configs/               # Configuration files
+├── scripts/               # Setup and utility scripts
+├── tests/                 # Test suites
+└── docs/                  # Documentation
+```
+
+### Package Dependencies
+
+```mermaid
+graph TB
+    subgraph "Application Layer"
+        Worker[cmd/worker]
+        TUI[cmd/tui]
+        DataMgr[cmd/data_manager]
+        UserMgr[cmd/user_manager]
+    end
+    
+    subgraph "Service Layer"
+        Auth[internal/auth]
+        Config[internal/config]
+        Container[internal/container]
+        Database[internal/database]
+    end
+    
+    subgraph "Utility Layer"
+        Logging[internal/logging]
+        Metrics[internal/metrics]
+        Network[internal/network]
+    end
+    
+    Worker --> Auth
+    Worker --> Config
+    Worker --> Container
+    TUI --> Auth
+    DataMgr --> Database
+    UserMgr --> Auth
+    
+    Auth --> Logging
+    Container --> Network
+    Database --> Metrics
+```
+
+## Monitoring & Observability
+
+### Metrics Collection
+
+```mermaid
+graph TB
+    subgraph "Metrics Pipeline"
+        App[Application] --> Metrics[Metrics Collector]
+        Metrics --> Export[Prometheus Exporter]
+        Export --> Prometheus[Prometheus Server]
+        Prometheus --> Grafana[Grafana Dashboard]
+        
+        subgraph "Metric Types"
+            Counter[Counters]
+            Gauge[Gauges]
+            Histogram[Histograms]
+            Timer[Timers]
+        end
+        
+        App --> Counter
+        App --> Gauge
+        App --> Histogram
+        App --> Timer
+    end
+```
+
+### Logging Architecture
+
+```mermaid
+graph TB
+    subgraph "Logging Pipeline"
+        App[Application] --> Logger[Structured Logger]
+        Logger --> File[File Output]
+        Logger --> Console[Console Output]
+        Logger --> Syslog[Syslog Forwarder]
+        Syslog --> Aggregator[Log Aggregator]
+        Aggregator --> Storage[Log Storage]
+        Storage --> Viewer[Log Viewer]
+    end
+```
+
+## Deployment Architecture
+
+### Container Deployment
+
+```mermaid
+graph TB
+    subgraph "Deployment Stack"
+        Image[Container Image]
+        Registry[Container Registry]
+        Orchestrator[Docker Compose]
+        Config[ConfigMaps/Secrets]
+        Storage[Persistent Storage]
+        
+        Image --> Registry
+        Registry --> Orchestrator
+        Config --> Orchestrator
+        Storage --> Orchestrator
+    end
+```
+
+### Service Discovery
+
+```mermaid
+graph TB
+    subgraph "Service Mesh"
+        Gateway[API Gateway]
+        Discovery[Service Discovery]
+        Worker[Worker Service]
+        Data[Data Service]
+        Redis[Redis Cluster]
+        
+        Gateway --> Discovery
+        Discovery --> Worker
+        Discovery --> Data
+        Discovery --> Redis
+    end
+```
+
+## Future Architecture Considerations
+
+### Microservices Evolution
+
+- **API Gateway**: Centralized routing and authentication
+- **Service Mesh**: Inter-service communication
+- **Event Streaming**: Kafka for job events
+- **Distributed Tracing**: OpenTelemetry integration
+- **Multi-tenant**: Tenant isolation and quotas
+
+### Homelab Features
+
+- **Docker Compose**: Simple container orchestration
+- **Local Development**: Easy setup and testing
+- **Security**: Built-in authentication and encryption
+- **Monitoring**: Basic health checks and logging
+
+---
+
+This architecture provides a solid foundation for secure, scalable machine learning experiments while maintaining simplicity and developer productivity.

docs/_pages/cicd.md

@@ -0,0 +1,165 @@
+---
+layout: page
+title: "CI/CD Pipeline"
+permalink: /cicd/
+nav_order: 5
+---
+
+# CI/CD Pipeline
+
+Automated testing, building, and releasing for fetch_ml.
+
+## Workflows
+
+### CI Workflow (`.github/workflows/ci.yml`)
+
+Runs on every push to `main`/`develop` and all pull requests.
+
+**Jobs:**
+1. **test** - Go backend tests with Redis
+2. **build** - Build all binaries (Go + Zig CLI)
+3. **test-scripts** - Validate deployment scripts
+4. **security-scan** - Trivy and Gosec security scans
+5. **docker-build** - Build and push Docker images (main branch only)
+
+**Test Coverage:**
+- Go unit tests with race detection
+- `internal/queue` package tests
+- Zig CLI tests  
+- Integration tests
+- Security audits
+
+### Release Workflow (`.github/workflows/release.yml`)
+
+Runs on version tags (e.g., `v1.0.0`).
+
+**Jobs:**
+
+1. **build-cli** (matrix build)
+   - Linux x86_64 (static musl)
+   - macOS x86_64
+   - macOS ARM64
+   - Downloads platform-specific static rsync
+   - Embeds rsync for zero-dependency releases
+
+2. **build-go-backends**
+   - Cross-platform Go builds
+   - api-server, worker, tui, data_manager, user_manager
+
+3. **create-release**
+   - Collects all artifacts
+   - Generates SHA256 checksums
+   - Creates GitHub release with notes
+
+## Release Process
+
+### Creating a Release
+
+```bash
+# 1. Update version
+git tag v1.0.0
+
+# 2. Push tag
+git push origin v1.0.0
+
+# 3. CI automatically builds and releases
+```
+
+### Release Artifacts
+
+**CLI Binaries (with embedded rsync):**
+- `ml-linux-x86_64.tar.gz` (~450-650KB)
+- `ml-macos-x86_64.tar.gz` (~450-650KB)
+- `ml-macos-arm64.tar.gz` (~450-650KB)
+
+**Go Backends:**
+- `fetch_ml_api-server.tar.gz`
+- `fetch_ml_worker.tar.gz`
+- `fetch_ml_tui.tar.gz`
+- `fetch_ml_data_manager.tar.gz`
+- `fetch_ml_user_manager.tar.gz`
+
+**Checksums:**
+- `checksums.txt` - Combined SHA256 sums
+- Individual `.sha256` files per binary
+
+## Development Workflow
+
+### Local Testing
+
+```bash
+# Run all tests
+make test
+
+# Run specific package tests
+go test ./internal/queue/...
+
+# Build CLI
+cd cli && zig build dev
+
+# Run formatters and linters
+make lint
+
+# Security scans are handled automatically in CI by the `security-scan` job
+```
+
+#### Optional heavy end-to-end tests
+
+Some e2e tests exercise full Docker deployments and performance scenarios and are
+**skipped by default** to keep local/CI runs fast. You can enable them explicitly
+with environment variables:
+
+```bash
+# Run Docker deployment e2e tests
+FETCH_ML_E2E_DOCKER=1 go test ./tests/e2e/...
+
+# Run performance-oriented e2e tests
+FETCH_ML_E2E_PERF=1 go test ./tests/e2e/...
+```
+
+Without these variables, `TestDockerDeploymentE2E` and `TestPerformanceE2E` will
+`t.Skip`, while all lighter e2e tests still run.
+
+### Pull Request Checks
+
+All PRs must pass:
+- ✅ Go tests (with Redis)
+- ✅ CLI tests
+- ✅ Security scans
+- ✅ Code linting
+- ✅ Build verification
+
+## Configuration
+
+### Environment Variables
+
+```yaml
+GO_VERSION: '1.25.0'
+ZIG_VERSION: '0.15.2'
+```
+
+### Secrets
+
+Required for releases:
+- `GITHUB_TOKEN` - Automatic, provided by GitHub Actions
+
+## Monitoring
+
+### Build Status
+
+Check workflow runs at:
+```
+https://github.com/jfraeys/fetch_ml/actions
+```
+
+### Artifacts
+
+Download build artifacts from:
+- Successful workflow runs (30-day retention)
+- GitHub Releases (permanent)
+
+---
+
+For implementation details:
+- [.github/workflows/ci.yml](https://github.com/jfraeys/fetch_ml/blob/main/.github/workflows/ci.yml)
+- [.github/workflows/release.yml](https://github.com/jfraeys/fetch_ml/blob/main/.github/workflows/release.yml)

docs/_pages/cli-reference.md

@@ -0,0 +1,404 @@
+---
+layout: page
+title: "CLI Reference"
+permalink: /cli-reference/
+nav_order: 2
+---
+
+# Fetch ML CLI Reference
+
+Comprehensive command-line tools for managing ML experiments in your homelab with Zig-based high-performance CLI.
+
+## Overview
+
+Fetch ML provides a comprehensive CLI toolkit built with performance and security in mind:
+
+- **Zig CLI** - High-performance experiment management written in Zig
+- **Go Commands** - API server, TUI, and data management utilities
+- **Management Scripts** - Service orchestration and deployment
+- **Setup Scripts** - One-command installation and configuration
+
+## Zig CLI (`./cli/zig-out/bin/ml`)
+
+High-performance command-line interface for experiment management, written in Zig for speed and efficiency.
+
+### Available Commands
+
+| Command | Description | Example |
+|---------|-------------|----------|
+| `init` | Interactive configuration setup | `ml init` |
+| `sync` | Sync project to worker with deduplication | `ml sync ./project --name myjob --queue` |
+| `queue` | Queue job for execution | `ml queue myjob --commit abc123 --priority 8` |
+| `status` | Get system and worker status | `ml status` |
+| `monitor` | Launch TUI monitoring via SSH | `ml monitor` |
+| `cancel` | Cancel running job | `ml cancel job123` |
+| `prune` | Clean up old experiments | `ml prune --keep 10` |
+| `watch` | Auto-sync directory on changes | `ml watch ./project --queue` |
+
+### Command Details
+
+#### `init` - Configuration Setup
+```bash
+ml init
+```
+Creates a configuration template at `~/.ml/config.toml` with:
+- Worker connection details
+- API authentication
+- Base paths and ports
+
+#### `sync` - Project Synchronization
+```bash
+# Basic sync
+ml sync ./my-project
+
+# Sync with custom name and queue
+ml sync ./my-project --name "experiment-1" --queue
+
+# Sync with priority
+ml sync ./my-project --priority 9
+```
+
+**Features:**
+- Content-addressed storage for deduplication
+- SHA256 commit ID generation
+- Rsync-based file transfer
+- Automatic queuing (with `--queue` flag)
+
+#### `queue` - Job Management
+```bash
+# Queue with commit ID
+ml queue my-job --commit abc123def456
+
+# Queue with priority (1-10, default 5)
+ml queue my-job --commit abc123 --priority 8
+```
+
+**Features:**
+- WebSocket-based communication
+- Priority queuing system
+- API key authentication
+
+#### `watch` - Auto-Sync Monitoring
+```bash
+# Watch directory for changes
+ml watch ./project
+
+# Watch and auto-queue on changes
+ml watch ./project --name "dev-exp" --queue
+```
+
+**Features:**
+- Real-time file system monitoring
+- Automatic re-sync on changes
+- Configurable polling interval (2 seconds)
+- Commit ID comparison for efficiency
+
+#### `prune` - Cleanup Management
+```bash
+# Keep last N experiments
+ml prune --keep 20
+
+# Remove experiments older than N days
+ml prune --older-than 30
+```
+
+#### `monitor` - Remote Monitoring
+```bash
+ml monitor
+```
+Launches TUI interface via SSH for real-time monitoring.
+
+#### `cancel` - Job Cancellation
+```bash
+ml cancel running-job-id
+```
+Cancels currently running jobs by ID.
+
+### Configuration
+
+The Zig CLI reads configuration from `~/.ml/config.toml`:
+
+```toml
+worker_host = "worker.local"
+worker_user = "mluser"
+worker_base = "/data/ml-experiments"
+worker_port = 22
+api_key = "your-api-key"
+```
+
+### Performance Features
+
+- **Content-Addressed Storage**: Automatic deduplication of identical files
+- **Incremental Sync**: Only transfers changed files
+- **SHA256 Hashing**: Reliable commit ID generation
+- **WebSocket Communication**: Efficient real-time messaging
+- **Multi-threaded**: Concurrent operations where applicable
+
+## Go Commands
+
+### API Server (`./cmd/api-server/main.go`)
+Main HTTPS API server for experiment management.
+
+```bash
+# Build and run
+go run ./cmd/api-server/main.go
+
+# With configuration
+./bin/api-server --config configs/config-local.yaml
+```
+
+**Features:**
+- HTTPS-only communication
+- API key authentication
+- Rate limiting and IP whitelisting
+- WebSocket support for real-time updates
+- Redis integration for caching
+
+### TUI (`./cmd/tui/main.go`)
+Terminal User Interface for monitoring experiments.
+
+```bash
+# Launch TUI
+go run ./cmd/tui/main.go
+
+# With custom config
+./tui --config configs/config-local.yaml
+```
+
+**Features:**
+- Real-time experiment monitoring
+- Interactive job management
+- Status visualization
+- Log viewing
+
+### Data Manager (`./cmd/data_manager/`)
+Utilities for data synchronization and management.
+
+```bash
+# Sync data
+./data_manager --sync ./data
+
+# Clean old data
+./data_manager --cleanup --older-than 30d
+```
+
+### Config Lint (`./cmd/configlint/main.go`)
+Configuration validation and linting tool.
+
+```bash
+# Validate configuration
+./configlint configs/config-local.yaml
+
+# Check schema compliance
+./configlint --schema configs/schema/config_schema.yaml
+```
+
+## Management Script (`./tools/manage.sh`)
+
+Simple service management for your homelab.
+
+### Commands
+```bash
+./tools/manage.sh start          # Start all services
+./tools/manage.sh stop           # Stop all services
+./tools/manage.sh status         # Check service status
+./tools/manage.sh logs           # View logs
+./tools/manage.sh monitor        # Basic monitoring
+./tools/manage.sh security       # Security status
+./tools/manage.sh cleanup        # Clean project artifacts
+```
+
+## Setup Script (`./setup.sh`)
+
+One-command homelab setup.
+
+### Usage
+```bash
+# Full setup
+./setup.sh
+
+# Setup includes:
+# - SSL certificate generation
+# - Configuration creation
+# - Build all components
+# - Start Redis
+# - Setup Fail2Ban (if available)
+```
+
+## API Testing
+
+Test the API with curl:
+
+```bash
+# Health check
+curl -k -H 'X-API-Key: password' https://localhost:9101/health
+
+# List experiments
+curl -k -H 'X-API-Key: password' https://localhost:9101/experiments
+
+# Submit experiment
+curl -k -X POST -H 'X-API-Key: password' \
+     -H 'Content-Type: application/json' \
+     -d '{"name":"test","config":{"type":"basic"}}' \
+     https://localhost:9101/experiments
+```
+
+## Zig CLI Architecture
+
+The Zig CLI is designed for performance and reliability:
+
+### Core Components
+- **Commands** (`cli/src/commands/`): Individual command implementations
+- **Config** (`cli/src/config.zig`): Configuration management
+- **Network** (`cli/src/net/ws.zig`): WebSocket client implementation
+- **Utils** (`cli/src/utils/`): Cryptography, storage, and rsync utilities
+- **Errors** (`cli/src/errors.zig`): Centralized error handling
+
+### Performance Optimizations
+- **Content-Addressed Storage**: Deduplicates identical files across experiments
+- **SHA256 Hashing**: Fast, reliable commit ID generation
+- **Rsync Integration**: Efficient incremental file transfers
+- **WebSocket Protocol**: Low-latency communication with worker
+- **Memory Management**: Efficient allocation with Zig's allocator system
+
+### Security Features
+- **API Key Hashing**: Secure authentication token handling
+- **SSH Integration**: Secure file transfers
+- **Input Validation**: Comprehensive argument checking
+- **Error Handling**: Secure error reporting without information leakage
+
+## Configuration
+
+Main configuration file: `configs/config-local.yaml`
+
+### Key Settings
+```yaml
+auth:
+  enabled: true
+  api_keys:
+    homelab_user:
+      hash: "5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8"
+      admin: true
+
+server:
+  address: ":9101"
+  tls:
+    enabled: true
+    cert_file: "./ssl/cert.pem"
+    key_file: "./ssl/key.pem"
+
+security:
+  rate_limit:
+    enabled: true
+    requests_per_minute: 30
+  ip_whitelist:
+    - "127.0.0.1"
+    - "::1"
+    - "192.168.0.0/16"
+    - "10.0.0.0/8"
+```
+
+## Docker Commands
+
+If using Docker Compose:
+
+```bash
+# Start services
+docker-compose up -d
+
+# View logs
+docker-compose logs -f
+
+# Stop services
+docker-compose down
+
+# Check status
+docker-compose ps
+```
+
+## Troubleshooting
+
+### Common Issues
+
+**Zig CLI not found:**
+```bash
+# Build the CLI
+cd cli && make build
+
+# Check binary exists
+ls -la ./cli/zig-out/bin/ml
+```
+
+**Configuration not found:**
+```bash
+# Create configuration
+./cli/zig-out/bin/ml init
+
+# Check config file
+ls -la ~/.ml/config.toml
+```
+
+**Worker connection failed:**
+```bash
+# Test SSH connection
+ssh -p 22 mluser@worker.local
+
+# Check configuration
+cat ~/.ml/config.toml
+```
+
+**Sync not working:**
+```bash
+# Check rsync availability
+rsync --version
+
+# Test manual sync
+rsync -avz ./project/ mluser@worker.local:/tmp/test/
+```
+
+**WebSocket connection failed:**
+```bash
+# Check worker WebSocket port
+telnet worker.local 9100
+
+# Verify API key
+./cli/zig-out/bin/ml status
+```
+
+**API not responding:**
+```bash
+./tools/manage.sh status
+./tools/manage.sh logs
+```
+
+**Authentication failed:**
+```bash
+# Check API key in config-local.yaml
+grep -A 5 "api_keys:" configs/config-local.yaml
+```
+
+**Redis connection failed:**
+```bash
+# Check Redis status
+redis-cli ping
+
+# Start Redis
+redis-server
+```
+
+### Getting Help
+
+```bash
+# CLI help
+./cli/zig-out/bin/ml help
+
+# Management script help
+./tools/manage.sh help
+
+# Check all available commands
+make help
+```
+
+---
+
+**That's it for the CLI reference!** For complete setup instructions, see the main [README](/).

docs/_pages/operations.md

@@ -0,0 +1,310 @@
+---
+layout: page
+title: "Operations Runbook"
+permalink: /operations/
+nav_order: 6
+---
+
+# Operations Runbook
+
+Operational guide for troubleshooting and maintaining the ML experiment system.
+
+## Task Queue Operations
+
+### Monitoring Queue Health
+
+```redis
+# Check queue depth
+ZCARD task:queue
+
+# List pending tasks
+ZRANGE task:queue 0 -1 WITHSCORES
+
+# Check dead letter queue
+KEYS task:dlq:*
+```
+
+### Handling Stuck Tasks
+
+**Symptom:** Tasks stuck in "running" status
+
+**Diagnosis:**
+```bash
+# Check for expired leases
+redis-cli GET task:{task-id}
+# Look for LeaseExpiry in past
+```
+
+**Rem
+
+ediation:**
+Tasks with expired leases are automatically reclaimed every 1 minute. To force immediate reclamation:
+```bash
+# Restart worker to trigger reclaim cycle
+systemctl restart ml-worker
+```
+
+### Dead Letter Queue Management
+
+**View failed tasks:**
+```redis
+KEYS task:dlq:*
+```
+
+**Inspect failed task:**
+```redis
+GET task:dlq:{task-id}
+```
+
+**Retry from DLQ:**
+```bash
+# Manual retry (requires custom script)
+# 1. Get task from DLQ
+# 2. Reset retry count
+# 3. Re-queue task
+```
+
+### Worker Crashes
+
+**Symptom:** Worker disappeared mid-task
+
+**What Happens:**
+1. Lease expires after 30 minutes (default)
+2. Background reclaim job detects expired lease
+3. Task is retried (up to 3 attempts)
+4. After max retries → Dead Letter Queue
+
+**Prevention:**
+- Monitor worker heartbeats
+- Set up alerts for worker down
+- Use process manager (systemd, supervisor)
+
+## Worker Operations
+
+### Graceful Shutdown
+
+```bash
+# Send SIGTERM for graceful shutdown
+kill -TERM $(pgrep ml-worker)
+
+# Worker will:
+# 1. Stop accepting new tasks
+# 2. Finish active tasks (up to 5min timeout)
+# 3. Release all leases
+# 4. Exit cleanly
+```
+
+### Force Shutdown
+
+```bash
+# Force kill (leases will be reclaimed automatically)
+kill -9 $(pgrep ml-worker)
+```
+
+### Worker Heartbeat Monitoring
+
+```redis
+# Check worker heartbeats
+HGETALL worker:heartbeat
+
+# Example output:
+# worker-abc123 1701234567
+# worker-def456 1701234580
+```
+
+**Alert if:** Heartbeat timestamp > 5 minutes old
+
+## Redis Operations
+
+### Backup
+
+```bash
+# Manual backup
+redis-cli SAVE
+cp /var/lib/redis/dump.rdb /backup/redis-$(date +%Y%m%d).rdb
+```
+
+### Restore
+
+```bash
+# Stop Redis
+systemctl stop redis
+
+#  Restore snapshot
+cp /backup/redis-20231201.rdb /var/lib/redis/dump.rdb
+
+# Start Redis
+systemctl start redis
+```
+
+### Memory Management
+
+```redis
+# Check memory usage
+INFO memory
+
+# Evict old data if needed
+FLUSHDB  # DANGER: Clears all data!
+```
+
+## Common Issues
+
+### Issue: Queue Growing Unbounded
+
+**Symptoms:**
+- `ZCARD task:queue` keeps increasing
+- No workers processing tasks
+
+**Diagnosis:**
+```bash
+# Check worker status
+systemctl status ml-worker
+
+# Check logs
+journalctl -u ml-worker -n 100
+```
+
+**Resolution:**
+1. Verify workers are running
+2. Check Redis connectivity
+3. Verify lease configuration
+
+### Issue: High Retry Rate
+
+**Symptoms:**
+- Many tasks in DLQ
+- `retry_count` field high on tasks
+
+**Diagnosis:**
+```bash
+# Check worker logs for errors
+journalctl -u ml-worker | grep "retry"
+
+# Look for patterns (network issues, resource limits, etc)
+```
+
+**Resolution:**
+- Fix underlying issue (network, resources, etc)
+- Adjust retry limits if permanent failures
+- Increase task timeout if jobs are slow
+
+### Issue: Leases Expiring Prematurely
+
+**Symptoms:**
+- Tasks retried even though worker is healthy
+- Logs show "lease expired" frequently
+
+**Diagnosis:**
+```yaml
+# Check worker config
+cat configs/worker-config.yaml | grep -A3 "lease"
+
+task_lease_duration: 30m  # Too short?
+heartbeat_interval: 1m    # Too infrequent?
+```
+
+**Resolution:**
+```yaml
+# Increase lease duration for long-running jobs
+task_lease_duration: 60m
+heartbeat_interval: 30s  # More frequent heartbeats
+```
+
+## Performance Tuning
+
+### Worker Concurrency
+
+```yaml
+# worker-config.yaml
+max_workers: 4  # Number of parallel tasks
+
+# Adjust based on:
+# - CPU cores available
+# - Memory per task
+# - GPU availability
+```
+
+### Redis Configuration
+
+```conf
+# /etc/redis/redis.conf
+
+# Persistence
+save 900 1
+save 300 10
+
+# Memory
+maxmemory 2gb
+maxmemory-policy noeviction
+
+# Performance
+tcp-keepalive 300
+timeout 0
+```
+
+## Alerting Rules
+
+### Critical Alerts
+
+1. **Worker Down** (no heartbeat > 5min)
+2. **Queue Depth** > 1000 tasks
+3. **DLQ Growth** > 100 tasks/hour
+4. **Redis Down** (connection failed)
+
+### Warning Alerts
+
+1. **High Retry Rate** > 10% of tasks
+2. **Slow Queue Drain** (depth increasing over 1 hour)
+3. **Worker Memory** > 80% usage
+
+## Health Checks
+
+```bash
+#!/bin/bash
+# health-check.sh
+
+# Check Redis
+redis-cli PING || echo "Redis DOWN"
+
+# Check worker heartbeat
+WORKER_ID=$(cat /var/run/ml-worker.pid)
+LAST_HB=$(redis-cli HGET worker:heartbeat "$WORKER_ID")
+NOW=$(date +%s)
+if [ $((NOW - LAST_HB)) -gt 300 ]; then
+  echo "Worker heartbeat stale"
+fi
+
+# Check queue depth
+DEPTH=$(redis-cli ZCARD task:queue)
+if [ "$DEPTH" -gt 1000 ]; then
+  echo "Queue depth critical: $DEPTH"
+fi
+```
+
+## Runbook Checklist
+
+### Daily Operations
+- [ ] Check queue depth
+- [ ] Verify worker heartbeats
+- [ ] Review DLQ for patterns
+- [ ] Check Redis memory usage
+
+### Weekly Operations
+- [ ] Review retry rates
+- [ ] Analyze failed task patterns
+- [ ] Backup Redis snapshot
+- [ ] Review worker logs
+
+### Monthly Operations
+- [ ] Performance tuning review
+- [ ] Capacity planning
+- [ ] Update documentation
+- [ ] Test disaster recovery
+
+---
+
+**For homelab setups:**
+Most of these operations can be simplified. Focus on:
+-  Basic monitoring (queue depth, worker status)
+- Periodic Redis backups
+- Graceful shutdowns for maintenance

docs/_pages/queue.md

@@ -0,0 +1,322 @@
+---
+layout: page
+title: "Task Queue Architecture"
+permalink: /queue/
+nav_order: 3
+---
+
+# Task Queue Architecture
+
+The task queue system enables reliable job processing between the API server and workers using Redis.
+
+## Overview
+
+```mermaid
+graph LR
+    CLI[CLI/Client] -->|WebSocket| API[API Server]
+    API -->|Enqueue| Redis[(Redis)]
+    Redis -->|Dequeue| Worker[Worker]
+    Worker -->|Update Status| Redis
+```
+
+## Components
+
+### TaskQueue (`internal/queue`)
+
+Shared package used by both API server and worker for job management.
+
+#### Task Structure
+
+```go
+type Task struct {
+    ID        string            // Unique task ID (UUID)
+    JobName   string            // User-defined job name  
+    Args      string            // Job arguments
+    Status    string            // queued, running, completed, failed
+    Priority  int64             // Higher = executed first
+    CreatedAt time.Time         
+    StartedAt *time.Time        
+    EndedAt   *time.Time        
+    WorkerID  string            
+    Error     string            
+    Datasets  []string          
+    Metadata  map[string]string // commit_id, user, etc
+}
+```
+
+#### TaskQueue Interface
+
+```go
+// Initialize queue
+queue, err := queue.NewTaskQueue(queue.Config{
+    RedisAddr:     "localhost:6379",
+    RedisPassword: "",
+    RedisDB:       0,
+})
+
+// Add task (API server)
+task := &queue.Task{
+    ID:       uuid.New().String(),
+    JobName:  "train-model",
+    Status:   "queued",
+    Priority: 5,
+    Metadata: map[string]string{
+        "commit_id": commitID,
+        "user":      username,
+    },
+}
+err = queue.AddTask(task)
+
+// Get next task (Worker)
+task, err := queue.GetNextTask()
+
+// Update task status
+task.Status = "running"
+err = queue.UpdateTask(task)
+```
+
+## Data Flow
+
+### Job Submission Flow
+
+```mermaid
+sequenceDiagram
+    participant CLI
+    participant API
+    participant Redis
+    participant Worker
+    
+    CLI->>API: Queue Job (WebSocket)
+    API->>API: Create Task (UUID)
+    API->>Redis: ZADD task:queue
+    API->>Redis: SET task:{id}
+    API->>CLI: Success Response
+    
+    Worker->>Redis: ZPOPMAX task:queue
+    Redis->>Worker: Task ID
+    Worker->>Redis: GET task:{id}
+    Redis->>Worker: Task Data
+    Worker->>Worker: Execute Job
+    Worker->>Redis: Update Status
+```
+
+### Protocol
+
+**CLI → API** (Binary WebSocket):
+```
+[opcode:1][api_key_hash:64][commit_id:64][priority:1][job_name_len:1][job_name:var]
+```
+
+**API → Redis**:
+- Priority queue: `ZADD task:queue {priority} {task_id}`
+- Task data: `SET task:{id} {json}`
+- Status: `HSET task:status:{job_name} ...`
+
+**Worker ← Redis**:
+- Poll: `ZPOPMAX task:queue 1` (highest priority first)
+- Fetch: `GET task:{id}`
+
+## Redis Data Structures
+
+### Keys
+
+```
+task:queue                    # ZSET: priority queue
+task:{uuid}                  # STRING: task JSON data
+task:status:{job_name}       # HASH: job status
+worker:heartbeat             # HASH: worker health
+job:metrics:{job_name}       # HASH: job metrics
+```
+
+### Priority Queue (ZSET)
+
+```redis
+ZADD task:queue 10 "uuid-1"   # Priority 10
+ZADD task:queue 5  "uuid-2"   # Priority 5
+ZPOPMAX task:queue 1          # Returns uuid-1 (highest)
+```
+
+## API Server Integration
+
+### Initialization
+
+```go
+// cmd/api-server/main.go
+queueCfg := queue.Config{
+    RedisAddr:     cfg.Redis.Addr,
+    RedisPassword: cfg.Redis.Password,
+    RedisDB:       cfg.Redis.DB,
+}
+taskQueue, err := queue.NewTaskQueue(queueCfg)
+```
+
+### WebSocket Handler
+
+```go
+// internal/api/ws.go
+func (h *WSHandler) handleQueueJob(conn *websocket.Conn, payload []byte) error {
+    // Parse request
+    apiKeyHash, commitID, priority, jobName := parsePayload(payload)
+    
+    // Create task with unique ID
+    taskID := uuid.New().String()
+    task := &queue.Task{
+        ID:       taskID,
+        JobName:  jobName,
+        Status:   "queued",
+        Priority: int64(priority),
+        Metadata: map[string]string{
+            "commit_id": commitID,
+            "user":      user,
+        },
+    }
+    
+    // Enqueue
+    if err := h.queue.AddTask(task); err != nil {
+        return h.sendErrorPacket(conn, ErrorCodeDatabaseError, ...)
+    }
+    
+    return h.sendSuccessPacket(conn, "Job queued")
+}
+```
+
+## Worker Integration
+
+### Task Polling
+
+```go
+// cmd/worker/worker_server.go
+func (w *Worker) Start() error {
+    for {
+        task, err := w.queue.WaitForNextTask(ctx, 5*time.Second)
+        if task != nil {
+            go w.executeTask(task)
+        }
+    }
+}
+```
+
+### Task Execution
+
+```go
+func (w *Worker) executeTask(task *queue.Task) {
+    // Update status
+    task.Status = "running"
+    task.StartedAt = &now
+    w.queue.UpdateTaskWithMetrics(task, "start")
+    
+    // Execute
+    err := w.runJob(task)
+    
+    // Finalize
+    task.Status = "completed" // or "failed"
+    task.EndedAt = &endTime
+    task.Error = err.Error() // if err != nil
+    w.queue.UpdateTaskWithMetrics(task, "final")
+}
+```
+
+## Configuration
+
+### API Server (`configs/config.yaml`)
+
+```yaml
+redis:
+  addr: "localhost:6379"
+  password: ""
+  db: 0
+```
+
+### Worker (`configs/worker-config.yaml`)
+
+```yaml
+redis:
+  addr: "localhost:6379"
+  password: ""
+  db: 0
+  
+metrics_flush_interval: 500ms
+```
+
+## Monitoring
+
+### Queue Depth
+
+```go
+depth, err := queue.QueueDepth()
+fmt.Printf("Pending tasks: %d\n", depth)
+```
+
+### Worker Heartbeat
+
+```go
+// Worker sends heartbeat every 30s
+err := queue.Heartbeat(workerID)
+```
+
+### Metrics
+
+```redis
+HGETALL job:metrics:{job_name}
+# Returns: timestamp, tasks_start, tasks_final, etc
+```
+
+## Error Handling
+
+### Task Failures
+
+```go
+if err := w.runJob(task); err != nil {
+    task.Status = "failed"
+    task.Error = err.Error()
+    w.queue.UpdateTask(task)
+}
+```
+
+### Redis Connection Loss
+
+```go
+// TaskQueue automatically reconnects
+// Workers should implement retry logic
+for retries := 0; retries < 3; retries++ {
+    task, err := queue.GetNextTask()
+    if err == nil {
+        break
+    }
+    time.Sleep(backoff)
+}
+```
+
+## Testing
+
+```go
+// tests using miniredis
+s, _ := miniredis.Run()
+defer s.Close()
+
+tq, _ := queue.NewTaskQueue(queue.Config{
+    RedisAddr: s.Addr(),
+})
+
+task := &queue.Task{ID: "test-1", JobName: "test"}
+tq.AddTask(task)
+
+fetched, _ := tq.GetNextTask()
+// assert fetched.ID == "test-1"
+```
+
+## Best Practices
+
+1. **Unique Task IDs**: Always use UUIDs to avoid conflicts
+2. **Metadata**: Store commit_id and user in task metadata
+3. **Priority**: Higher values execute first (0-255 range)
+4. **Status Updates**: Update status at each lifecycle stage
+5. **Error Logging**: Store detailed errors in task.Error
+6. **Heartbeats**: Workers should send heartbeats regularly
+7. **Metrics**: Use UpdateTaskWithMetrics for atomic updates
+
+---
+
+For implementation details, see:
+- [internal/queue/task.go](https://github.com/jfraeys/fetch_ml/blob/main/internal/queue/task.go)
+- [internal/queue/queue.go](https://github.com/jfraeys/fetch_ml/blob/main/internal/queue/queue.go)

docs/_pages/redis-ha.md

@@ -0,0 +1,95 @@
+---
+layout: page
+title: "Redis High Availability (Optional)"
+permalink: /redis-ha/
+nav_order: 7
+---
+
+# Redis High Availability
+
+**Note:** This is optional for homelab setups. Single Redis instance is sufficient for most use cases.
+
+## When You Need HA
+
+Consider Redis HA if:
+- Running production workloads
+- Uptime > 99.9% required
+- Can't afford to lose queued tasks
+- Multiple workers across machines
+
+## Redis Sentinel (Recommended)
+
+### Setup
+
+```yaml
+# docker-compose.yml
+version: '3.8'
+services:
+  redis-master:
+    image: redis:7-alpine
+    command: redis-server --maxmemory 2gb
+    
+  redis-replica:
+    image: redis:7-alpine
+    command: redis-server --slaveof redis-master 6379
+    
+  redis-sentinel-1:
+    image: redis:7-alpine
+    command: redis-sentinel /etc/redis/sentinel.conf
+    volumes:
+      - ./sentinel.conf:/etc/redis/sentinel.conf
+```
+
+**sentinel.conf:**
+```conf
+sentinel monitor mymaster redis-master 6379 2
+sentinel down-after-milliseconds mymaster 5000
+sentinel parallel-syncs mymaster 1
+sentinel failover-timeout mymaster 10000
+```
+
+### Application Configuration
+
+```yaml
+# worker-config.yaml
+redis_addr: "redis-sentinel-1:26379,redis-sentinel-2:26379"
+redis_master_name: "mymaster"
+```
+
+## Redis Cluster (Advanced)
+
+For larger deployments with sharding needs.
+
+```yaml
+# Minimum 3 masters + 3 replicas
+services:
+  redis-1:
+    image: redis:7-alpine
+    command: redis-server --cluster-enabled yes
+    
+  redis-2:
+    # ... similar config
+```
+
+## Homelab Alternative: Persistence Only
+
+**For most homelabs, just enable persistence:**
+
+```yaml
+# docker-compose.yml
+services:
+  redis:
+    image: redis:7-alpine
+    command: redis-server --appendonly yes
+    volumes:
+      - redis_data:/data
+      
+volumes:
+  redis_data:
+```
+
+This ensures tasks survive Redis restarts without full HA complexity.
+
+---
+
+**Recommendation:** Start simple. Add HA only if you experience actual downtime issues.

docs/_pages/zig-cli.md

@@ -0,0 +1,452 @@
+---
+layout: page
+title: "Zig CLI Guide"
+permalink: /zig-cli/
+nav_order: 3
+---
+
+# Zig CLI Guide
+
+High-performance command-line interface for ML experiment management, written in Zig for maximum speed and efficiency.
+
+## Overview
+
+The Zig CLI (`ml`) is the primary interface for managing ML experiments in your homelab. Built with Zig, it provides exceptional performance for file operations, network communication, and experiment management.
+
+## Installation
+
+### Pre-built Binaries (Recommended)
+
+Download from [GitHub Releases](https://github.com/jfraeys/fetch_ml/releases):
+
+```bash
+# Download for your platform
+curl -LO https://github.com/jfraeys/fetch_ml/releases/latest/download/ml-<platform>.tar.gz
+
+# Extract
+tar -xzf ml-<platform>.tar.gz
+
+# Install
+chmod +x ml-<platform>
+sudo mv ml-<platform> /usr/local/bin/ml
+
+# Verify
+ml --help
+```
+
+**Platforms:**
+- `ml-linux-x86_64.tar.gz` - Linux (fully static, zero dependencies)
+- `ml-macos-x86_64.tar.gz` - macOS Intel  
+- `ml-macos-arm64.tar.gz` - macOS Apple Silicon
+
+All release binaries include **embedded static rsync** for complete independence.
+
+### Build from Source
+
+**Development Build** (uses system rsync):
+```bash
+cd cli
+zig build dev
+./zig-out/dev/ml-dev --help
+```
+
+**Production Build** (embedded rsync):
+```bash
+cd cli
+# For testing: uses rsync wrapper
+zig build prod
+
+# For release with static rsync:
+# 1. Place static rsync binary at src/assets/rsync_release.bin
+# 2. Build
+zig build prod
+strip zig-out/prod/ml  # Optional: reduce size
+
+# Verify
+./zig-out/prod/ml --help
+ls -lh zig-out/prod/ml
+```
+
+See [cli/src/assets/README.md](https://github.com/jfraeys/fetch_ml/blob/main/cli/src/assets/README.md) for details on obtaining static rsync binaries.
+
+### Verify Installation
+```bash
+ml --help
+ml --version  # Shows build config
+```
+
+## Quick Start
+
+1. **Initialize Configuration**
+   ```bash
+   ./cli/zig-out/bin/ml init
+   ```
+
+2. **Sync Your First Project**
+   ```bash
+   ./cli/zig-out/bin/ml sync ./my-project --queue
+   ```
+
+3. **Monitor Progress**
+   ```bash
+   ./cli/zig-out/bin/ml status
+   ```
+
+## Command Reference
+
+### `init` - Configuration Setup
+
+Initialize the CLI configuration file.
+
+```bash
+ml init
+```
+
+**Creates:** `~/.ml/config.toml`
+
+**Configuration Template:**
+```toml
+worker_host = "worker.local"
+worker_user = "mluser"
+worker_base = "/data/ml-experiments"
+worker_port = 22
+api_key = "your-api-key"
+```
+
+### `sync` - Project Synchronization
+
+Sync project files to the worker with intelligent deduplication.
+
+```bash
+# Basic sync
+ml sync ./project
+
+# Sync with custom name and auto-queue
+ml sync ./project --name "experiment-1" --queue
+
+# Sync with priority
+ml sync ./project --priority 8
+```
+
+**Options:**
+- `--name <name>`: Custom experiment name
+- `--queue`: Automatically queue after sync
+- `--priority N`: Set priority (1-10, default 5)
+
+**Features:**
+- **Content-Addressed Storage**: Automatic deduplication
+- **SHA256 Commit IDs**: Reliable change detection
+- **Incremental Transfer**: Only sync changed files
+- **Rsync Backend**: Efficient file transfer
+
+### `queue` - Job Management
+
+Queue experiments for execution on the worker.
+
+```bash
+# Queue with commit ID
+ml queue my-job --commit abc123def456
+
+# Queue with priority
+ml queue my-job --commit abc123 --priority 8
+```
+
+**Options:**
+- `--commit <id>`: Commit ID from sync output
+- `--priority N`: Execution priority (1-10)
+
+**Features:**
+- **WebSocket Communication**: Real-time job submission
+- **Priority Queuing**: Higher priority jobs run first
+- **API Authentication**: Secure job submission
+
+### `watch` - Auto-Sync Monitoring
+
+Monitor directories for changes and auto-sync.
+
+```bash
+# Watch for changes
+ml watch ./project
+
+# Watch and auto-queue on changes
+ml watch ./project --name "dev-exp" --queue
+```
+
+**Options:**
+- `--name <name>`: Custom experiment name
+- `--queue`: Auto-queue on changes
+- `--priority N`: Set priority for queued jobs
+
+**Features:**
+- **Real-time Monitoring**: 2-second polling interval
+- **Change Detection**: File modification time tracking
+- **Commit Comparison**: Only sync when content changes
+- **Automatic Queuing**: Seamless development workflow
+
+### `status` - System Status
+
+Check system and worker status.
+
+```bash
+ml status
+```
+
+**Displays:**
+- Worker connectivity
+- Queue status
+- Running jobs
+- System health
+
+### `monitor` - Remote Monitoring
+
+Launch TUI interface via SSH for real-time monitoring.
+
+```bash
+ml monitor
+```
+
+**Features:**
+- **Real-time Updates**: Live experiment status
+- **Interactive Interface**: Browse and manage experiments
+- **SSH Integration**: Secure remote access
+
+### `cancel` - Job Cancellation
+
+Cancel running or queued jobs.
+
+```bash
+ml cancel job-id
+```
+
+**Options:**
+- `job-id`: Job identifier from status output
+
+### `prune` - Cleanup Management
+
+Clean up old experiments to save space.
+
+```bash
+# Keep last N experiments
+ml prune --keep 20
+
+# Remove experiments older than N days
+ml prune --older-than 30
+```
+
+**Options:**
+- `--keep N`: Keep N most recent experiments
+- `--older-than N`: Remove experiments older than N days
+
+## Architecture
+
+### Core Components
+
+```
+cli/src/
+├── commands/        # Command implementations
+│   ├── init.zig     # Configuration setup
+│   ├── sync.zig     # Project synchronization
+│   ├── queue.zig    # Job management
+│   ├── watch.zig    # Auto-sync monitoring
+│   ├── status.zig   # System status
+│   ├── monitor.zig  # Remote monitoring
+│   ├── cancel.zig   # Job cancellation
+│   └── prune.zig    # Cleanup operations
+├── config.zig       # Configuration management
+├── errors.zig       # Error handling
+├── net/            # Network utilities
+│   └── ws.zig       # WebSocket client
+└── utils/          # Utility functions
+    ├── crypto.zig   # Hashing and encryption
+    ├── storage.zig  # Content-addressed storage
+    └── rsync.zig    # File synchronization
+```
+
+### Performance Features
+
+#### Content-Addressed Storage
+- **Deduplication**: Identical files shared across experiments
+- **Hash-based Storage**: Files stored by SHA256 hash
+- **Space Efficiency**: Reduces storage by up to 90%
+
+#### SHA256 Commit IDs
+- **Reliable Detection**: Cryptographic change detection
+- **Collision Resistance**: Guaranteed unique identifiers
+- **Fast Computation**: Optimized for large directories
+
+#### WebSocket Protocol
+- **Low Latency**: Real-time communication
+- **Binary Protocol**: Efficient message format
+- **Connection Pooling**: Reused connections
+
+#### Memory Management
+- **Arena Allocators**: Efficient memory allocation
+- **Zero-copy Operations**: Minimized memory usage
+- **Resource Cleanup**: Automatic resource management
+
+### Security Features
+
+#### Authentication
+- **API Key Hashing**: Secure token storage
+- **SHA256 Hashes**: Irreversible token protection
+- **Config Validation**: Input sanitization
+
+#### Secure Communication
+- **SSH Integration**: Encrypted file transfers
+- **WebSocket Security**: TLS-protected communication
+- **Input Validation**: Comprehensive argument checking
+
+#### Error Handling
+- **Secure Reporting**: No sensitive information leakage
+- **Graceful Degradation**: Safe error recovery
+- **Audit Logging**: Operation tracking
+
+## Advanced Usage
+
+### Workflow Integration
+
+#### Development Workflow
+```bash
+# 1. Initialize project
+ml sync ./project --name "dev" --queue
+
+# 2. Auto-sync during development
+ml watch ./project --name "dev" --queue
+
+# 3. Monitor progress
+ml status
+```
+
+#### Batch Processing
+```bash
+# Process multiple experiments
+for dir in experiments/*/; do
+    ml sync "$dir" --queue
+done
+```
+
+#### Priority Management
+```bash
+# High priority experiment
+ml sync ./urgent --priority 10 --queue
+
+# Background processing
+ml sync ./background --priority 1 --queue
+```
+
+### Configuration Management
+
+#### Multiple Workers
+```toml
+# ~/.ml/config.toml
+worker_host = "worker.local"
+worker_user = "mluser"
+worker_base = "/data/ml-experiments"
+worker_port = 22
+api_key = "your-api-key"
+```
+
+#### Security Settings
+```bash
+# Set restrictive permissions
+chmod 600 ~/.ml/config.toml
+
+# Verify configuration
+ml status
+```
+
+## Troubleshooting
+
+### Common Issues
+
+#### Build Problems
+```bash
+# Check Zig installation
+zig version
+
+# Clean build
+cd cli && make clean && make build
+```
+
+#### Connection Issues
+```bash
+# Test SSH connectivity
+ssh -p $worker_port $worker_user@$worker_host
+
+# Verify configuration
+cat ~/.ml/config.toml
+```
+
+#### Sync Failures
+```bash
+# Check rsync
+rsync --version
+
+# Manual sync test
+rsync -avz ./test/ $worker_user@$worker_host:/tmp/
+```
+
+#### Performance Issues
+```bash
+# Monitor resource usage
+top -p $(pgrep ml)
+
+# Check disk space
+df -h $worker_base
+```
+
+### Debug Mode
+
+Enable verbose logging:
+```bash
+# Environment variable
+export ML_DEBUG=1
+ml sync ./project
+
+# Or use debug build
+cd cli && make debug
+```
+
+## Performance Benchmarks
+
+### File Operations
+- **Sync Speed**: 100MB/s+ (network limited)
+- **Hash Computation**: 500MB/s+ (CPU limited)
+- **Deduplication**: 90%+ space savings
+
+### Memory Usage
+- **Base Memory**: ~10MB
+- **Large Projects**: ~50MB (1GB+ projects)
+- **Memory Efficiency**: Constant per-file overhead
+
+### Network Performance
+- **WebSocket Latency**: <10ms (local network)
+- **Connection Setup**: <100ms
+- **Throughput**: Network limited
+
+## Contributing
+
+### Development Setup
+```bash
+cd cli
+zig build-exe src/main.zig
+```
+
+### Testing
+```bash
+# Run tests
+cd cli && zig test src/
+
+# Integration tests
+zig test tests/
+```
+
+### Code Style
+- Follow Zig style guidelines
+- Use explicit error handling
+- Document public APIs
+- Add comprehensive tests
+
+---
+
+**For more information, see the [CLI Reference](/cli-reference/) and [Architecture](/architecture/) pages.**

docs/_site/assets/javascripts/lunr/min/lunr.da.min.js

@@ -0,0 +1,18 @@
+/*!
+ * Lunr languages, `Danish` language
+ * https://github.com/MihaiValentin/lunr-languages
+ *
+ * Copyright 2014, Mihai Valentin
+ * http://www.mozilla.org/MPL/
+ */
+/*!
+ * based on
+ * Snowball JavaScript Library v0.3
+ * http://code.google.com/p/urim/
+ * http://snowball.tartarus.org/
+ *
+ * Copyright 2010, Oleg Mazko
+ * http://www.mozilla.org/MPL/
+ */
+
+!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.da=function(){this.pipeline.reset(),this.pipeline.add(e.da.trimmer,e.da.stopWordFilter,e.da.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.da.stemmer))},e.da.wordCharacters="A-Za-zªºÀ-ÖØ-öø-ʸˠ-ˤᴀ-ᴥᴬ-ᵜᵢ-ᵥᵫ-ᵷᵹ-ᶾḀ-ỿⁱⁿₐ-ₜKÅℲⅎⅠ-ↈⱠ-ⱿꜢ-ꞇꞋ-ꞭꞰ-ꞷꟷ-ꟿꬰ-ꭚꭜ-ꭤff-stＡ-Ｚａ-ｚ",e.da.trimmer=e.trimmerSupport.generateTrimmer(e.da.wordCharacters),e.Pipeline.registerFunction(e.da.trimmer,"trimmer-da"),e.da.stemmer=function(){var r=e.stemmerSupport.Among,i=e.stemmerSupport.SnowballProgram,n=new function(){function e(){var e,r=f.cursor+3;if(d=f.limit,0<=r&&r<=f.limit){for(a=r;;){if(e=f.cursor,f.in_grouping(w,97,248)){f.cursor=e;break}if(f.cursor=e,e>=f.limit)return;f.cursor++}for(;!f.out_grouping(w,97,248);){if(f.cursor>=f.limit)return;f.cursor++}d=f.cursor,d<a&&(d=a)}}function n(){var e,r;if(f.cursor>=d&&(r=f.limit_backward,f.limit_backward=d,f.ket=f.cursor,e=f.find_among_b(c,32),f.limit_backward=r,e))switch(f.bra=f.cursor,e){case 1:f.slice_del();break;case 2:f.in_grouping_b(p,97,229)&&f.slice_del()}}function t(){var e,r=f.limit-f.cursor;f.cursor>=d&&(e=f.limit_backward,f.limit_backward=d,f.ket=f.cursor,f.find_among_b(l,4)?(f.bra=f.cursor,f.limit_backward=e,f.cursor=f.limit-r,f.cursor>f.limit_backward&&(f.cursor--,f.bra=f.cursor,f.slice_del())):f.limit_backward=e)}function s(){var e,r,i,n=f.limit-f.cursor;if(f.ket=f.cursor,f.eq_s_b(2,"st")&&(f.bra=f.cursor,f.eq_s_b(2,"ig")&&f.slice_del()),f.cursor=f.limit-n,f.cursor>=d&&(r=f.limit_backward,f.limit_backward=d,f.ket=f.cursor,e=f.find_among_b(m,5),f.limit_backward=r,e))switch(f.bra=f.cursor,e){case 1:f.slice_del(),i=f.limit-f.cursor,t(),f.cursor=f.limit-i;break;case 2:f.slice_from("løs")}}function o(){var e;f.cursor>=d&&(e=f.limit_backward,f.limit_backward=d,f.ket=f.cursor,f.out_grouping_b(w,97,248)?(f.bra=f.cursor,u=f.slice_to(u),f.limit_backward=e,f.eq_v_b(u)&&f.slice_del()):f.limit_backward=e)}var a,d,u,c=[new r("hed",-1,1),new r("ethed",0,1),new r("ered",-1,1),new r("e",-1,1),new r("erede",3,1),new r("ende",3,1),new r("erende",5,1),new r("ene",3,1),new r("erne",3,1),new r("ere",3,1),new r("en",-1,1),new r("heden",10,1),new r("eren",10,1),new r("er",-1,1),new r("heder",13,1),new r("erer",13,1),new r("s",-1,2),new r("heds",16,1),new r("es",16,1),new r("endes",18,1),new r("erendes",19,1),new r("enes",18,1),new r("ernes",18,1),new r("eres",18,1),new r("ens",16,1),new r("hedens",24,1),new r("erens",24,1),new r("ers",16,1),new r("ets",16,1),new r("erets",28,1),new r("et",-1,1),new r("eret",30,1)],l=[new r("gd",-1,-1),new r("dt",-1,-1),new r("gt",-1,-1),new r("kt",-1,-1)],m=[new r("ig",-1,1),new r("lig",0,1),new r("elig",1,1),new r("els",-1,1),new r("løst",-1,2)],w=[17,65,16,1,0,0,0,0,0,0,0,0,0,0,0,0,48,0,128],p=[239,254,42,3,0,0,0,0,0,0,0,0,0,0,0,0,16],f=new i;this.setCurrent=function(e){f.setCurrent(e)},this.getCurrent=function(){return f.getCurrent()},this.stem=function(){var r=f.cursor;return e(),f.limit_backward=r,f.cursor=f.limit,n(),f.cursor=f.limit,t(),f.cursor=f.limit,s(),f.cursor=f.limit,o(),!0}};return function(e){return"function"==typeof e.update?e.update(function(e){return n.setCurrent(e),n.stem(),n.getCurrent()}):(n.setCurrent(e),n.stem(),n.getCurrent())}}(),e.Pipeline.registerFunction(e.da.stemmer,"stemmer-da"),e.da.stopWordFilter=e.generateStopWordFilter("ad af alle alt anden at blev blive bliver da de dem den denne der deres det dette dig din disse dog du efter eller en end er et for fra ham han hans har havde have hende hendes her hos hun hvad hvis hvor i ikke ind jeg jer jo kunne man mange med meget men mig min mine mit mod ned noget nogle nu når og også om op os over på selv sig sin sine sit skal skulle som sådan thi til ud under var vi vil ville vor være været".split(" ")),e.Pipeline.registerFunction(e.da.stopWordFilter,"stopWordFilter-da")}});

docs/_site/assets/javascripts/lunr/min/lunr.hi.min.js

@@ -0,0 +1 @@
+!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.hi=function(){this.pipeline.reset(),this.pipeline.add(e.hi.trimmer,e.hi.stopWordFilter,e.hi.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.hi.stemmer))},e.hi.wordCharacters="ऀ-ःऄ-एऐ-टठ-यर-िी-ॏॐ-य़ॠ-९॰-ॿa-zA-Zａ-ｚＡ-Ｚ0-9０-９",e.hi.trimmer=e.trimmerSupport.generateTrimmer(e.hi.wordCharacters),e.Pipeline.registerFunction(e.hi.trimmer,"trimmer-hi"),e.hi.stopWordFilter=e.generateStopWordFilter("अत अपना अपनी अपने अभी अंदर आदि आप इत्यादि इन इनका इन्हीं इन्हें इन्हों इस इसका इसकी इसके इसमें इसी इसे उन उनका उनकी उनके उनको उन्हीं उन्हें उन्हों उस उसके उसी उसे एक एवं एस ऐसे और कई कर करता करते करना करने करें कहते कहा का काफ़ी कि कितना किन्हें किन्हों किया किर किस किसी किसे की कुछ कुल के को कोई कौन कौनसा गया घर जब जहाँ जा जितना जिन जिन्हें जिन्हों जिस जिसे जीधर जैसा जैसे जो तक तब तरह तिन तिन्हें तिन्हों तिस तिसे तो था थी थे दबारा दिया दुसरा दूसरे दो द्वारा न नके नहीं ना निहायत नीचे ने पर पहले पूरा पे फिर बनी बही बहुत बाद बाला बिलकुल भी भीतर मगर मानो मे में यदि यह यहाँ यही या यिह ये रखें रहा रहे ऱ्वासा लिए लिये लेकिन व वग़ैरह वर्ग वह वहाँ वहीं वाले वुह वे वो सकता सकते सबसे सभी साथ साबुत साभ सारा से सो संग ही हुआ हुई हुए है हैं हो होता होती होते होना होने".split(" ")),e.hi.stemmer=function(){return function(e){return"function"==typeof e.update?e.update(function(e){return e}):e}}();var r=e.wordcut;r.init(),e.hi.tokenizer=function(i){if(!arguments.length||null==i||void 0==i)return[];if(Array.isArray(i))return i.map(function(r){return isLunr2?new e.Token(r.toLowerCase()):r.toLowerCase()});var t=i.toString().toLowerCase().replace(/^\s+/,"");return r.cut(t).split("|")},e.Pipeline.registerFunction(e.hi.stemmer,"stemmer-hi"),e.Pipeline.registerFunction(e.hi.stopWordFilter,"stopWordFilter-hi")}});

docs/_site/assets/javascripts/lunr/min/lunr.hy.min.js

@@ -0,0 +1 @@
+!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.hy=function(){this.pipeline.reset(),this.pipeline.add(e.hy.trimmer,e.hy.stopWordFilter)},e.hy.wordCharacters="[A-Za-z԰-֏ff-ﭏ]",e.hy.trimmer=e.trimmerSupport.generateTrimmer(e.hy.wordCharacters),e.Pipeline.registerFunction(e.hy.trimmer,"trimmer-hy"),e.hy.stopWordFilter=e.generateStopWordFilter("դու և եք էիր էիք հետո նաև նրանք որը վրա է որ պիտի են այս մեջ ն իր ու ի այդ որոնք այն կամ էր մի ես համար այլ իսկ էին ենք հետ ին թ էինք մենք նրա նա դուք եմ էի ըստ որպես ում".split(" ")),e.Pipeline.registerFunction(e.hy.stopWordFilter,"stopWordFilter-hy"),e.hy.stemmer=function(){return function(e){return"function"==typeof e.update?e.update(function(e){return e}):e}}(),e.Pipeline.registerFunction(e.hy.stemmer,"stemmer-hy")}});

docs/_site/assets/javascripts/lunr/min/lunr.ja.min.js

@@ -0,0 +1 @@
+!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");var r="2"==e.version[0];e.ja=function(){this.pipeline.reset(),this.pipeline.add(e.ja.trimmer,e.ja.stopWordFilter,e.ja.stemmer),r?this.tokenizer=e.ja.tokenizer:(e.tokenizer&&(e.tokenizer=e.ja.tokenizer),this.tokenizerFn&&(this.tokenizerFn=e.ja.tokenizer))};var t=new e.TinySegmenter;e.ja.tokenizer=function(i){var n,o,s,p,a,u,m,l,c,f;if(!arguments.length||null==i||void 0==i)return[];if(Array.isArray(i))return i.map(function(t){return r?new e.Token(t.toLowerCase()):t.toLowerCase()});for(o=i.toString().toLowerCase().replace(/^\s+/,""),n=o.length-1;n>=0;n--)if(/\S/.test(o.charAt(n))){o=o.substring(0,n+1);break}for(a=[],s=o.length,c=0,l=0;c<=s;c++)if(u=o.charAt(c),m=c-l,u.match(/\s/)||c==s){if(m>0)for(p=t.segment(o.slice(l,c)).filter(function(e){return!!e}),f=l,n=0;n<p.length;n++)r?a.push(new e.Token(p[n],{position:[f,p[n].length],index:a.length})):a.push(p[n]),f+=p[n].length;l=c+1}return a},e.ja.stemmer=function(){return function(e){return e}}(),e.Pipeline.registerFunction(e.ja.stemmer,"stemmer-ja"),e.ja.wordCharacters="一二三四五六七八九十百千万億兆一-龠々〆ヵヶぁ-んァ-ヴーｱ-ﾝﾞa-zA-Zａ-ｚＡ-Ｚ0-9０-９",e.ja.trimmer=e.trimmerSupport.generateTrimmer(e.ja.wordCharacters),e.Pipeline.registerFunction(e.ja.trimmer,"trimmer-ja"),e.ja.stopWordFilter=e.generateStopWordFilter("これ それ あれ この その あの ここ そこ あそこ こちら どこ だれ なに なん 何 私 貴方 貴方方 我々 私達 あの人 あのかた 彼女 彼 です あります おります います は が の に を で え から まで より も どの と し それで しかし".split(" ")),e.Pipeline.registerFunction(e.ja.stopWordFilter,"stopWordFilter-ja"),e.jp=e.ja,e.Pipeline.registerFunction(e.jp.stemmer,"stemmer-jp"),e.Pipeline.registerFunction(e.jp.trimmer,"trimmer-jp"),e.Pipeline.registerFunction(e.jp.stopWordFilter,"stopWordFilter-jp")}});

docs/_site/assets/javascripts/lunr/min/lunr.jp.min.js

@@ -0,0 +1 @@
+module.exports=require("./lunr.ja");

docs/_site/assets/javascripts/lunr/min/lunr.kn.min.js

@@ -0,0 +1 @@
+!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.kn=function(){this.pipeline.reset(),this.pipeline.add(e.kn.trimmer,e.kn.stopWordFilter,e.kn.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.kn.stemmer))},e.kn.wordCharacters="ಀ-಄ಅ-ಔಕ-ಹಾ-ೌ಼-ಽೕ-ೖೝ-ೞೠ-ೡೢ-ೣ೤೥೦-೯ೱ-ೳ",e.kn.trimmer=e.trimmerSupport.generateTrimmer(e.kn.wordCharacters),e.Pipeline.registerFunction(e.kn.trimmer,"trimmer-kn"),e.kn.stopWordFilter=e.generateStopWordFilter("ಮತ್ತು ಈ ಒಂದು ರಲ್ಲಿ ಹಾಗೂ ಎಂದು ಅಥವಾ ಇದು ರ ಅವರು ಎಂಬ ಮೇಲೆ ಅವರ ತನ್ನ ಆದರೆ ತಮ್ಮ ನಂತರ ಮೂಲಕ ಹೆಚ್ಚು ನ ಆ ಕೆಲವು ಅನೇಕ ಎರಡು ಹಾಗು ಪ್ರಮುಖ ಇದನ್ನು ಇದರ ಸುಮಾರು ಅದರ ಅದು ಮೊದಲ ಬಗ್ಗೆ ನಲ್ಲಿ ರಂದು ಇತರ ಅತ್ಯಂತ ಹೆಚ್ಚಿನ ಸಹ ಸಾಮಾನ್ಯವಾಗಿ ನೇ ಹಲವಾರು ಹೊಸ ದಿ ಕಡಿಮೆ ಯಾವುದೇ ಹೊಂದಿದೆ ದೊಡ್ಡ ಅನ್ನು ಇವರು ಪ್ರಕಾರ ಇದೆ ಮಾತ್ರ ಕೂಡ ಇಲ್ಲಿ ಎಲ್ಲಾ ವಿವಿಧ ಅದನ್ನು ಹಲವು ರಿಂದ ಕೇವಲ ದ ದಕ್ಷಿಣ ಗೆ ಅವನ ಅತಿ ನೆಯ ಬಹಳ ಕೆಲಸ ಎಲ್ಲ ಪ್ರತಿ ಇತ್ಯಾದಿ ಇವು ಬೇರೆ ಹೀಗೆ ನಡುವೆ ಇದಕ್ಕೆ ಎಸ್ ಇವರ ಮೊದಲು ಶ್ರೀ ಮಾಡುವ ಇದರಲ್ಲಿ ರೀತಿಯ ಮಾಡಿದ ಕಾಲ ಅಲ್ಲಿ ಮಾಡಲು ಅದೇ ಈಗ ಅವು ಗಳು ಎ ಎಂಬುದು ಅವನು ಅಂದರೆ ಅವರಿಗೆ ಇರುವ ವಿಶೇಷ ಮುಂದೆ ಅವುಗಳ ಮುಂತಾದ ಮೂಲ ಬಿ ಮೀ ಒಂದೇ ಇನ್ನೂ ಹೆಚ್ಚಾಗಿ ಮಾಡಿ ಅವರನ್ನು ಇದೇ ಯ ರೀತಿಯಲ್ಲಿ ಜೊತೆ ಅದರಲ್ಲಿ ಮಾಡಿದರು ನಡೆದ ಆಗ ಮತ್ತೆ ಪೂರ್ವ ಆತ ಬಂದ ಯಾವ ಒಟ್ಟು ಇತರೆ ಹಿಂದೆ ಪ್ರಮಾಣದ ಗಳನ್ನು ಕುರಿತು ಯು ಆದ್ದರಿಂದ ಅಲ್ಲದೆ ನಗರದ ಮೇಲಿನ ಏಕೆಂದರೆ ರಷ್ಟು ಎಂಬುದನ್ನು ಬಾರಿ ಎಂದರೆ ಹಿಂದಿನ ಆದರೂ ಆದ ಸಂಬಂಧಿಸಿದ ಮತ್ತೊಂದು ಸಿ ಆತನ ".split(" ")),e.kn.stemmer=function(){return function(e){return"function"==typeof e.update?e.update(function(e){return e}):e}}();var r=e.wordcut;r.init(),e.kn.tokenizer=function(t){if(!arguments.length||null==t||void 0==t)return[];if(Array.isArray(t))return t.map(function(r){return isLunr2?new e.Token(r.toLowerCase()):r.toLowerCase()});var n=t.toString().toLowerCase().replace(/^\s+/,"");return r.cut(n).split("|")},e.Pipeline.registerFunction(e.kn.stemmer,"stemmer-kn"),e.Pipeline.registerFunction(e.kn.stopWordFilter,"stopWordFilter-kn")}});

docs/_site/assets/javascripts/lunr/min/lunr.multi.min.js

@@ -0,0 +1 @@
+!function(e,t){"function"==typeof define&&define.amd?define(t):"object"==typeof exports?module.exports=t():t()(e.lunr)}(this,function(){return function(e){e.multiLanguage=function(){for(var t=Array.prototype.slice.call(arguments),i=t.join("-"),r="",n=[],s=[],p=0;p<t.length;++p)"en"==t[p]?(r+="\\w",n.unshift(e.stopWordFilter),n.push(e.stemmer),s.push(e.stemmer)):(r+=e[t[p]].wordCharacters,e[t[p]].stopWordFilter&&n.unshift(e[t[p]].stopWordFilter),e[t[p]].stemmer&&(n.push(e[t[p]].stemmer),s.push(e[t[p]].stemmer)));var o=e.trimmerSupport.generateTrimmer(r);return e.Pipeline.registerFunction(o,"lunr-multi-trimmer-"+i),n.unshift(o),function(){this.pipeline.reset(),this.pipeline.add.apply(this.pipeline,n),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add.apply(this.searchPipeline,s))}}}});

docs/_site/assets/javascripts/lunr/min/lunr.no.min.js

@@ -0,0 +1,18 @@
+/*!
+ * Lunr languages, `Norwegian` language
+ * https://github.com/MihaiValentin/lunr-languages
+ *
+ * Copyright 2014, Mihai Valentin
+ * http://www.mozilla.org/MPL/
+ */
+/*!
+ * based on
+ * Snowball JavaScript Library v0.3
+ * http://code.google.com/p/urim/
+ * http://snowball.tartarus.org/
+ *
+ * Copyright 2010, Oleg Mazko
+ * http://www.mozilla.org/MPL/
+ */
+
+!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.no=function(){this.pipeline.reset(),this.pipeline.add(e.no.trimmer,e.no.stopWordFilter,e.no.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.no.stemmer))},e.no.wordCharacters="A-Za-zªºÀ-ÖØ-öø-ʸˠ-ˤᴀ-ᴥᴬ-ᵜᵢ-ᵥᵫ-ᵷᵹ-ᶾḀ-ỿⁱⁿₐ-ₜKÅℲⅎⅠ-ↈⱠ-ⱿꜢ-ꞇꞋ-ꞭꞰ-ꞷꟷ-ꟿꬰ-ꭚꭜ-ꭤff-stＡ-Ｚａ-ｚ",e.no.trimmer=e.trimmerSupport.generateTrimmer(e.no.wordCharacters),e.Pipeline.registerFunction(e.no.trimmer,"trimmer-no"),e.no.stemmer=function(){var r=e.stemmerSupport.Among,n=e.stemmerSupport.SnowballProgram,i=new function(){function e(){var e,r=w.cursor+3;if(a=w.limit,0<=r||r<=w.limit){for(s=r;;){if(e=w.cursor,w.in_grouping(d,97,248)){w.cursor=e;break}if(e>=w.limit)return;w.cursor=e+1}for(;!w.out_grouping(d,97,248);){if(w.cursor>=w.limit)return;w.cursor++}a=w.cursor,a<s&&(a=s)}}function i(){var e,r,n;if(w.cursor>=a&&(r=w.limit_backward,w.limit_backward=a,w.ket=w.cursor,e=w.find_among_b(m,29),w.limit_backward=r,e))switch(w.bra=w.cursor,e){case 1:w.slice_del();break;case 2:n=w.limit-w.cursor,w.in_grouping_b(c,98,122)?w.slice_del():(w.cursor=w.limit-n,w.eq_s_b(1,"k")&&w.out_grouping_b(d,97,248)&&w.slice_del());break;case 3:w.slice_from("er")}}function t(){var e,r=w.limit-w.cursor;w.cursor>=a&&(e=w.limit_backward,w.limit_backward=a,w.ket=w.cursor,w.find_among_b(u,2)?(w.bra=w.cursor,w.limit_backward=e,w.cursor=w.limit-r,w.cursor>w.limit_backward&&(w.cursor--,w.bra=w.cursor,w.slice_del())):w.limit_backward=e)}function o(){var e,r;w.cursor>=a&&(r=w.limit_backward,w.limit_backward=a,w.ket=w.cursor,e=w.find_among_b(l,11),e?(w.bra=w.cursor,w.limit_backward=r,1==e&&w.slice_del()):w.limit_backward=r)}var s,a,m=[new r("a",-1,1),new r("e",-1,1),new r("ede",1,1),new r("ande",1,1),new r("ende",1,1),new r("ane",1,1),new r("ene",1,1),new r("hetene",6,1),new r("erte",1,3),new r("en",-1,1),new r("heten",9,1),new r("ar",-1,1),new r("er",-1,1),new r("heter",12,1),new r("s",-1,2),new r("as",14,1),new r("es",14,1),new r("edes",16,1),new r("endes",16,1),new r("enes",16,1),new r("hetenes",19,1),new r("ens",14,1),new r("hetens",21,1),new r("ers",14,1),new r("ets",14,1),new r("et",-1,1),new r("het",25,1),new r("ert",-1,3),new r("ast",-1,1)],u=[new r("dt",-1,-1),new r("vt",-1,-1)],l=[new r("leg",-1,1),new r("eleg",0,1),new r("ig",-1,1),new r("eig",2,1),new r("lig",2,1),new r("elig",4,1),new r("els",-1,1),new r("lov",-1,1),new r("elov",7,1),new r("slov",7,1),new r("hetslov",9,1)],d=[17,65,16,1,0,0,0,0,0,0,0,0,0,0,0,0,48,0,128],c=[119,125,149,1],w=new n;this.setCurrent=function(e){w.setCurrent(e)},this.getCurrent=function(){return w.getCurrent()},this.stem=function(){var r=w.cursor;return e(),w.limit_backward=r,w.cursor=w.limit,i(),w.cursor=w.limit,t(),w.cursor=w.limit,o(),!0}};return function(e){return"function"==typeof e.update?e.update(function(e){return i.setCurrent(e),i.stem(),i.getCurrent()}):(i.setCurrent(e),i.stem(),i.getCurrent())}}(),e.Pipeline.registerFunction(e.no.stemmer,"stemmer-no"),e.no.stopWordFilter=e.generateStopWordFilter("alle at av bare begge ble blei bli blir blitt både båe da de deg dei deim deira deires dem den denne der dere deres det dette di din disse ditt du dykk dykkar då eg ein eit eitt eller elles en enn er et ett etter for fordi fra før ha hadde han hans har hennar henne hennes her hjå ho hoe honom hoss hossen hun hva hvem hver hvilke hvilken hvis hvor hvordan hvorfor i ikke ikkje ikkje ingen ingi inkje inn inni ja jeg kan kom korleis korso kun kunne kva kvar kvarhelst kven kvi kvifor man mange me med medan meg meget mellom men mi min mine mitt mot mykje ned no noe noen noka noko nokon nokor nokre nå når og også om opp oss over på samme seg selv si si sia sidan siden sin sine sitt sjøl skal skulle slik so som som somme somt så sånn til um upp ut uten var vart varte ved vere verte vi vil ville vore vors vort vår være være vært å".split(" ")),e.Pipeline.registerFunction(e.no.stopWordFilter,"stopWordFilter-no")}});

docs/_site/assets/javascripts/lunr/min/lunr.sa.min.js

@@ -0,0 +1 @@
+!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.sa=function(){this.pipeline.reset(),this.pipeline.add(e.sa.trimmer,e.sa.stopWordFilter,e.sa.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.sa.stemmer))},e.sa.wordCharacters="ऀ-ःऄ-एऐ-टठ-यर-िी-ॏॐ-य़ॠ-९॰-ॿ꣠-꣱ꣲ-ꣷ꣸-ꣻ꣼-ꣽꣾ-ꣿᆰ0-ᆰ9",e.sa.trimmer=e.trimmerSupport.generateTrimmer(e.sa.wordCharacters),e.Pipeline.registerFunction(e.sa.trimmer,"trimmer-sa"),e.sa.stopWordFilter=e.generateStopWordFilter('तथा अयम्‌ एकम्‌ इत्यस्मिन्‌ तथा तत्‌ वा अयम्‌ इत्यस्य ते आहूत उपरि तेषाम्‌  किन्तु तेषाम्‌ तदा इत्यनेन अधिकः इत्यस्य तत्‌ केचन बहवः द्वि तथा महत्वपूर्णः अयम्‌ अस्य  विषये अयं अस्ति तत्‌ प्रथमः विषये इत्युपरि इत्युपरि इतर अधिकतमः अधिकः अपि सामान्यतया ठ इतरेतर नूतनम्‌ द  न्यूनम्‌ कश्चित्‌ वा विशालः द  सः अस्ति तदनुसारम् तत्र अस्ति केवलम्‌ अपि अत्र सर्वे विविधाः तत्‌ बहवः यतः इदानीम्‌ द  दक्षिण इत्यस्मै तस्य उपरि नथ अतीव कार्यम्‌ सर्वे एकैकम्‌ इत्यादि। एते सन्ति  उत इत्थम्‌ मध्ये एतदर्थं . स कस्य प्रथमः श्री. करोति अस्मिन् प्रकारः निर्मिता कालः तत्र कर्तुं  समान अधुना ते सन्ति स एकः अस्ति सः अर्थात् तेषां कृते . स्थितम्  विशेषः अग्रिम तेषाम्‌ समान स्रोतः ख म समान इदानीमपि अधिकतया करोतु ते समान इत्यस्य वीथी सह यस्मिन्  कृतवान्‌ धृतः तदा पुनः पूर्वं सः आगतः किम्‌ कुल इतर पुरा  मात्रा स विषये उ अतएव अपि नगरस्य  उपरि यतः प्रतिशतं  कतरः कालः साधनानि भूत तथापि जात सम्बन्धि अन्यत्‌ ग अतः अस्माकं स्वकीयाः अस्माकं इदानीं अन्तः इत्यादयः भवन्तः इत्यादयः एते एताः तस्य अस्य इदम् एते तेषां तेषां तेषां तान् तेषां तेषां तेषां समानः सः एकः च तादृशाः बहवः अन्ये च वदन्ति यत् कियत् कस्मै  कस्मै  यस्मै  यस्मै  यस्मै  यस्मै न अतिनीचः किन्तु प्रथमं सम्पूर्णतया  ततः चिरकालानन्तरं पुस्तकं सम्पूर्णतया अन्तः  किन्तु अत्र वा इह इव श्रद्धाय अवशिष्यते  परन्तु अन्ये वर्गाः सन्ति ते सन्ति शक्नुवन्ति सर्वे मिलित्वा सर्वे एकत्र"'.split(" ")),e.sa.stemmer=function(){return function(e){return"function"==typeof e.update?e.update(function(e){return e}):e}}();var r=e.wordcut;r.init(),e.sa.tokenizer=function(t){if(!arguments.length||null==t||void 0==t)return[];if(Array.isArray(t))return t.map(function(r){return isLunr2?new e.Token(r.toLowerCase()):r.toLowerCase()});var i=t.toString().toLowerCase().replace(/^\s+/,"");return r.cut(i).split("|")},e.Pipeline.registerFunction(e.sa.stemmer,"stemmer-sa"),e.Pipeline.registerFunction(e.sa.stopWordFilter,"stopWordFilter-sa")}});

docs/_site/assets/javascripts/lunr/min/lunr.stemmer.support.min.js

@@ -0,0 +1 @@
+!function(r,t){"function"==typeof define&&define.amd?define(t):"object"==typeof exports?module.exports=t():t()(r.lunr)}(this,function(){return function(r){r.stemmerSupport={Among:function(r,t,i,s){if(this.toCharArray=function(r){for(var t=r.length,i=new Array(t),s=0;s<t;s++)i[s]=r.charCodeAt(s);return i},!r&&""!=r||!t&&0!=t||!i)throw"Bad Among initialisation: s:"+r+", substring_i: "+t+", result: "+i;this.s_size=r.length,this.s=this.toCharArray(r),this.substring_i=t,this.result=i,this.method=s},SnowballProgram:function(){var r;return{bra:0,ket:0,limit:0,cursor:0,limit_backward:0,setCurrent:function(t){r=t,this.cursor=0,this.limit=t.length,this.limit_backward=0,this.bra=this.cursor,this.ket=this.limit},getCurrent:function(){var t=r;return r=null,t},in_grouping:function(t,i,s){if(this.cursor<this.limit){var e=r.charCodeAt(this.cursor);if(e<=s&&e>=i&&(e-=i,t[e>>3]&1<<(7&e)))return this.cursor++,!0}return!1},in_grouping_b:function(t,i,s){if(this.cursor>this.limit_backward){var e=r.charCodeAt(this.cursor-1);if(e<=s&&e>=i&&(e-=i,t[e>>3]&1<<(7&e)))return this.cursor--,!0}return!1},out_grouping:function(t,i,s){if(this.cursor<this.limit){var e=r.charCodeAt(this.cursor);if(e>s||e<i)return this.cursor++,!0;if(e-=i,!(t[e>>3]&1<<(7&e)))return this.cursor++,!0}return!1},out_grouping_b:function(t,i,s){if(this.cursor>this.limit_backward){var e=r.charCodeAt(this.cursor-1);if(e>s||e<i)return this.cursor--,!0;if(e-=i,!(t[e>>3]&1<<(7&e)))return this.cursor--,!0}return!1},eq_s:function(t,i){if(this.limit-this.cursor<t)return!1;for(var s=0;s<t;s++)if(r.charCodeAt(this.cursor+s)!=i.charCodeAt(s))return!1;return this.cursor+=t,!0},eq_s_b:function(t,i){if(this.cursor-this.limit_backward<t)return!1;for(var s=0;s<t;s++)if(r.charCodeAt(this.cursor-t+s)!=i.charCodeAt(s))return!1;return this.cursor-=t,!0},find_among:function(t,i){for(var s=0,e=i,n=this.cursor,u=this.limit,o=0,h=0,c=!1;;){for(var a=s+(e-s>>1),f=0,l=o<h?o:h,_=t[a],m=l;m<_.s_size;m++){if(n+l==u){f=-1;break}if(f=r.charCodeAt(n+l)-_.s[m])break;l++}if(f<0?(e=a,h=l):(s=a,o=l),e-s<=1){if(s>0||e==s||c)break;c=!0}}for(;;){var _=t[s];if(o>=_.s_size){if(this.cursor=n+_.s_size,!_.method)return _.result;var b=_.method();if(this.cursor=n+_.s_size,b)return _.result}if((s=_.substring_i)<0)return 0}},find_among_b:function(t,i){for(var s=0,e=i,n=this.cursor,u=this.limit_backward,o=0,h=0,c=!1;;){for(var a=s+(e-s>>1),f=0,l=o<h?o:h,_=t[a],m=_.s_size-1-l;m>=0;m--){if(n-l==u){f=-1;break}if(f=r.charCodeAt(n-1-l)-_.s[m])break;l++}if(f<0?(e=a,h=l):(s=a,o=l),e-s<=1){if(s>0||e==s||c)break;c=!0}}for(;;){var _=t[s];if(o>=_.s_size){if(this.cursor=n-_.s_size,!_.method)return _.result;var b=_.method();if(this.cursor=n-_.s_size,b)return _.result}if((s=_.substring_i)<0)return 0}},replace_s:function(t,i,s){var e=s.length-(i-t),n=r.substring(0,t),u=r.substring(i);return r=n+s+u,this.limit+=e,this.cursor>=i?this.cursor+=e:this.cursor>t&&(this.cursor=t),e},slice_check:function(){if(this.bra<0||this.bra>this.ket||this.ket>this.limit||this.limit>r.length)throw"faulty slice operation"},slice_from:function(r){this.slice_check(),this.replace_s(this.bra,this.ket,r)},slice_del:function(){this.slice_from("")},insert:function(r,t,i){var s=this.replace_s(r,t,i);r<=this.bra&&(this.bra+=s),r<=this.ket&&(this.ket+=s)},slice_to:function(){return this.slice_check(),r.substring(this.bra,this.ket)},eq_v_b:function(r){return this.eq_s_b(r.length,r)}}}},r.trimmerSupport={generateTrimmer:function(r){var t=new RegExp("^[^"+r+"]+"),i=new RegExp("[^"+r+"]+$");return function(r){return"function"==typeof r.update?r.update(function(r){return r.replace(t,"").replace(i,"")}):r.replace(t,"").replace(i,"")}}}}});

docs/_site/assets/javascripts/lunr/min/lunr.sv.min.js

@@ -0,0 +1,18 @@
+/*!
+ * Lunr languages, `Swedish` language
+ * https://github.com/MihaiValentin/lunr-languages
+ *
+ * Copyright 2014, Mihai Valentin
+ * http://www.mozilla.org/MPL/
+ */
+/*!
+ * based on
+ * Snowball JavaScript Library v0.3
+ * http://code.google.com/p/urim/
+ * http://snowball.tartarus.org/
+ *
+ * Copyright 2010, Oleg Mazko
+ * http://www.mozilla.org/MPL/
+ */
+
+!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.sv=function(){this.pipeline.reset(),this.pipeline.add(e.sv.trimmer,e.sv.stopWordFilter,e.sv.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.sv.stemmer))},e.sv.wordCharacters="A-Za-zªºÀ-ÖØ-öø-ʸˠ-ˤᴀ-ᴥᴬ-ᵜᵢ-ᵥᵫ-ᵷᵹ-ᶾḀ-ỿⁱⁿₐ-ₜKÅℲⅎⅠ-ↈⱠ-ⱿꜢ-ꞇꞋ-ꞭꞰ-ꞷꟷ-ꟿꬰ-ꭚꭜ-ꭤff-stＡ-Ｚａ-ｚ",e.sv.trimmer=e.trimmerSupport.generateTrimmer(e.sv.wordCharacters),e.Pipeline.registerFunction(e.sv.trimmer,"trimmer-sv"),e.sv.stemmer=function(){var r=e.stemmerSupport.Among,n=e.stemmerSupport.SnowballProgram,t=new function(){function e(){var e,r=w.cursor+3;if(o=w.limit,0<=r||r<=w.limit){for(a=r;;){if(e=w.cursor,w.in_grouping(l,97,246)){w.cursor=e;break}if(w.cursor=e,w.cursor>=w.limit)return;w.cursor++}for(;!w.out_grouping(l,97,246);){if(w.cursor>=w.limit)return;w.cursor++}o=w.cursor,o<a&&(o=a)}}function t(){var e,r=w.limit_backward;if(w.cursor>=o&&(w.limit_backward=o,w.cursor=w.limit,w.ket=w.cursor,e=w.find_among_b(u,37),w.limit_backward=r,e))switch(w.bra=w.cursor,e){case 1:w.slice_del();break;case 2:w.in_grouping_b(d,98,121)&&w.slice_del()}}function i(){var e=w.limit_backward;w.cursor>=o&&(w.limit_backward=o,w.cursor=w.limit,w.find_among_b(c,7)&&(w.cursor=w.limit,w.ket=w.cursor,w.cursor>w.limit_backward&&(w.bra=--w.cursor,w.slice_del())),w.limit_backward=e)}function s(){var e,r;if(w.cursor>=o){if(r=w.limit_backward,w.limit_backward=o,w.cursor=w.limit,w.ket=w.cursor,e=w.find_among_b(m,5))switch(w.bra=w.cursor,e){case 1:w.slice_del();break;case 2:w.slice_from("lös");break;case 3:w.slice_from("full")}w.limit_backward=r}}var a,o,u=[new r("a",-1,1),new r("arna",0,1),new r("erna",0,1),new r("heterna",2,1),new r("orna",0,1),new r("ad",-1,1),new r("e",-1,1),new r("ade",6,1),new r("ande",6,1),new r("arne",6,1),new r("are",6,1),new r("aste",6,1),new r("en",-1,1),new r("anden",12,1),new r("aren",12,1),new r("heten",12,1),new r("ern",-1,1),new r("ar",-1,1),new r("er",-1,1),new r("heter",18,1),new r("or",-1,1),new r("s",-1,2),new r("as",21,1),new r("arnas",22,1),new r("ernas",22,1),new r("ornas",22,1),new r("es",21,1),new r("ades",26,1),new r("andes",26,1),new r("ens",21,1),new r("arens",29,1),new r("hetens",29,1),new r("erns",21,1),new r("at",-1,1),new r("andet",-1,1),new r("het",-1,1),new r("ast",-1,1)],c=[new r("dd",-1,-1),new r("gd",-1,-1),new r("nn",-1,-1),new r("dt",-1,-1),new r("gt",-1,-1),new r("kt",-1,-1),new r("tt",-1,-1)],m=[new r("ig",-1,1),new r("lig",0,1),new r("els",-1,1),new r("fullt",-1,3),new r("löst",-1,2)],l=[17,65,16,1,0,0,0,0,0,0,0,0,0,0,0,0,24,0,32],d=[119,127,149],w=new n;this.setCurrent=function(e){w.setCurrent(e)},this.getCurrent=function(){return w.getCurrent()},this.stem=function(){var r=w.cursor;return e(),w.limit_backward=r,w.cursor=w.limit,t(),w.cursor=w.limit,i(),w.cursor=w.limit,s(),!0}};return function(e){return"function"==typeof e.update?e.update(function(e){return t.setCurrent(e),t.stem(),t.getCurrent()}):(t.setCurrent(e),t.stem(),t.getCurrent())}}(),e.Pipeline.registerFunction(e.sv.stemmer,"stemmer-sv"),e.sv.stopWordFilter=e.generateStopWordFilter("alla allt att av blev bli blir blivit de dem den denna deras dess dessa det detta dig din dina ditt du där då efter ej eller en er era ert ett från för ha hade han hans har henne hennes hon honom hur här i icke ingen inom inte jag ju kan kunde man med mellan men mig min mina mitt mot mycket ni nu när någon något några och om oss på samma sedan sig sin sina sitta själv skulle som så sådan sådana sådant till under upp ut utan vad var vara varför varit varje vars vart vem vi vid vilka vilkas vilken vilket vår våra vårt än är åt över".split(" ")),e.Pipeline.registerFunction(e.sv.stopWordFilter,"stopWordFilter-sv")}});

docs/_site/assets/javascripts/lunr/min/lunr.ta.min.js

@@ -0,0 +1 @@
+!function(e,t){"function"==typeof define&&define.amd?define(t):"object"==typeof exports?module.exports=t():t()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.ta=function(){this.pipeline.reset(),this.pipeline.add(e.ta.trimmer,e.ta.stopWordFilter,e.ta.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.ta.stemmer))},e.ta.wordCharacters="஀-உஊ-ஏஐ-ஙச-ட஠-னப-யர-ஹ஺-ிீ-௉ொ-௏ௐ-௙௚-௟௠-௩௪-௯௰-௹௺-௿a-zA-Zａ-ｚＡ-Ｚ0-9０-９",e.ta.trimmer=e.trimmerSupport.generateTrimmer(e.ta.wordCharacters),e.Pipeline.registerFunction(e.ta.trimmer,"trimmer-ta"),e.ta.stopWordFilter=e.generateStopWordFilter("அங்கு அங்கே அது அதை அந்த அவர் அவர்கள் அவள் அவன் அவை ஆக ஆகவே ஆகையால் ஆதலால் ஆதலினால் ஆனாலும் ஆனால் இங்கு இங்கே இது இதை இந்த இப்படி இவர் இவர்கள் இவள் இவன் இவை இவ்வளவு உனக்கு உனது உன் உன்னால் எங்கு எங்கே எது எதை எந்த எப்படி எவர் எவர்கள் எவள் எவன் எவை எவ்வளவு எனக்கு எனது எனவே என் என்ன என்னால் ஏது ஏன் தனது தன்னால் தானே தான் நாங்கள் நாம் நான் நீ நீங்கள்".split(" ")),e.ta.stemmer=function(){return function(e){return"function"==typeof e.update?e.update(function(e){return e}):e}}();var t=e.wordcut;t.init(),e.ta.tokenizer=function(r){if(!arguments.length||null==r||void 0==r)return[];if(Array.isArray(r))return r.map(function(t){return isLunr2?new e.Token(t.toLowerCase()):t.toLowerCase()});var i=r.toString().toLowerCase().replace(/^\s+/,"");return t.cut(i).split("|")},e.Pipeline.registerFunction(e.ta.stemmer,"stemmer-ta"),e.Pipeline.registerFunction(e.ta.stopWordFilter,"stopWordFilter-ta")}});

docs/_site/assets/javascripts/lunr/min/lunr.te.min.js

@@ -0,0 +1 @@
+!function(e,t){"function"==typeof define&&define.amd?define(t):"object"==typeof exports?module.exports=t():t()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.te=function(){this.pipeline.reset(),this.pipeline.add(e.te.trimmer,e.te.stopWordFilter,e.te.stemmer),this.searchPipeline&&(this.searchPipeline.reset(),this.searchPipeline.add(e.te.stemmer))},e.te.wordCharacters="ఀ-ఄఅ-ఔక-హా-ౌౕ-ౖౘ-ౚౠ-ౡౢ-ౣ౦-౯౸-౿఼ఽ్ౝ౷౤౥",e.te.trimmer=e.trimmerSupport.generateTrimmer(e.te.wordCharacters),e.Pipeline.registerFunction(e.te.trimmer,"trimmer-te"),e.te.stopWordFilter=e.generateStopWordFilter("అందరూ అందుబాటులో అడగండి అడగడం అడ్డంగా అనుగుణంగా అనుమతించు అనుమతిస్తుంది అయితే ఇప్పటికే ఉన్నారు ఎక్కడైనా ఎప్పుడు ఎవరైనా ఎవరో ఏ ఏదైనా ఏమైనప్పటికి ఒక ఒకరు కనిపిస్తాయి కాదు కూడా గా గురించి చుట్టూ చేయగలిగింది తగిన తర్వాత దాదాపు దూరంగా నిజంగా పై ప్రకారం ప్రక్కన మధ్య మరియు మరొక మళ్ళీ మాత్రమే మెచ్చుకో వద్ద వెంట వేరుగా వ్యతిరేకంగా సంబంధం".split(" ")),e.te.stemmer=function(){return function(e){return"function"==typeof e.update?e.update(function(e){return e}):e}}();var t=e.wordcut;t.init(),e.te.tokenizer=function(r){if(!arguments.length||null==r||void 0==r)return[];if(Array.isArray(r))return r.map(function(t){return isLunr2?new e.Token(t.toLowerCase()):t.toLowerCase()});var i=r.toString().toLowerCase().replace(/^\s+/,"");return t.cut(i).split("|")},e.Pipeline.registerFunction(e.te.stemmer,"stemmer-te"),e.Pipeline.registerFunction(e.te.stopWordFilter,"stopWordFilter-te")}});

docs/_site/assets/javascripts/lunr/min/lunr.th.min.js

@@ -0,0 +1 @@
+!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");var r="2"==e.version[0];e.th=function(){this.pipeline.reset(),this.pipeline.add(e.th.trimmer),r?this.tokenizer=e.th.tokenizer:(e.tokenizer&&(e.tokenizer=e.th.tokenizer),this.tokenizerFn&&(this.tokenizerFn=e.th.tokenizer))},e.th.wordCharacters="[฀-๿]",e.th.trimmer=e.trimmerSupport.generateTrimmer(e.th.wordCharacters),e.Pipeline.registerFunction(e.th.trimmer,"trimmer-th");var t=e.wordcut;t.init(),e.th.tokenizer=function(i){if(!arguments.length||null==i||void 0==i)return[];if(Array.isArray(i))return i.map(function(t){return r?new e.Token(t):t});var n=i.toString().replace(/^\s+/,"");return t.cut(n).split("|")}}});

docs/_site/assets/javascripts/lunr/min/lunr.vi.min.js

@@ -0,0 +1 @@
+!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r():r()(e.lunr)}(this,function(){return function(e){if(void 0===e)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===e.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");e.vi=function(){this.pipeline.reset(),this.pipeline.add(e.vi.stopWordFilter,e.vi.trimmer)},e.vi.wordCharacters="[A-Za-ẓ̀͐́͑̉̃̓ÂâÊêÔôĂ-ăĐ-đƠ-ơƯ-ư]",e.vi.trimmer=e.trimmerSupport.generateTrimmer(e.vi.wordCharacters),e.Pipeline.registerFunction(e.vi.trimmer,"trimmer-vi"),e.vi.stopWordFilter=e.generateStopWordFilter("là cái nhưng mà".split(" "))}});

docs/_site/assets/javascripts/lunr/min/lunr.zh.min.js

@@ -0,0 +1 @@
+!function(e,r){"function"==typeof define&&define.amd?define(r):"object"==typeof exports?module.exports=r(require("@node-rs/jieba")):r()(e.lunr)}(this,function(e){return function(r,t){if(void 0===r)throw new Error("Lunr is not present. Please include / require Lunr before this script.");if(void 0===r.stemmerSupport)throw new Error("Lunr stemmer support is not present. Please include / require Lunr stemmer support before this script.");var i="2"==r.version[0];r.zh=function(){this.pipeline.reset(),this.pipeline.add(r.zh.trimmer,r.zh.stopWordFilter,r.zh.stemmer),i?this.tokenizer=r.zh.tokenizer:(r.tokenizer&&(r.tokenizer=r.zh.tokenizer),this.tokenizerFn&&(this.tokenizerFn=r.zh.tokenizer))},r.zh.tokenizer=function(n){if(!arguments.length||null==n||void 0==n)return[];if(Array.isArray(n))return n.map(function(e){return i?new r.Token(e.toLowerCase()):e.toLowerCase()});t&&e.load(t);var o=n.toString().trim().toLowerCase(),s=[];e.cut(o,!0).forEach(function(e){s=s.concat(e.split(" "))}),s=s.filter(function(e){return!!e});var u=0;return s.map(function(e,t){if(i){var n=o.indexOf(e,u),s={};return s.position=[n,e.length],s.index=t,u=n,new r.Token(e,s)}return e})},r.zh.wordCharacters="\\w一-龥",r.zh.trimmer=r.trimmerSupport.generateTrimmer(r.zh.wordCharacters),r.Pipeline.registerFunction(r.zh.trimmer,"trimmer-zh"),r.zh.stemmer=function(){return function(e){return e}}(),r.Pipeline.registerFunction(r.zh.stemmer,"stemmer-zh"),r.zh.stopWordFilter=r.generateStopWordFilter("的 一 不 在 人 有 是 为 為 以 于 於 上 他 而 后 後 之 来 來 及 了 因 下 可 到 由 这 這 与 與 也 此 但 并 並 个 個 其 已 无 無 小 我 们 們 起 最 再 今 去 好 只 又 或 很 亦 某 把 那 你 乃 它 吧 被 比 别 趁 当 當 从 從 得 打 凡 儿 兒 尔 爾 该 該 各 给 給 跟 和 何 还 還 即 几 幾 既 看 据 據 距 靠 啦 另 么 麽 每 嘛 拿 哪 您 凭 憑 且 却 卻 让 讓 仍 啥 如 若 使 谁 誰 虽 雖 随 隨 同 所 她 哇 嗡 往 些 向 沿 哟 喲 用 咱 则 則 怎 曾 至 致 着 著 诸 諸 自".split(" ")),r.Pipeline.registerFunction(r.zh.stopWordFilter,"stopWordFilter-zh")}});

docs/_site/assets/javascripts/lunr/tinyseg.js

@@ -0,0 +1,206 @@
+/**
+ * export the module via AMD, CommonJS or as a browser global
+ * Export code from https://github.com/umdjs/umd/blob/master/returnExports.js
+ */
+;(function (root, factory) {
+    if (typeof define === 'function' && define.amd) {
+        // AMD. Register as an anonymous module.
+        define(factory)
+    } else if (typeof exports === 'object') {
+        /**
+         * Node. Does not work with strict CommonJS, but
+         * only CommonJS-like environments that support module.exports,
+         * like Node.
+         */
+        module.exports = factory()
+    } else {
+        // Browser globals (root is window)
+        factory()(root.lunr);
+    }
+}(this, function () {
+    /**
+     * Just return a value to define the module export.
+     * This example returns an object, but the module
+     * can return a function as the exported value.
+     */
+
+    return function(lunr) {
+        // TinySegmenter 0.1 -- Super compact Japanese tokenizer in Javascript
+        // (c) 2008 Taku Kudo <taku@chasen.org>
+        // TinySegmenter is freely distributable under the terms of a new BSD licence.
+        // For details, see http://chasen.org/~taku/software/TinySegmenter/LICENCE.txt
+
+        function TinySegmenter() {
+          var patterns = {
+            "[一二三四五六七八九十百千万億兆]":"M",
+            "[一-龠々〆ヵヶ]":"H",
+            "[ぁ-ん]":"I",
+            "[ァ-ヴーｱ-ﾝﾞｰ]":"K",
+            "[a-zA-Zａ-ｚＡ-Ｚ]":"A",
+            "[0-9０-９]":"N"
+          }
+          this.chartype_ = [];
+          for (var i in patterns) {
+            var regexp = new RegExp(i);
+            this.chartype_.push([regexp, patterns[i]]);
+          }
+
+          this.BIAS__ = -332
+          this.BC1__ = {"HH":6,"II":2461,"KH":406,"OH":-1378};
+          this.BC2__ = {"AA":-3267,"AI":2744,"AN":-878,"HH":-4070,"HM":-1711,"HN":4012,"HO":3761,"IA":1327,"IH":-1184,"II":-1332,"IK":1721,"IO":5492,"KI":3831,"KK":-8741,"MH":-3132,"MK":3334,"OO":-2920};
+          this.BC3__ = {"HH":996,"HI":626,"HK":-721,"HN":-1307,"HO":-836,"IH":-301,"KK":2762,"MK":1079,"MM":4034,"OA":-1652,"OH":266};
+          this.BP1__ = {"BB":295,"OB":304,"OO":-125,"UB":352};
+          this.BP2__ = {"BO":60,"OO":-1762};
+          this.BQ1__ = {"BHH":1150,"BHM":1521,"BII":-1158,"BIM":886,"BMH":1208,"BNH":449,"BOH":-91,"BOO":-2597,"OHI":451,"OIH":-296,"OKA":1851,"OKH":-1020,"OKK":904,"OOO":2965};
+          this.BQ2__ = {"BHH":118,"BHI":-1159,"BHM":466,"BIH":-919,"BKK":-1720,"BKO":864,"OHH":-1139,"OHM":-181,"OIH":153,"UHI":-1146};
+          this.BQ3__ = {"BHH":-792,"BHI":2664,"BII":-299,"BKI":419,"BMH":937,"BMM":8335,"BNN":998,"BOH":775,"OHH":2174,"OHM":439,"OII":280,"OKH":1798,"OKI":-793,"OKO":-2242,"OMH":-2402,"OOO":11699};
+          this.BQ4__ = {"BHH":-3895,"BIH":3761,"BII":-4654,"BIK":1348,"BKK":-1806,"BMI":-3385,"BOO":-12396,"OAH":926,"OHH":266,"OHK":-2036,"ONN":-973};
+          this.BW1__ = {",と":660,",同":727,"B1あ":1404,"B1同":542,"、と":660,"、同":727,"」と":1682,"あっ":1505,"いう":1743,"いっ":-2055,"いる":672,"うし":-4817,"うん":665,"から":3472,"がら":600,"こう":-790,"こと":2083,"こん":-1262,"さら":-4143,"さん":4573,"した":2641,"して":1104,"すで":-3399,"そこ":1977,"それ":-871,"たち":1122,"ため":601,"った":3463,"つい":-802,"てい":805,"てき":1249,"でき":1127,"です":3445,"では":844,"とい":-4915,"とみ":1922,"どこ":3887,"ない":5713,"なっ":3015,"など":7379,"なん":-1113,"にし":2468,"には":1498,"にも":1671,"に対":-912,"の一":-501,"の中":741,"ませ":2448,"まで":1711,"まま":2600,"まる":-2155,"やむ":-1947,"よっ":-2565,"れた":2369,"れで":-913,"をし":1860,"を見":731,"亡く":-1886,"京都":2558,"取り":-2784,"大き":-2604,"大阪":1497,"平方":-2314,"引き":-1336,"日本":-195,"本当":-2423,"毎日":-2113,"目指":-724,"Ｂ１あ":1404,"Ｂ１同":542,"｣と":1682};
+          this.BW2__ = {"..":-11822,"11":-669,"――":-5730,"−−":-13175,"いう":-1609,"うか":2490,"かし":-1350,"かも":-602,"から":-7194,"かれ":4612,"がい":853,"がら":-3198,"きた":1941,"くな":-1597,"こと":-8392,"この":-4193,"させ":4533,"され":13168,"さん":-3977,"しい":-1819,"しか":-545,"した":5078,"して":972,"しな":939,"その":-3744,"たい":-1253,"たた":-662,"ただ":-3857,"たち":-786,"たと":1224,"たは":-939,"った":4589,"って":1647,"っと":-2094,"てい":6144,"てき":3640,"てく":2551,"ては":-3110,"ても":-3065,"でい":2666,"でき":-1528,"でし":-3828,"です":-4761,"でも":-4203,"とい":1890,"とこ":-1746,"とと":-2279,"との":720,"とみ":5168,"とも":-3941,"ない":-2488,"なが":-1313,"など":-6509,"なの":2614,"なん":3099,"にお":-1615,"にし":2748,"にな":2454,"によ":-7236,"に対":-14943,"に従":-4688,"に関":-11388,"のか":2093,"ので":-7059,"のに":-6041,"のの":-6125,"はい":1073,"はが":-1033,"はず":-2532,"ばれ":1813,"まし":-1316,"まで":-6621,"まれ":5409,"めて":-3153,"もい":2230,"もの":-10713,"らか":-944,"らし":-1611,"らに":-1897,"りし":651,"りま":1620,"れた":4270,"れて":849,"れば":4114,"ろう":6067,"われ":7901,"を通":-11877,"んだ":728,"んな":-4115,"一人":602,"一方":-1375,"一日":970,"一部":-1051,"上が":-4479,"会社":-1116,"出て":2163,"分の":-7758,"同党":970,"同日":-913,"大阪":-2471,"委員":-1250,"少な":-1050,"年度":-8669,"年間":-1626,"府県":-2363,"手権":-1982,"新聞":-4066,"日新":-722,"日本":-7068,"日米":3372,"曜日":-601,"朝鮮":-2355,"本人":-2697,"東京":-1543,"然と":-1384,"社会":-1276,"立て":-990,"第に":-1612,"米国":-4268,"１１":-669};
+          this.BW3__ = {"あた":-2194,"あり":719,"ある":3846,"い.":-1185,"い。":-1185,"いい":5308,"いえ":2079,"いく":3029,"いた":2056,"いっ":1883,"いる":5600,"いわ":1527,"うち":1117,"うと":4798,"えと":1454,"か.":2857,"か。":2857,"かけ":-743,"かっ":-4098,"かに":-669,"から":6520,"かり":-2670,"が,":1816,"が、":1816,"がき":-4855,"がけ":-1127,"がっ":-913,"がら":-4977,"がり":-2064,"きた":1645,"けど":1374,"こと":7397,"この":1542,"ころ":-2757,"さい":-714,"さを":976,"し,":1557,"し、":1557,"しい":-3714,"した":3562,"して":1449,"しな":2608,"しま":1200,"す.":-1310,"す。":-1310,"する":6521,"ず,":3426,"ず、":3426,"ずに":841,"そう":428,"た.":8875,"た。":8875,"たい":-594,"たの":812,"たり":-1183,"たる":-853,"だ.":4098,"だ。":4098,"だっ":1004,"った":-4748,"って":300,"てい":6240,"てお":855,"ても":302,"です":1437,"でに":-1482,"では":2295,"とう":-1387,"とし":2266,"との":541,"とも":-3543,"どう":4664,"ない":1796,"なく":-903,"など":2135,"に,":-1021,"に、":-1021,"にし":1771,"にな":1906,"には":2644,"の,":-724,"の、":-724,"の子":-1000,"は,":1337,"は、":1337,"べき":2181,"まし":1113,"ます":6943,"まっ":-1549,"まで":6154,"まれ":-793,"らし":1479,"られ":6820,"るる":3818,"れ,":854,"れ、":854,"れた":1850,"れて":1375,"れば":-3246,"れる":1091,"われ":-605,"んだ":606,"んで":798,"カ月":990,"会議":860,"入り":1232,"大会":2217,"始め":1681,"市":965,"新聞":-5055,"日,":974,"日、":974,"社会":2024,"ｶ月":990};
+          this.TC1__ = {"AAA":1093,"HHH":1029,"HHM":580,"HII":998,"HOH":-390,"HOM":-331,"IHI":1169,"IOH":-142,"IOI":-1015,"IOM":467,"MMH":187,"OOI":-1832};
+          this.TC2__ = {"HHO":2088,"HII":-1023,"HMM":-1154,"IHI":-1965,"KKH":703,"OII":-2649};
+          this.TC3__ = {"AAA":-294,"HHH":346,"HHI":-341,"HII":-1088,"HIK":731,"HOH":-1486,"IHH":128,"IHI":-3041,"IHO":-1935,"IIH":-825,"IIM":-1035,"IOI":-542,"KHH":-1216,"KKA":491,"KKH":-1217,"KOK":-1009,"MHH":-2694,"MHM":-457,"MHO":123,"MMH":-471,"NNH":-1689,"NNO":662,"OHO":-3393};
+          this.TC4__ = {"HHH":-203,"HHI":1344,"HHK":365,"HHM":-122,"HHN":182,"HHO":669,"HIH":804,"HII":679,"HOH":446,"IHH":695,"IHO":-2324,"IIH":321,"III":1497,"IIO":656,"IOO":54,"KAK":4845,"KKA":3386,"KKK":3065,"MHH":-405,"MHI":201,"MMH":-241,"MMM":661,"MOM":841};
+          this.TQ1__ = {"BHHH":-227,"BHHI":316,"BHIH":-132,"BIHH":60,"BIII":1595,"BNHH":-744,"BOHH":225,"BOOO":-908,"OAKK":482,"OHHH":281,"OHIH":249,"OIHI":200,"OIIH":-68};
+          this.TQ2__ = {"BIHH":-1401,"BIII":-1033,"BKAK":-543,"BOOO":-5591};
+          this.TQ3__ = {"BHHH":478,"BHHM":-1073,"BHIH":222,"BHII":-504,"BIIH":-116,"BIII":-105,"BMHI":-863,"BMHM":-464,"BOMH":620,"OHHH":346,"OHHI":1729,"OHII":997,"OHMH":481,"OIHH":623,"OIIH":1344,"OKAK":2792,"OKHH":587,"OKKA":679,"OOHH":110,"OOII":-685};
+          this.TQ4__ = {"BHHH":-721,"BHHM":-3604,"BHII":-966,"BIIH":-607,"BIII":-2181,"OAAA":-2763,"OAKK":180,"OHHH":-294,"OHHI":2446,"OHHO":480,"OHIH":-1573,"OIHH":1935,"OIHI":-493,"OIIH":626,"OIII":-4007,"OKAK":-8156};
+          this.TW1__ = {"につい":-4681,"東京都":2026};
+          this.TW2__ = {"ある程":-2049,"いった":-1256,"ころが":-2434,"しょう":3873,"その後":-4430,"だって":-1049,"ていた":1833,"として":-4657,"ともに":-4517,"もので":1882,"一気に":-792,"初めて":-1512,"同時に":-8097,"大きな":-1255,"対して":-2721,"社会党":-3216};
+          this.TW3__ = {"いただ":-1734,"してい":1314,"として":-4314,"につい":-5483,"にとっ":-5989,"に当た":-6247,"ので,":-727,"ので、":-727,"のもの":-600,"れから":-3752,"十二月":-2287};
+          this.TW4__ = {"いう.":8576,"いう。":8576,"からな":-2348,"してい":2958,"たが,":1516,"たが、":1516,"ている":1538,"という":1349,"ました":5543,"ません":1097,"ようと":-4258,"よると":5865};
+          this.UC1__ = {"A":484,"K":93,"M":645,"O":-505};
+          this.UC2__ = {"A":819,"H":1059,"I":409,"M":3987,"N":5775,"O":646};
+          this.UC3__ = {"A":-1370,"I":2311};
+          this.UC4__ = {"A":-2643,"H":1809,"I":-1032,"K":-3450,"M":3565,"N":3876,"O":6646};
+          this.UC5__ = {"H":313,"I":-1238,"K":-799,"M":539,"O":-831};
+          this.UC6__ = {"H":-506,"I":-253,"K":87,"M":247,"O":-387};
+          this.UP1__ = {"O":-214};
+          this.UP2__ = {"B":69,"O":935};
+          this.UP3__ = {"B":189};
+          this.UQ1__ = {"BH":21,"BI":-12,"BK":-99,"BN":142,"BO":-56,"OH":-95,"OI":477,"OK":410,"OO":-2422};
+          this.UQ2__ = {"BH":216,"BI":113,"OK":1759};
+          this.UQ3__ = {"BA":-479,"BH":42,"BI":1913,"BK":-7198,"BM":3160,"BN":6427,"BO":14761,"OI":-827,"ON":-3212};
+          this.UW1__ = {",":156,"、":156,"「":-463,"あ":-941,"う":-127,"が":-553,"き":121,"こ":505,"で":-201,"と":-547,"ど":-123,"に":-789,"の":-185,"は":-847,"も":-466,"や":-470,"よ":182,"ら":-292,"り":208,"れ":169,"を":-446,"ん":-137,"・":-135,"主":-402,"京":-268,"区":-912,"午":871,"国":-460,"大":561,"委":729,"市":-411,"日":-141,"理":361,"生":-408,"県":-386,"都":-718,"｢":-463,"･":-135};
+          this.UW2__ = {",":-829,"、":-829,"〇":892,"「":-645,"」":3145,"あ":-538,"い":505,"う":134,"お":-502,"か":1454,"が":-856,"く":-412,"こ":1141,"さ":878,"ざ":540,"し":1529,"す":-675,"せ":300,"そ":-1011,"た":188,"だ":1837,"つ":-949,"て":-291,"で":-268,"と":-981,"ど":1273,"な":1063,"に":-1764,"の":130,"は":-409,"ひ":-1273,"べ":1261,"ま":600,"も":-1263,"や":-402,"よ":1639,"り":-579,"る":-694,"れ":571,"を":-2516,"ん":2095,"ア":-587,"カ":306,"キ":568,"ッ":831,"三":-758,"不":-2150,"世":-302,"中":-968,"主":-861,"事":492,"人":-123,"会":978,"保":362,"入":548,"初":-3025,"副":-1566,"北":-3414,"区":-422,"大":-1769,"天":-865,"太":-483,"子":-1519,"学":760,"実":1023,"小":-2009,"市":-813,"年":-1060,"強":1067,"手":-1519,"揺":-1033,"政":1522,"文":-1355,"新":-1682,"日":-1815,"明":-1462,"最":-630,"朝":-1843,"本":-1650,"東":-931,"果":-665,"次":-2378,"民":-180,"気":-1740,"理":752,"発":529,"目":-1584,"相":-242,"県":-1165,"立":-763,"第":810,"米":509,"自":-1353,"行":838,"西":-744,"見":-3874,"調":1010,"議":1198,"込":3041,"開":1758,"間":-1257,"｢":-645,"｣":3145,"ｯ":831,"ｱ":-587,"ｶ":306,"ｷ":568};
+          this.UW3__ = {",":4889,"1":-800,"−":-1723,"、":4889,"々":-2311,"〇":5827,"」":2670,"〓":-3573,"あ":-2696,"い":1006,"う":2342,"え":1983,"お":-4864,"か":-1163,"が":3271,"く":1004,"け":388,"げ":401,"こ":-3552,"ご":-3116,"さ":-1058,"し":-395,"す":584,"せ":3685,"そ":-5228,"た":842,"ち":-521,"っ":-1444,"つ":-1081,"て":6167,"で":2318,"と":1691,"ど":-899,"な":-2788,"に":2745,"の":4056,"は":4555,"ひ":-2171,"ふ":-1798,"へ":1199,"ほ":-5516,"ま":-4384,"み":-120,"め":1205,"も":2323,"や":-788,"よ":-202,"ら":727,"り":649,"る":5905,"れ":2773,"わ":-1207,"を":6620,"ん":-518,"ア":551,"グ":1319,"ス":874,"ッ":-1350,"ト":521,"ム":1109,"ル":1591,"ロ":2201,"ン":278,"・":-3794,"一":-1619,"下":-1759,"世":-2087,"両":3815,"中":653,"主":-758,"予":-1193,"二":974,"人":2742,"今":792,"他":1889,"以":-1368,"低":811,"何":4265,"作":-361,"保":-2439,"元":4858,"党":3593,"全":1574,"公":-3030,"六":755,"共":-1880,"円":5807,"再":3095,"分":457,"初":2475,"別":1129,"前":2286,"副":4437,"力":365,"動":-949,"務":-1872,"化":1327,"北":-1038,"区":4646,"千":-2309,"午":-783,"協":-1006,"口":483,"右":1233,"各":3588,"合":-241,"同":3906,"和":-837,"員":4513,"国":642,"型":1389,"場":1219,"外":-241,"妻":2016,"学":-1356,"安":-423,"実":-1008,"家":1078,"小":-513,"少":-3102,"州":1155,"市":3197,"平":-1804,"年":2416,"広":-1030,"府":1605,"度":1452,"建":-2352,"当":-3885,"得":1905,"思":-1291,"性":1822,"戸":-488,"指":-3973,"政":-2013,"教":-1479,"数":3222,"文":-1489,"新":1764,"日":2099,"旧":5792,"昨":-661,"時":-1248,"曜":-951,"最":-937,"月":4125,"期":360,"李":3094,"村":364,"東":-805,"核":5156,"森":2438,"業":484,"氏":2613,"民":-1694,"決":-1073,"法":1868,"海":-495,"無":979,"物":461,"特":-3850,"生":-273,"用":914,"町":1215,"的":7313,"直":-1835,"省":792,"県":6293,"知":-1528,"私":4231,"税":401,"立":-960,"第":1201,"米":7767,"系":3066,"約":3663,"級":1384,"統":-4229,"総":1163,"線":1255,"者":6457,"能":725,"自":-2869,"英":785,"見":1044,"調":-562,"財":-733,"費":1777,"車":1835,"軍":1375,"込":-1504,"通":-1136,"選":-681,"郎":1026,"郡":4404,"部":1200,"金":2163,"長":421,"開":-1432,"間":1302,"関":-1282,"雨":2009,"電":-1045,"非":2066,"駅":1620,"１":-800,"｣":2670,"･":-3794,"ｯ":-1350,"ｱ":551,"ｸﾞ":1319,"ｽ":874,"ﾄ":521,"ﾑ":1109,"ﾙ":1591,"ﾛ":2201,"ﾝ":278};
+          this.UW4__ = {",":3930,".":3508,"―":-4841,"、":3930,"。":3508,"〇":4999,"「":1895,"」":3798,"〓":-5156,"あ":4752,"い":-3435,"う":-640,"え":-2514,"お":2405,"か":530,"が":6006,"き":-4482,"ぎ":-3821,"く":-3788,"け":-4376,"げ":-4734,"こ":2255,"ご":1979,"さ":2864,"し":-843,"じ":-2506,"す":-731,"ず":1251,"せ":181,"そ":4091,"た":5034,"だ":5408,"ち":-3654,"っ":-5882,"つ":-1659,"て":3994,"で":7410,"と":4547,"な":5433,"に":6499,"ぬ":1853,"ね":1413,"の":7396,"は":8578,"ば":1940,"ひ":4249,"び":-4134,"ふ":1345,"へ":6665,"べ":-744,"ほ":1464,"ま":1051,"み":-2082,"む":-882,"め":-5046,"も":4169,"ゃ":-2666,"や":2795,"ょ":-1544,"よ":3351,"ら":-2922,"り":-9726,"る":-14896,"れ":-2613,"ろ":-4570,"わ":-1783,"を":13150,"ん":-2352,"カ":2145,"コ":1789,"セ":1287,"ッ":-724,"ト":-403,"メ":-1635,"ラ":-881,"リ":-541,"ル":-856,"ン":-3637,"・":-4371,"ー":-11870,"一":-2069,"中":2210,"予":782,"事":-190,"井":-1768,"人":1036,"以":544,"会":950,"体":-1286,"作":530,"側":4292,"先":601,"党":-2006,"共":-1212,"内":584,"円":788,"初":1347,"前":1623,"副":3879,"力":-302,"動":-740,"務":-2715,"化":776,"区":4517,"協":1013,"参":1555,"合":-1834,"和":-681,"員":-910,"器":-851,"回":1500,"国":-619,"園":-1200,"地":866,"場":-1410,"塁":-2094,"士":-1413,"多":1067,"大":571,"子":-4802,"学":-1397,"定":-1057,"寺":-809,"小":1910,"屋":-1328,"山":-1500,"島":-2056,"川":-2667,"市":2771,"年":374,"庁":-4556,"後":456,"性":553,"感":916,"所":-1566,"支":856,"改":787,"政":2182,"教":704,"文":522,"方":-856,"日":1798,"時":1829,"最":845,"月":-9066,"木":-485,"来":-442,"校":-360,"業":-1043,"氏":5388,"民":-2716,"気":-910,"沢":-939,"済":-543,"物":-735,"率":672,"球":-1267,"生":-1286,"産":-1101,"田":-2900,"町":1826,"的":2586,"目":922,"省":-3485,"県":2997,"空":-867,"立":-2112,"第":788,"米":2937,"系":786,"約":2171,"経":1146,"統":-1169,"総":940,"線":-994,"署":749,"者":2145,"能":-730,"般":-852,"行":-792,"規":792,"警":-1184,"議":-244,"谷":-1000,"賞":730,"車":-1481,"軍":1158,"輪":-1433,"込":-3370,"近":929,"道":-1291,"選":2596,"郎":-4866,"都":1192,"野":-1100,"銀":-2213,"長":357,"間":-2344,"院":-2297,"際":-2604,"電":-878,"領":-1659,"題":-792,"館":-1984,"首":1749,"高":2120,"｢":1895,"｣":3798,"･":-4371,"ｯ":-724,"ｰ":-11870,"ｶ":2145,"ｺ":1789,"ｾ":1287,"ﾄ":-403,"ﾒ":-1635,"ﾗ":-881,"ﾘ":-541,"ﾙ":-856,"ﾝ":-3637};
+          this.UW5__ = {",":465,".":-299,"1":-514,"E2":-32768,"]":-2762,"、":465,"。":-299,"「":363,"あ":1655,"い":331,"う":-503,"え":1199,"お":527,"か":647,"が":-421,"き":1624,"ぎ":1971,"く":312,"げ":-983,"さ":-1537,"し":-1371,"す":-852,"だ":-1186,"ち":1093,"っ":52,"つ":921,"て":-18,"で":-850,"と":-127,"ど":1682,"な":-787,"に":-1224,"の":-635,"は":-578,"べ":1001,"み":502,"め":865,"ゃ":3350,"ょ":854,"り":-208,"る":429,"れ":504,"わ":419,"を":-1264,"ん":327,"イ":241,"ル":451,"ン":-343,"中":-871,"京":722,"会":-1153,"党":-654,"務":3519,"区":-901,"告":848,"員":2104,"大":-1296,"学":-548,"定":1785,"嵐":-1304,"市":-2991,"席":921,"年":1763,"思":872,"所":-814,"挙":1618,"新":-1682,"日":218,"月":-4353,"査":932,"格":1356,"機":-1508,"氏":-1347,"田":240,"町":-3912,"的":-3149,"相":1319,"省":-1052,"県":-4003,"研":-997,"社":-278,"空":-813,"統":1955,"者":-2233,"表":663,"語":-1073,"議":1219,"選":-1018,"郎":-368,"長":786,"間":1191,"題":2368,"館":-689,"１":-514,"Ｅ２":-32768,"｢":363,"ｲ":241,"ﾙ":451,"ﾝ":-343};
+          this.UW6__ = {",":227,".":808,"1":-270,"E1":306,"、":227,"。":808,"あ":-307,"う":189,"か":241,"が":-73,"く":-121,"こ":-200,"じ":1782,"す":383,"た":-428,"っ":573,"て":-1014,"で":101,"と":-105,"な":-253,"に":-149,"の":-417,"は":-236,"も":-206,"り":187,"る":-135,"を":195,"ル":-673,"ン":-496,"一":-277,"中":201,"件":-800,"会":624,"前":302,"区":1792,"員":-1212,"委":798,"学":-960,"市":887,"広":-695,"後":535,"業":-697,"相":753,"社":-507,"福":974,"空":-822,"者":1811,"連":463,"郎":1082,"１":-270,"Ｅ１":306,"ﾙ":-673,"ﾝ":-496};
+          
+          return this;
+        }
+        TinySegmenter.prototype.ctype_ = function(str) {
+          for (var i in this.chartype_) {
+            if (str.match(this.chartype_[i][0])) {
+              return this.chartype_[i][1];
+            }
+          }
+          return "O";
+        }
+
+        TinySegmenter.prototype.ts_ = function(v) {
+          if (v) { return v; }
+          return 0;
+        }
+
+        TinySegmenter.prototype.segment = function(input) {
+          if (input == null || input == undefined || input == "") {
+            return [];
+          }
+          var result = [];
+          var seg = ["B3","B2","B1"];
+          var ctype = ["O","O","O"];
+          var o = input.split("");
+          for (i = 0; i < o.length; ++i) {
+            seg.push(o[i]);
+            ctype.push(this.ctype_(o[i]))
+          }
+          seg.push("E1");
+          seg.push("E2");
+          seg.push("E3");
+          ctype.push("O");
+          ctype.push("O");
+          ctype.push("O");
+          var word = seg[3];
+          var p1 = "U";
+          var p2 = "U";
+          var p3 = "U";
+          for (var i = 4; i < seg.length - 3; ++i) {
+            var score = this.BIAS__;
+            var w1 = seg[i-3];
+            var w2 = seg[i-2];
+            var w3 = seg[i-1];
+            var w4 = seg[i];
+            var w5 = seg[i+1];
+            var w6 = seg[i+2];
+            var c1 = ctype[i-3];
+            var c2 = ctype[i-2];
+            var c3 = ctype[i-1];
+            var c4 = ctype[i];
+            var c5 = ctype[i+1];
+            var c6 = ctype[i+2];
+            score += this.ts_(this.UP1__[p1]);
+            score += this.ts_(this.UP2__[p2]);
+            score += this.ts_(this.UP3__[p3]);
+            score += this.ts_(this.BP1__[p1 + p2]);
+            score += this.ts_(this.BP2__[p2 + p3]);
+            score += this.ts_(this.UW1__[w1]);
+            score += this.ts_(this.UW2__[w2]);
+            score += this.ts_(this.UW3__[w3]);
+            score += this.ts_(this.UW4__[w4]);
+            score += this.ts_(this.UW5__[w5]);
+            score += this.ts_(this.UW6__[w6]);
+            score += this.ts_(this.BW1__[w2 + w3]);
+            score += this.ts_(this.BW2__[w3 + w4]);
+            score += this.ts_(this.BW3__[w4 + w5]);
+            score += this.ts_(this.TW1__[w1 + w2 + w3]);
+            score += this.ts_(this.TW2__[w2 + w3 + w4]);
+            score += this.ts_(this.TW3__[w3 + w4 + w5]);
+            score += this.ts_(this.TW4__[w4 + w5 + w6]);
+            score += this.ts_(this.UC1__[c1]);
+            score += this.ts_(this.UC2__[c2]);
+            score += this.ts_(this.UC3__[c3]);
+            score += this.ts_(this.UC4__[c4]);
+            score += this.ts_(this.UC5__[c5]);
+            score += this.ts_(this.UC6__[c6]);
+            score += this.ts_(this.BC1__[c2 + c3]);
+            score += this.ts_(this.BC2__[c3 + c4]);
+            score += this.ts_(this.BC3__[c4 + c5]);
+            score += this.ts_(this.TC1__[c1 + c2 + c3]);
+            score += this.ts_(this.TC2__[c2 + c3 + c4]);
+            score += this.ts_(this.TC3__[c3 + c4 + c5]);
+            score += this.ts_(this.TC4__[c4 + c5 + c6]);
+        //  score += this.ts_(this.TC5__[c4 + c5 + c6]);    
+            score += this.ts_(this.UQ1__[p1 + c1]);
+            score += this.ts_(this.UQ2__[p2 + c2]);
+            score += this.ts_(this.UQ3__[p3 + c3]);
+            score += this.ts_(this.BQ1__[p2 + c2 + c3]);
+            score += this.ts_(this.BQ2__[p2 + c3 + c4]);
+            score += this.ts_(this.BQ3__[p3 + c2 + c3]);
+            score += this.ts_(this.BQ4__[p3 + c3 + c4]);
+            score += this.ts_(this.TQ1__[p2 + c1 + c2 + c3]);
+            score += this.ts_(this.TQ2__[p2 + c2 + c3 + c4]);
+            score += this.ts_(this.TQ3__[p3 + c1 + c2 + c3]);
+            score += this.ts_(this.TQ4__[p3 + c2 + c3 + c4]);
+            var p = "O";
+            if (score > 0) {
+              result.push(word);
+              word = "";
+              p = "B";
+            }
+            p1 = p2;
+            p2 = p3;
+            p3 = p;
+            word += seg[i];
+          }
+          result.push(word);
+
+          return result;
+        }
+
+        lunr.TinySegmenter = TinySegmenter;
+    };
+
+}));

docs/_site/assets/stylesheets/palette.ab4e12ef.min.css.map

@@ -0,0 +1 @@
+{"version":3,"sources":["src/templates/assets/stylesheets/palette/_scheme.scss","../../../../src/templates/assets/stylesheets/palette.scss","src/templates/assets/stylesheets/palette/_accent.scss","src/templates/assets/stylesheets/palette/_primary.scss","src/templates/assets/stylesheets/utilities/_break.scss"],"names":[],"mappings":"AA2BA,cAGE,6BAME,sDAAA,CACA,6DAAA,CACA,+DAAA,CACA,gEAAA,CACA,mDAAA,CACA,6DAAA,CACA,+DAAA,CACA,gEAAA,CAGA,mDAAA,CACA,gDAAA,CACA,yDAAA,CACA,4DAAA,CAGA,0BAAA,CACA,mCAAA,CAGA,iCAAA,CACA,kCAAA,CACA,mCAAA,CACA,mCAAA,CACA,kCAAA,CACA,iCAAA,CACA,+CAAA,CACA,6DAAA,CACA,gEAAA,CACA,4DAAA,CACA,4DAAA,CACA,6DAAA,CAGA,6CAAA,CAGA,+CAAA,CAGA,uDAAA,CACA,6DAAA,CACA,2DAAA,CAGA,iCAAA,CAGA,yDAAA,CACA,iEAAA,CAGA,mDAAA,CACA,mDAAA,CAGA,qDAAA,CACA,uDAAA,CAGA,8DAAA,CAKA,8DAAA,CAKA,0DAAA,CAzEA,iBCiBF,CD6DE,kHAEE,YC3DJ,CDkFE,yDACE,4BChFJ,CD+EE,2DACE,4BC7EJ,CD4EE,gEACE,4BC1EJ,CDyEE,2DACE,4BCvEJ,CDsEE,yDACE,4BCpEJ,CDmEE,0DACE,4BCjEJ,CDgEE,gEACE,4BC9DJ,CD6DE,0DACE,4BC3DJ,CD0DE,2OACE,4BC/CJ,CDsDA,+FAGE,iCCpDF,CACF,CCjDE,2BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCD6CN,CCvDE,4BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDoDN,CC9DE,8BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCD2DN,CCrEE,mCACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDkEN,CC5EE,8BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDyEN,CCnFE,4BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDgFN,CC1FE,kCACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDuFN,CCjGE,4BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCD8FN,CCxGE,4BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDqGN,CC/GE,6BACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCD4GN,CCtHE,mCACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDmHN,CC7HE,4BACE,4BAAA,CACA,2CAAA,CAIE,8BAAA,CACA,qCD6HN,CCpIE,8BACE,4BAAA,CACA,2CAAA,CAIE,8BAAA,CACA,qCDoIN,CC3IE,6BACE,yBAAA,CACA,2CAAA,CAIE,8BAAA,CACA,qCD2IN,CClJE,8BACE,4BAAA,CACA,2CAAA,CAIE,8BAAA,CACA,qCDkJN,CCzJE,mCACE,4BAAA,CACA,2CAAA,CAOE,yBAAA,CACA,qCDsJN,CE3JE,4BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFwJN,CEnKE,6BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFgKN,CE3KE,+BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFwKN,CEnLE,oCACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFgLN,CE3LE,+BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFwLN,CEnME,6BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFgMN,CE3ME,mCACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFwMN,CEnNE,6BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFgNN,CE3NE,6BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFwNN,CEnOE,8BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFgON,CE3OE,oCACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFwON,CEnPE,6BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAIE,+BAAA,CACA,sCFmPN,CE3PE,+BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAIE,+BAAA,CACA,sCF2PN,CEnQE,8BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAIE,+BAAA,CACA,sCFmQN,CE3QE,+BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAIE,+BAAA,CACA,sCF2QN,CEnRE,oCACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFgRN,CE3RE,8BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCFwRN,CEnSE,6BACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCAAA,CAKA,4BF4RN,CE5SE,kCACE,6BAAA,CACA,oCAAA,CACA,mCAAA,CAOE,0BAAA,CACA,sCAAA,CAKA,4BFqSN,CEtRE,sEACE,4BFyRJ,CE1RE,+DACE,4BF6RJ,CE9RE,iEACE,4BFiSJ,CElSE,gEACE,4BFqSJ,CEtSE,iEACE,4BFySJ,CEhSA,8BACE,mDAAA,CACA,4DAAA,CACA,0DAAA,CACA,oDAAA,CACA,2DAAA,CAGA,4BFiSF,CE9RE,yCACE,+BFgSJ,CE7RI,kDAEE,0CAAA,CACA,sCAAA,CAFA,mCFiSN,CG7MI,mCD1EA,+CACE,8CF0RJ,CEvRI,qDACE,8CFyRN,CEpRE,iEACE,mCFsRJ,CACF,CGxNI,sCDvDA,uCACE,oCFkRJ,CACF,CEzQA,8BACE,kDAAA,CACA,4DAAA,CACA,wDAAA,CACA,oDAAA,CACA,6DAAA,CAGA,4BF0QF,CEvQE,yCACE,+BFyQJ,CEtQI,kDAEE,0CAAA,CACA,sCAAA,CAFA,mCF0QN,CEnQE,yCACE,6CFqQJ,CG9NI,0CDhCA,8CACE,gDFiQJ,CACF,CGnOI,0CDvBA,iFACE,6CF6PJ,CACF,CG3PI,sCDKA,uCACE,6CFyPJ,CACF","file":"palette.css"}

docs/_site/sitemap.xml

@@ -0,0 +1,3 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
+</urlset>

docs/mkdocs.yml

@@ -0,0 +1,71 @@
+site_name: Fetch ML Documentation
+site_description: Secure Machine Learning Platform
+repo_url: https://github.com/jfraeys/fetch_ml
+edit_uri: edit/main/docs/
+
+docs_dir: src
+site_dir: _site
+
+nav:
+  - Home: index.md
+  - Getting Started:
+      - quick-start.md
+      - installation.md
+      - first-experiment.md
+  - Development:
+      - development-setup.md
+      - testing.md
+      - architecture.md
+      - cli-reference.md
+      - zig-cli.md
+      - queue.md
+      - smart-defaults.md
+      - cicd.md
+  - Operations & Production:
+      - deployment.md
+      - environment-variables.md
+      - production-monitoring.md
+      - operations.md
+      - redis-ha.md
+      - release-checklist.md
+  - Security:
+      - security.md
+      - api-key-process.md
+      - user-permissions.md
+  - Reference:
+      - configuration-schema.md
+      - troubleshooting.md
+
+theme:
+  name: material
+  palette:
+    - scheme: default
+      primary: blue
+      accent: blue
+    - scheme: slate
+      primary: blue
+      accent: blue
+  font:
+    text: Roboto
+    code: Roboto Mono
+  features:
+    - navigation.instant
+    - navigation.tracking
+    - navigation.tabs
+    - navigation.sections
+    - navigation.expand
+    - navigation.indexes
+    - toc.integrate
+    - search.highlight
+    - search.share
+
+plugins:
+  - search
+
+markdown_extensions:
+  - codehilite
+  - admonition
+  - toc:
+      permalink: true
+  - pymdownx.superfences
+  - pymdownx.highlight

docs/src/api-key-process.md

@@ -0,0 +1,126 @@
+# FetchML API Key Process
+
+This document describes how API keys are issued and how team members should configure the `ml` CLI to use them.
+
+The goal is to keep access easy for your homelab while treating API keys as sensitive secrets.
+
+## Overview
+
+- Each user gets a **personal API key** (no shared admin keys for normal use).
+- API keys are used by the `ml` CLI to authenticate to the FetchML API.
+- API keys and their **SHA256 hashes** must both be treated as secrets.
+
+There are two supported ways to receive your key:
+
+1. **Bitwarden (recommended)** – for users who already use Bitwarden.
+2. **Direct share (minimal tools)** – for users who do not use Bitwarden.
+
+---
+
+## 1. Bitwarden-based process (recommended)
+
+### For the admin
+
+- Use the helper script to create a Bitwarden item for each user:
+
+  ```bash
+  ./scripts/create_bitwarden_fetchml_item.sh <username> <api_key> <api_key_hash>
+  ```
+
+  This script:
+
+  - Creates a Bitwarden item named `FetchML API – <username>`.
+  - Stores:
+    - Username: `<username>`
+    - Password: `<api_key>` (the actual API key)
+    - Custom field `api_key_hash`: `<api_key_hash>`
+
+- Share that item with the user in Bitwarden (for example, via a shared collection like `FetchML`).
+
+### For the user
+
+1. Open Bitwarden and locate the item:
+
+   - **Name:** `FetchML API – <your-name>`
+
+2. Copy the **password** field (this is your FetchML API key).
+
+3. Configure the CLI, e.g. in `~/.ml/config.toml`:
+
+   ```toml
+   api_key     = "<paste-from-bitwarden>"
+   worker_host = "localhost"
+   worker_port = 9100
+   api_url     = "ws://localhost:9100/ws"
+   ```
+
+4. Test your setup:
+
+   ```bash
+   ml status
+   ```
+
+If the command works, your key and tunnel/config are correct.
+
+---
+
+## 2. Direct share (no password manager required)
+
+For users who do not use Bitwarden, a lightweight alternative is a direct one-to-one share.
+
+### For the admin
+
+1. Generate a **per-user** API key and hash as usual.
+2. Store them securely on your side (for example, in your own Bitwarden vault or configuration files).
+3. Share **only the API key** with the user via a direct channel you both trust, such as:
+
+   - Signal / WhatsApp direct message
+   - SMS
+   - Short call/meeting where you read it to them
+
+4. Ask the user to:
+
+   - Paste the key into their local config.
+   - Avoid keeping the key in plain chat history if possible.
+
+### For the user
+
+1. When you receive your FetchML API key from the admin, create or edit `~/.ml/config.toml`:
+
+   ```toml
+   api_key     = "<your-api-key>"
+   worker_host = "localhost"
+   worker_port = 9100
+   api_url     = "ws://localhost:9100/ws"
+   ```
+
+2. Save the file and run:
+
+   ```bash
+   ml status
+   ```
+
+3. If it works, you are ready to use the CLI:
+
+   ```bash
+   ml queue my-training-job
+   ml cancel my-training-job
+   ```
+
+---
+
+## 3. Security notes
+
+- **API key and hash are secrets**
+  - The 64-character `api_key_hash` is as sensitive as the API key itself.
+  - Do not commit keys or hashes to Git or share them in screenshots or tickets.
+
+- **Rotation**
+  - If you suspect a key has leaked, notify the admin.
+  - The admin will revoke the old key, generate a new one, and update Bitwarden or share a new key.
+
+- **Transport security**
+  - The `api_url` is typically `ws://localhost:9100/ws` when used through an SSH tunnel to the homelab.
+  - The SSH tunnel and nginx/TLS provide encryption over the network.
+
+Following these steps keeps API access easy for the team while maintaining a reasonable security posture for a personal homelab deployment.

docs/src/architecture.md

@@ -0,0 +1,738 @@
+---
+layout: page
+title: "Homelab Architecture"
+permalink: /architecture/
+nav_order: 1
+---
+
+# Homelab Architecture
+
+Simple, secure architecture for ML experiments in your homelab.
+
+## Components Overview
+
+```mermaid
+graph TB
+    subgraph "Homelab Stack"
+        CLI[Zig CLI]
+        API[HTTPS API]
+        REDIS[Redis Cache]
+        FS[Local Storage]
+    end
+    
+    CLI --> API
+    API --> REDIS
+    API --> FS
+```
+
+## Core Services
+
+### API Server
+- **Purpose**: Secure HTTPS API for ML experiments
+- **Port**: 9101 (HTTPS only)
+- **Auth**: API key authentication
+- **Security**: Rate limiting, IP whitelisting
+
+### Redis
+- **Purpose**: Caching and job queuing
+- **Port**: 6379 (localhost only)
+- **Storage**: Temporary data only
+- **Persistence**: Local volume
+
+### Zig CLI
+- **Purpose**: High-performance experiment management
+- **Language**: Zig for maximum speed and efficiency
+- **Features**:
+  - Content-addressed storage with deduplication
+  - SHA256-based commit ID generation
+  - WebSocket communication for real-time updates
+  - Rsync-based incremental file transfers
+  - Multi-threaded operations
+  - Secure API key authentication
+  - Auto-sync monitoring with file system watching
+  - Priority-based job queuing
+  - Memory-efficient operations with arena allocators
+
+## Security Architecture
+
+```mermaid
+graph LR
+    USER[User] --> AUTH[API Key Auth]
+    AUTH --> RATE[Rate Limiting]
+    RATE --> WHITELIST[IP Whitelist]
+    WHITELIST --> API[Secure API]
+    API --> AUDIT[Audit Logging]
+```
+
+### Security Layers
+1. **API Key Authentication** - Hashed keys with roles
+2. **Rate Limiting** - 30 requests/minute
+3. **IP Whitelisting** - Local networks only
+4. **Fail2Ban** - Automatic IP blocking
+5. **HTTPS/TLS** - Encrypted communication
+6. **Audit Logging** - Complete action tracking
+
+## Data Flow
+
+```mermaid
+sequenceDiagram
+    participant CLI
+    participant API
+    participant Redis
+    participant Storage
+    
+    CLI->>API: HTTPS Request
+    API->>API: Validate Auth
+    API->>Redis: Cache/Queue
+    API->>Storage: Experiment Data
+    Storage->>API: Results
+    API->>CLI: Response
+```
+
+## Deployment Options
+
+### Docker Compose (Recommended)
+```yaml
+services:
+  redis:
+    image: redis:7-alpine
+    ports: ["6379:6379"]
+    volumes: [redis_data:/data]
+    
+  api-server:
+    build: .
+    ports: ["9101:9101"]
+    depends_on: [redis]
+```
+
+### Local Setup
+```bash
+./setup.sh && ./manage.sh start
+```
+
+## Network Architecture
+
+- **Private Network**: Docker internal network
+- **Localhost Access**: Redis only on localhost
+- **HTTPS API**: Port 9101, TLS encrypted
+- **No External Dependencies**: Everything runs locally
+
+## Storage Architecture
+
+```
+data/
+├── experiments/     # ML experiment results
+├── cache/          # Temporary cache files
+└── backups/        # Local backups
+
+logs/
+├── app.log         # Application logs
+├── audit.log       # Security events
+└── access.log      # API access logs
+```
+
+## Monitoring Architecture
+
+Simple, lightweight monitoring:
+- **Health Checks**: Service availability
+- **Log Files**: Structured logging
+- **Basic Metrics**: Request counts, error rates
+- **Security Events**: Failed auth, rate limits
+
+## Homelab Benefits
+
+- ✅ **Simple Setup**: One-command installation
+- ✅ **Local Only**: No external dependencies
+- ✅ **Secure by Default**: HTTPS, auth, rate limiting
+- ✅ **Low Resource**: Minimal CPU/memory usage
+- ✅ **Easy Backup**: Local file system
+- ✅ **Privacy**: Everything stays on your network
+
+## High-Level Architecture
+
+```mermaid
+graph TB
+    subgraph "Client Layer"
+        CLI[CLI Tools]
+        TUI[Terminal UI]
+        API[REST API]
+    end
+    
+    subgraph "Authentication Layer"
+        Auth[Authentication Service]
+        RBAC[Role-Based Access Control]
+        Perm[Permission Manager]
+    end
+    
+    subgraph "Core Services"
+        Worker[ML Worker Service]
+        DataMgr[Data Manager Service]
+        Queue[Job Queue]
+    end
+    
+    subgraph "Storage Layer"
+        Redis[(Redis Cache)]
+        DB[(SQLite/PostgreSQL)]
+        Files[File Storage]
+    end
+    
+    subgraph "Container Runtime"
+        Podman[Podman/Docker]
+        Containers[ML Containers]
+    end
+    
+    CLI --> Auth
+    TUI --> Auth
+    API --> Auth
+    
+    Auth --> RBAC
+    RBAC --> Perm
+    
+    Worker --> Queue
+    Worker --> DataMgr
+    Worker --> Podman
+    
+    DataMgr --> DB
+    DataMgr --> Files
+    
+    Queue --> Redis
+    
+    Podman --> Containers
+```
+
+## Zig CLI Architecture
+
+### Component Structure
+
+```mermaid
+graph TB
+    subgraph "Zig CLI Components"
+        Main[main.zig] --> Commands[commands/]
+        Commands --> Config[config.zig]
+        Commands --> Utils[utils/]
+        Commands --> Net[net/]
+        Commands --> Errors[errors.zig]
+        
+        subgraph "Commands"
+            Init[init.zig]
+            Sync[sync.zig]
+            Queue[queue.zig]
+            Watch[watch.zig]
+            Status[status.zig]
+            Monitor[monitor.zig]
+            Cancel[cancel.zig]
+            Prune[prune.zig]
+        end
+        
+        subgraph "Utils"
+            Crypto[crypto.zig]
+            Storage[storage.zig]
+            Rsync[rsync.zig]
+        end
+        
+        subgraph "Network"
+            WS[ws.zig]
+        end
+    end
+```
+
+### Performance Optimizations
+
+#### Content-Addressed Storage
+- **Deduplication**: Files stored by SHA256 hash
+- **Space Efficiency**: Shared files across experiments
+- **Fast Lookup**: Hash-based file retrieval
+
+#### Memory Management
+- **Arena Allocators**: Efficient bulk allocation
+- **Zero-Copy Operations**: Minimized memory copying
+- **Automatic Cleanup**: Resource deallocation
+
+#### Network Communication
+- **WebSocket Protocol**: Real-time bidirectional communication
+- **Connection Pooling**: Reused connections
+- **Binary Messaging**: Efficient data transfer
+
+### Security Implementation
+
+```mermaid
+graph LR
+    subgraph "CLI Security"
+        Config[Config File] --> Hash[SHA256 Hashing]
+        Hash --> Auth[API Authentication]
+        Auth --> SSH[SSH Transfer]
+        SSH --> WS[WebSocket Security]
+    end
+```
+
+## Core Components
+
+### 1. Authentication & Authorization
+
+```mermaid
+graph LR
+    subgraph "Auth Flow"
+        Client[Client] --> APIKey[API Key]
+        APIKey --> Hash[Hash Validation]
+        Hash --> Roles[Role Resolution]
+        Roles --> Perms[Permission Check]
+        Perms --> Access[Grant/Deny Access]
+    end
+    
+    subgraph "Permission Sources"
+        YAML[YAML Config]
+        Inline[Inline Fallback]
+        Roles --> YAML
+        Roles --> Inline
+    end
+```
+
+**Features:**
+- API key-based authentication
+- Role-based access control (RBAC)
+- YAML-based permission configuration
+- Fallback to inline permissions
+- Admin wildcard permissions
+
+### 2. Worker Service
+
+```mermaid
+graph TB
+    subgraph "Worker Architecture"
+        API[HTTP API] --> Router[Request Router]
+        Router --> Auth[Auth Middleware]
+        Auth --> Queue[Job Queue]
+        Queue --> Processor[Job Processor]
+        Processor --> Runtime[Container Runtime]
+        Runtime --> Storage[Result Storage]
+        
+        subgraph "Job Lifecycle"
+            Submit[Submit Job] --> Queue
+            Queue --> Execute[Execute]
+            Execute --> Monitor[Monitor]
+            Monitor --> Complete[Complete]
+            Complete --> Store[Store Results]
+        end
+    end
+```
+
+**Responsibilities:**
+- HTTP API for job submission
+- Job queue management
+- Container orchestration
+- Result collection and storage
+- Metrics and monitoring
+
+### 3. Data Manager Service
+
+```mermaid
+graph TB
+    subgraph "Data Management"
+        API[Data API] --> Storage[Storage Layer]
+        Storage --> Metadata[Metadata DB]
+        Storage --> Files[File System]
+        Storage --> Cache[Redis Cache]
+        
+        subgraph "Data Operations"
+            Upload[Upload Data] --> Validate[Validate]
+            Validate --> Store[Store]
+            Store --> Index[Index]
+            Index --> Catalog[Catalog]
+        end
+    end
+```
+
+**Features:**
+- Data upload and validation
+- Metadata management
+- File system abstraction
+- Caching layer
+- Data catalog
+
+### 4. Terminal UI (TUI)
+
+```mermaid
+graph TB
+    subgraph "TUI Architecture"
+        UI[UI Components] --> Model[Data Model]
+        Model --> Update[Update Loop]
+        Update --> Render[Render]
+        
+        subgraph "UI Panels"
+            Jobs[Job List]
+            Details[Job Details]
+            Logs[Log Viewer]
+            Status[Status Bar]
+        end
+        
+        UI --> Jobs
+        UI --> Details
+        UI --> Logs
+        UI --> Status
+    end
+```
+
+**Components:**
+- Bubble Tea framework
+- Component-based architecture
+- Real-time updates
+- Keyboard navigation
+- Theme support
+
+## Data Flow
+
+### Job Execution Flow
+
+```mermaid
+sequenceDiagram
+    participant Client
+    participant Auth
+    participant Worker
+    participant Queue
+    participant Container
+    participant Storage
+    
+    Client->>Auth: Submit job with API key
+    Auth->>Client: Validate and return job ID
+    
+    Client->>Worker: Execute job request
+    Worker->>Queue: Queue job
+    Queue->>Worker: Job ready
+    Worker->>Container: Start ML container
+    Container->>Worker: Execute experiment
+    Worker->>Storage: Store results
+    Worker->>Client: Return results
+```
+
+### Authentication Flow
+
+```mermaid
+sequenceDiagram
+    participant Client
+    participant Auth
+    participant PermMgr
+    participant Config
+    
+    Client->>Auth: Request with API key
+    Auth->>Auth: Validate key hash
+    Auth->>PermMgr: Get user permissions
+    PermMgr->>Config: Load YAML permissions
+    Config->>PermMgr: Return permissions
+    PermMgr->>Auth: Return resolved permissions
+    Auth->>Client: Grant/deny access
+```
+
+## Security Architecture
+
+### Defense in Depth
+
+```mermaid
+graph TB
+    subgraph "Security Layers"
+        Network[Network Security]
+        Auth[Authentication]
+        AuthZ[Authorization]
+        Container[Container Security]
+        Data[Data Protection]
+        Audit[Audit Logging]
+    end
+    
+    Network --> Auth
+    Auth --> AuthZ
+    AuthZ --> Container
+    Container --> Data
+    Data --> Audit
+```
+
+**Security Features:**
+- API key authentication
+- Role-based permissions
+- Container isolation
+- File system sandboxing
+- Comprehensive audit logs
+- Input validation and sanitization
+
+### Container Security
+
+```mermaid
+graph TB
+    subgraph "Container Isolation"
+        Host[Host System]
+        Podman[Podman Runtime]
+        Network[Network Isolation]
+        FS[File System Isolation]
+        User[User Namespaces]
+        ML[ML Container]
+        
+        Host --> Podman
+        Podman --> Network
+        Podman --> FS
+        Podman --> User
+        User --> ML
+    end
+```
+
+**Isolation Features:**
+- Rootless containers
+- Network isolation
+- File system sandboxing
+- User namespace mapping
+- Resource limits
+
+## Configuration Architecture
+
+### Configuration Hierarchy
+
+```mermaid
+graph TB
+    subgraph "Config Sources"
+        Env[Environment Variables]
+        File[Config Files]
+        CLI[CLI Flags]
+        Defaults[Default Values]
+    end
+    
+    subgraph "Config Processing"
+        Merge[Config Merger]
+        Validate[Schema Validator]
+        Apply[Config Applier]
+    end
+    
+    Env --> Merge
+    File --> Merge
+    CLI --> Merge
+    Defaults --> Merge
+    
+    Merge --> Validate
+    Validate --> Apply
+```
+
+**Configuration Priority:**
+1. CLI flags (highest)
+2. Environment variables
+3. Configuration files
+4. Default values (lowest)
+
+## Scalability Architecture
+
+### Horizontal Scaling
+
+```mermaid
+graph TB
+    subgraph "Scaled Architecture"
+        LB[Load Balancer]
+        W1[Worker 1]
+        W2[Worker 2]
+        W3[Worker N]
+        Redis[Redis Cluster]
+        Storage[Shared Storage]
+        
+        LB --> W1
+        LB --> W2
+        LB --> W3
+        
+        W1 --> Redis
+        W2 --> Redis
+        W3 --> Redis
+        
+        W1 --> Storage
+        W2 --> Storage
+        W3 --> Storage
+    end
+```
+
+**Scaling Features:**
+- Stateless worker services
+- Shared job queue (Redis)
+- Distributed storage
+- Load balancer ready
+- Health checks and monitoring
+
+## Technology Stack
+
+### Backend Technologies
+
+| Component | Technology | Purpose |
+|-----------|------------|---------|
+| **Language** | Go 1.25+ | Core application |
+| **Web Framework** | Standard library | HTTP server |
+| **Authentication** | Custom | API key + RBAC |
+| **Database** | SQLite/PostgreSQL | Metadata storage |
+| **Cache** | Redis | Job queue & caching |
+| **Containers** | Podman/Docker | Job isolation |
+| **UI Framework** | Bubble Tea | Terminal UI |
+
+### Dependencies
+
+```go
+// Core dependencies
+require (
+    github.com/charmbracelet/bubbletea v1.3.10  // TUI framework
+    github.com/go-redis/redis/v8 v8.11.5        // Redis client
+    github.com/google/uuid v1.6.0               // UUID generation
+    github.com/mattn/go-sqlite3 v1.14.32        // SQLite driver
+    golang.org/x/crypto v0.45.0                 // Crypto utilities
+    gopkg.in/yaml.v3 v3.0.1                     // YAML parsing
+)
+```
+
+## Development Architecture
+
+### Project Structure
+
+```
+fetch_ml/
+├── cmd/                    # CLI applications
+│   ├── worker/            # ML worker service
+│   ├── tui/               # Terminal UI
+│   ├── data_manager/      # Data management
+│   └── user_manager/      # User management
+├── internal/              # Internal packages
+│   ├── auth/              # Authentication system
+│   ├── config/            # Configuration management
+│   ├── container/         # Container operations
+│   ├── database/          # Database operations
+│   ├── logging/           # Logging utilities
+│   ├── metrics/           # Metrics collection
+│   └── network/           # Network utilities
+├── configs/               # Configuration files
+├── scripts/               # Setup and utility scripts
+├── tests/                 # Test suites
+└── docs/                  # Documentation
+```
+
+### Package Dependencies
+
+```mermaid
+graph TB
+    subgraph "Application Layer"
+        Worker[cmd/worker]
+        TUI[cmd/tui]
+        DataMgr[cmd/data_manager]
+        UserMgr[cmd/user_manager]
+    end
+    
+    subgraph "Service Layer"
+        Auth[internal/auth]
+        Config[internal/config]
+        Container[internal/container]
+        Database[internal/database]
+    end
+    
+    subgraph "Utility Layer"
+        Logging[internal/logging]
+        Metrics[internal/metrics]
+        Network[internal/network]
+    end
+    
+    Worker --> Auth
+    Worker --> Config
+    Worker --> Container
+    TUI --> Auth
+    DataMgr --> Database
+    UserMgr --> Auth
+    
+    Auth --> Logging
+    Container --> Network
+    Database --> Metrics
+```
+
+## Monitoring & Observability
+
+### Metrics Collection
+
+```mermaid
+graph TB
+    subgraph "Metrics Pipeline"
+        App[Application] --> Metrics[Metrics Collector]
+        Metrics --> Export[Prometheus Exporter]
+        Export --> Prometheus[Prometheus Server]
+        Prometheus --> Grafana[Grafana Dashboard]
+        
+        subgraph "Metric Types"
+            Counter[Counters]
+            Gauge[Gauges]
+            Histogram[Histograms]
+            Timer[Timers]
+        end
+        
+        App --> Counter
+        App --> Gauge
+        App --> Histogram
+        App --> Timer
+    end
+```
+
+### Logging Architecture
+
+```mermaid
+graph TB
+    subgraph "Logging Pipeline"
+        App[Application] --> Logger[Structured Logger]
+        Logger --> File[File Output]
+        Logger --> Console[Console Output]
+        Logger --> Syslog[Syslog Forwarder]
+        Syslog --> Aggregator[Log Aggregator]
+        Aggregator --> Storage[Log Storage]
+        Storage --> Viewer[Log Viewer]
+    end
+```
+
+## Deployment Architecture
+
+### Container Deployment
+
+```mermaid
+graph TB
+    subgraph "Deployment Stack"
+        Image[Container Image]
+        Registry[Container Registry]
+        Orchestrator[Docker Compose]
+        Config[ConfigMaps/Secrets]
+        Storage[Persistent Storage]
+        
+        Image --> Registry
+        Registry --> Orchestrator
+        Config --> Orchestrator
+        Storage --> Orchestrator
+    end
+```
+
+### Service Discovery
+
+```mermaid
+graph TB
+    subgraph "Service Mesh"
+        Gateway[API Gateway]
+        Discovery[Service Discovery]
+        Worker[Worker Service]
+        Data[Data Service]
+        Redis[Redis Cluster]
+        
+        Gateway --> Discovery
+        Discovery --> Worker
+        Discovery --> Data
+        Discovery --> Redis
+    end
+```
+
+## Future Architecture Considerations
+
+### Microservices Evolution
+
+- **API Gateway**: Centralized routing and authentication
+- **Service Mesh**: Inter-service communication
+- **Event Streaming**: Kafka for job events
+- **Distributed Tracing**: OpenTelemetry integration
+- **Multi-tenant**: Tenant isolation and quotas
+
+### Homelab Features
+
+- **Docker Compose**: Simple container orchestration
+- **Local Development**: Easy setup and testing
+- **Security**: Built-in authentication and encryption
+- **Monitoring**: Basic health checks and logging
+
+---
+
+This architecture provides a solid foundation for secure, scalable machine learning experiments while maintaining simplicity and developer productivity.

docs/src/cicd.md

@@ -0,0 +1,165 @@
+---
+layout: page
+title: "CI/CD Pipeline"
+permalink: /cicd/
+nav_order: 5
+---
+
+# CI/CD Pipeline
+
+Automated testing, building, and releasing for fetch_ml.
+
+## Workflows
+
+### CI Workflow (`.github/workflows/ci.yml`)
+
+Runs on every push to `main`/`develop` and all pull requests.
+
+**Jobs:**
+1. **test** - Go backend tests with Redis
+2. **build** - Build all binaries (Go + Zig CLI)
+3. **test-scripts** - Validate deployment scripts
+4. **security-scan** - Trivy and Gosec security scans
+5. **docker-build** - Build and push Docker images (main branch only)
+
+**Test Coverage:**
+- Go unit tests with race detection
+- `internal/queue` package tests
+- Zig CLI tests  
+- Integration tests
+- Security audits
+
+### Release Workflow (`.github/workflows/release.yml`)
+
+Runs on version tags (e.g., `v1.0.0`).
+
+**Jobs:**
+
+1. **build-cli** (matrix build)
+   - Linux x86_64 (static musl)
+   - macOS x86_64
+   - macOS ARM64
+   - Downloads platform-specific static rsync
+   - Embeds rsync for zero-dependency releases
+
+2. **build-go-backends**
+   - Cross-platform Go builds
+   - api-server, worker, tui, data_manager, user_manager
+
+3. **create-release**
+   - Collects all artifacts
+   - Generates SHA256 checksums
+   - Creates GitHub release with notes
+
+## Release Process
+
+### Creating a Release
+
+```bash
+# 1. Update version
+git tag v1.0.0
+
+# 2. Push tag
+git push origin v1.0.0
+
+# 3. CI automatically builds and releases
+```
+
+### Release Artifacts
+
+**CLI Binaries (with embedded rsync):**
+- `ml-linux-x86_64.tar.gz` (~450-650KB)
+- `ml-macos-x86_64.tar.gz` (~450-650KB)
+- `ml-macos-arm64.tar.gz` (~450-650KB)
+
+**Go Backends:**
+- `fetch_ml_api-server.tar.gz`
+- `fetch_ml_worker.tar.gz`
+- `fetch_ml_tui.tar.gz`
+- `fetch_ml_data_manager.tar.gz`
+- `fetch_ml_user_manager.tar.gz`
+
+**Checksums:**
+- `checksums.txt` - Combined SHA256 sums
+- Individual `.sha256` files per binary
+
+## Development Workflow
+
+### Local Testing
+
+```bash
+# Run all tests
+make test
+
+# Run specific package tests
+go test ./internal/queue/...
+
+# Build CLI
+cd cli && zig build dev
+
+# Run formatters and linters
+make lint
+
+# Security scans are handled automatically in CI by the `security-scan` job
+```
+
+#### Optional heavy end-to-end tests
+
+Some e2e tests exercise full Docker deployments and performance scenarios and are
+**skipped by default** to keep local/CI runs fast. You can enable them explicitly
+with environment variables:
+
+```bash
+# Run Docker deployment e2e tests
+FETCH_ML_E2E_DOCKER=1 go test ./tests/e2e/...
+
+# Run performance-oriented e2e tests
+FETCH_ML_E2E_PERF=1 go test ./tests/e2e/...
+```
+
+Without these variables, `TestDockerDeploymentE2E` and `TestPerformanceE2E` will
+`t.Skip`, while all lighter e2e tests still run.
+
+### Pull Request Checks
+
+All PRs must pass:
+- ✅ Go tests (with Redis)
+- ✅ CLI tests
+- ✅ Security scans
+- ✅ Code linting
+- ✅ Build verification
+
+## Configuration
+
+### Environment Variables
+
+```yaml
+GO_VERSION: '1.25.0'
+ZIG_VERSION: '0.15.2'
+```
+
+### Secrets
+
+Required for releases:
+- `GITHUB_TOKEN` - Automatic, provided by GitHub Actions
+
+## Monitoring
+
+### Build Status
+
+Check workflow runs at:
+```
+https://github.com/jfraeys/fetch_ml/actions
+```
+
+### Artifacts
+
+Download build artifacts from:
+- Successful workflow runs (30-day retention)
+- GitHub Releases (permanent)
+
+---
+
+For implementation details:
+- [.github/workflows/ci.yml](https://github.com/jfraeys/fetch_ml/blob/main/.github/workflows/ci.yml)
+- [.github/workflows/release.yml](https://github.com/jfraeys/fetch_ml/blob/main/.github/workflows/release.yml)

docs/src/cli-reference.md

@@ -0,0 +1,404 @@
+---
+layout: page
+title: "CLI Reference"
+permalink: /cli-reference/
+nav_order: 2
+---
+
+# Fetch ML CLI Reference
+
+Comprehensive command-line tools for managing ML experiments in your homelab with Zig-based high-performance CLI.
+
+## Overview
+
+Fetch ML provides a comprehensive CLI toolkit built with performance and security in mind:
+
+- **Zig CLI** - High-performance experiment management written in Zig
+- **Go Commands** - API server, TUI, and data management utilities
+- **Management Scripts** - Service orchestration and deployment
+- **Setup Scripts** - One-command installation and configuration
+
+## Zig CLI (`./cli/zig-out/bin/ml`)
+
+High-performance command-line interface for experiment management, written in Zig for speed and efficiency.
+
+### Available Commands
+
+| Command | Description | Example |
+|---------|-------------|----------|
+| `init` | Interactive configuration setup | `ml init` |
+| `sync` | Sync project to worker with deduplication | `ml sync ./project --name myjob --queue` |
+| `queue` | Queue job for execution | `ml queue myjob --commit abc123 --priority 8` |
+| `status` | Get system and worker status | `ml status` |
+| `monitor` | Launch TUI monitoring via SSH | `ml monitor` |
+| `cancel` | Cancel running job | `ml cancel job123` |
+| `prune` | Clean up old experiments | `ml prune --keep 10` |
+| `watch` | Auto-sync directory on changes | `ml watch ./project --queue` |
+
+### Command Details
+
+#### `init` - Configuration Setup
+```bash
+ml init
+```
+Creates a configuration template at `~/.ml/config.toml` with:
+- Worker connection details
+- API authentication
+- Base paths and ports
+
+#### `sync` - Project Synchronization
+```bash
+# Basic sync
+ml sync ./my-project
+
+# Sync with custom name and queue
+ml sync ./my-project --name "experiment-1" --queue
+
+# Sync with priority
+ml sync ./my-project --priority 9
+```
+
+**Features:**
+- Content-addressed storage for deduplication
+- SHA256 commit ID generation
+- Rsync-based file transfer
+- Automatic queuing (with `--queue` flag)
+
+#### `queue` - Job Management
+```bash
+# Queue with commit ID
+ml queue my-job --commit abc123def456
+
+# Queue with priority (1-10, default 5)
+ml queue my-job --commit abc123 --priority 8
+```
+
+**Features:**
+- WebSocket-based communication
+- Priority queuing system
+- API key authentication
+
+#### `watch` - Auto-Sync Monitoring
+```bash
+# Watch directory for changes
+ml watch ./project
+
+# Watch and auto-queue on changes
+ml watch ./project --name "dev-exp" --queue
+```
+
+**Features:**
+- Real-time file system monitoring
+- Automatic re-sync on changes
+- Configurable polling interval (2 seconds)
+- Commit ID comparison for efficiency
+
+#### `prune` - Cleanup Management
+```bash
+# Keep last N experiments
+ml prune --keep 20
+
+# Remove experiments older than N days
+ml prune --older-than 30
+```
+
+#### `monitor` - Remote Monitoring
+```bash
+ml monitor
+```
+Launches TUI interface via SSH for real-time monitoring.
+
+#### `cancel` - Job Cancellation
+```bash
+ml cancel running-job-id
+```
+Cancels currently running jobs by ID.
+
+### Configuration
+
+The Zig CLI reads configuration from `~/.ml/config.toml`:
+
+```toml
+worker_host = "worker.local"
+worker_user = "mluser"
+worker_base = "/data/ml-experiments"
+worker_port = 22
+api_key = "your-api-key"
+```
+
+### Performance Features
+
+- **Content-Addressed Storage**: Automatic deduplication of identical files
+- **Incremental Sync**: Only transfers changed files
+- **SHA256 Hashing**: Reliable commit ID generation
+- **WebSocket Communication**: Efficient real-time messaging
+- **Multi-threaded**: Concurrent operations where applicable
+
+## Go Commands
+
+### API Server (`./cmd/api-server/main.go`)
+Main HTTPS API server for experiment management.
+
+```bash
+# Build and run
+go run ./cmd/api-server/main.go
+
+# With configuration
+./bin/api-server --config configs/config-local.yaml
+```
+
+**Features:**
+- HTTPS-only communication
+- API key authentication
+- Rate limiting and IP whitelisting
+- WebSocket support for real-time updates
+- Redis integration for caching
+
+### TUI (`./cmd/tui/main.go`)
+Terminal User Interface for monitoring experiments.
+
+```bash
+# Launch TUI
+go run ./cmd/tui/main.go
+
+# With custom config
+./tui --config configs/config-local.yaml
+```
+
+**Features:**
+- Real-time experiment monitoring
+- Interactive job management
+- Status visualization
+- Log viewing
+
+### Data Manager (`./cmd/data_manager/`)
+Utilities for data synchronization and management.
+
+```bash
+# Sync data
+./data_manager --sync ./data
+
+# Clean old data
+./data_manager --cleanup --older-than 30d
+```
+
+### Config Lint (`./cmd/configlint/main.go`)
+Configuration validation and linting tool.
+
+```bash
+# Validate configuration
+./configlint configs/config-local.yaml
+
+# Check schema compliance
+./configlint --schema configs/schema/config_schema.yaml
+```
+
+## Management Script (`./tools/manage.sh`)
+
+Simple service management for your homelab.
+
+### Commands
+```bash
+./tools/manage.sh start          # Start all services
+./tools/manage.sh stop           # Stop all services
+./tools/manage.sh status         # Check service status
+./tools/manage.sh logs           # View logs
+./tools/manage.sh monitor        # Basic monitoring
+./tools/manage.sh security       # Security status
+./tools/manage.sh cleanup        # Clean project artifacts
+```
+
+## Setup Script (`./setup.sh`)
+
+One-command homelab setup.
+
+### Usage
+```bash
+# Full setup
+./setup.sh
+
+# Setup includes:
+# - SSL certificate generation
+# - Configuration creation
+# - Build all components
+# - Start Redis
+# - Setup Fail2Ban (if available)
+```
+
+## API Testing
+
+Test the API with curl:
+
+```bash
+# Health check
+curl -k -H 'X-API-Key: password' https://localhost:9101/health
+
+# List experiments
+curl -k -H 'X-API-Key: password' https://localhost:9101/experiments
+
+# Submit experiment
+curl -k -X POST -H 'X-API-Key: password' \
+     -H 'Content-Type: application/json' \
+     -d '{"name":"test","config":{"type":"basic"}}' \
+     https://localhost:9101/experiments
+```
+
+## Zig CLI Architecture
+
+The Zig CLI is designed for performance and reliability:
+
+### Core Components
+- **Commands** (`cli/src/commands/`): Individual command implementations
+- **Config** (`cli/src/config.zig`): Configuration management
+- **Network** (`cli/src/net/ws.zig`): WebSocket client implementation
+- **Utils** (`cli/src/utils/`): Cryptography, storage, and rsync utilities
+- **Errors** (`cli/src/errors.zig`): Centralized error handling
+
+### Performance Optimizations
+- **Content-Addressed Storage**: Deduplicates identical files across experiments
+- **SHA256 Hashing**: Fast, reliable commit ID generation
+- **Rsync Integration**: Efficient incremental file transfers
+- **WebSocket Protocol**: Low-latency communication with worker
+- **Memory Management**: Efficient allocation with Zig's allocator system
+
+### Security Features
+- **API Key Hashing**: Secure authentication token handling
+- **SSH Integration**: Secure file transfers
+- **Input Validation**: Comprehensive argument checking
+- **Error Handling**: Secure error reporting without information leakage
+
+## Configuration
+
+Main configuration file: `configs/config-local.yaml`
+
+### Key Settings
+```yaml
+auth:
+  enabled: true
+  api_keys:
+    homelab_user:
+      hash: "5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8"
+      admin: true
+
+server:
+  address: ":9101"
+  tls:
+    enabled: true
+    cert_file: "./ssl/cert.pem"
+    key_file: "./ssl/key.pem"
+
+security:
+  rate_limit:
+    enabled: true
+    requests_per_minute: 30
+  ip_whitelist:
+    - "127.0.0.1"
+    - "::1"
+    - "192.168.0.0/16"
+    - "10.0.0.0/8"
+```
+
+## Docker Commands
+
+If using Docker Compose:
+
+```bash
+# Start services
+docker-compose up -d (testing only)
+
+# View logs
+docker-compose logs -f
+
+# Stop services
+docker-compose down
+
+# Check status
+docker-compose ps
+```
+
+## Troubleshooting
+
+### Common Issues
+
+**Zig CLI not found:**
+```bash
+# Build the CLI
+cd cli && make build
+
+# Check binary exists
+ls -la ./cli/zig-out/bin/ml
+```
+
+**Configuration not found:**
+```bash
+# Create configuration
+./cli/zig-out/bin/ml init
+
+# Check config file
+ls -la ~/.ml/config.toml
+```
+
+**Worker connection failed:**
+```bash
+# Test SSH connection
+ssh -p 22 mluser@worker.local
+
+# Check configuration
+cat ~/.ml/config.toml
+```
+
+**Sync not working:**
+```bash
+# Check rsync availability
+rsync --version
+
+# Test manual sync
+rsync -avz ./project/ mluser@worker.local:/tmp/test/
+```
+
+**WebSocket connection failed:**
+```bash
+# Check worker WebSocket port
+telnet worker.local 9100
+
+# Verify API key
+./cli/zig-out/bin/ml status
+```
+
+**API not responding:**
+```bash
+./tools/manage.sh status
+./tools/manage.sh logs
+```
+
+**Authentication failed:**
+```bash
+# Check API key in config-local.yaml
+grep -A 5 "api_keys:" configs/config-local.yaml
+```
+
+**Redis connection failed:**
+```bash
+# Check Redis status
+redis-cli ping
+
+# Start Redis
+redis-server
+```
+
+### Getting Help
+
+```bash
+# CLI help
+./cli/zig-out/bin/ml help
+
+# Management script help
+./tools/manage.sh help
+
+# Check all available commands
+make help
+```
+
+---
+
+**That's it for the CLI reference!** For complete setup instructions, see the main [index](index.md).

docs/src/configuration-schema.md

@@ -0,0 +1,55 @@
+# Configuration Schema
+
+Complete reference for Fetch ML configuration options.
+
+## Configuration File Structure
+
+Fetch ML uses YAML configuration files. The main configuration file is typically `config.yaml`.
+
+### Full Schema
+
+```yaml
+# Server Configuration
+server:
+  address: ":9101"
+  tls:
+    enabled: false
+    cert_file: ""
+    key_file: ""
+
+# Database Configuration
+database:
+  type: "sqlite"  # sqlite, postgres, mysql
+  connection: "fetch_ml.db"
+  host: "localhost"
+  port: 5432
+  username: "postgres"
+  password: ""
+  database: "fetch_ml"
+
+# Redis Configuration
+
+
+## Quick Reference
+
+### Database Types
+- **SQLite**: `type: sqlite, connection: file.db`
+- **PostgreSQL**: `type: postgres, host: localhost, port: 5432`
+
+### Key Settings
+- `server.address: :9101`
+- `database.type: sqlite`
+- `redis.addr: localhost:6379`
+- `auth.enabled: true`
+- `logging.level: info`
+
+### Environment Override
+```bash
+export FETCHML_SERVER_ADDRESS=:8080
+export FETCHML_DATABASE_TYPE=postgres
+```
+
+### Validation
+```bash
+make configlint
+``` 

docs/src/deployment.md

@@ -0,0 +1,288 @@
+# ML Experiment Manager - Deployment Guide
+
+## Overview
+
+The ML Experiment Manager supports multiple deployment methods from local development to homelab Docker setups.
+
+## Quick Start
+
+### Docker Compose (Recommended for Development)
+
+```bash
+# Clone repository
+git clone https://github.com/your-org/fetch_ml.git
+cd fetch_ml
+
+# Start all services
+docker-compose up -d (testing only)
+
+# Check status
+docker-compose ps
+
+# View logs
+docker-compose logs -f api-server
+```
+
+Access the API at `http://localhost:9100`
+
+## Deployment Options
+
+### 1. Local Development
+
+#### Prerequisites
+
+**Container Runtimes:**
+- **Docker Compose**: For testing and development only
+- **Podman**: For production experiment execution
+- Go 1.25+
+- Zig 0.15.2
+- Redis 7+
+- Docker & Docker Compose (optional)
+
+#### Manual Setup
+```bash
+# Start Redis
+redis-server
+
+# Build and run Go server
+go build -o bin/api-server ./cmd/api-server
+./bin/api-server -config configs/config-local.yaml
+
+# Build Zig CLI
+cd cli
+zig build prod
+./zig-out/bin/ml --help
+```
+
+### 2. Docker Deployment
+
+#### Build Image
+```bash
+docker build -t ml-experiment-manager:latest .
+```
+
+#### Run Container
+```bash
+docker run -d \
+  --name ml-api \
+  -p 9100:9100 \
+  -p 9101:9101 \
+  -v $(pwd)/configs:/app/configs:ro \
+  -v experiment-data:/data/ml-experiments \
+  ml-experiment-manager:latest
+```
+
+#### Docker Compose
+```bash
+# Production mode
+docker-compose -f docker-compose.yml up -d
+
+# Development mode with logs
+docker-compose -f docker-compose.yml up
+```
+
+### 3. Homelab Setup
+
+```bash
+# Use the simple setup script
+./setup.sh
+
+# Or manually with Docker Compose
+docker-compose up -d (testing only)
+```
+
+### 4. Cloud Deployment
+
+#### AWS ECS
+```bash
+# Build and push to ECR
+aws ecr get-login-password | docker login --username AWS --password-stdin $ECR_REGISTRY
+docker build -t $ECR_REGISTRY/ml-experiment-manager:latest .
+docker push $ECR_REGISTRY/ml-experiment-manager:latest
+
+# Deploy with ECS CLI
+ecs-cli compose --project-name ml-experiment-manager up
+```
+
+#### Google Cloud Run
+```bash
+# Build and push
+gcloud builds submit --tag gcr.io/$PROJECT_ID/ml-experiment-manager
+
+# Deploy
+gcloud run deploy ml-experiment-manager \
+  --image gcr.io/$PROJECT_ID/ml-experiment-manager \
+  --platform managed \
+  --region us-central1 \
+  --allow-unauthenticated
+```
+
+## Configuration
+
+### Environment Variables
+```yaml
+# configs/config-local.yaml
+base_path: "/data/ml-experiments"
+auth:
+  enabled: true
+  api_keys:
+    - "your-production-api-key"
+server:
+  address: ":9100"
+  tls:
+    enabled: true
+    cert_file: "/app/ssl/cert.pem"
+    key_file: "/app/ssl/key.pem"
+```
+
+### Docker Compose Environment
+```yaml
+# docker-compose.yml
+version: '3.8'
+services:
+  api-server:
+    environment:
+      - REDIS_URL=redis://redis:6379
+      - LOG_LEVEL=info
+    volumes:
+      - ./configs:/configs:ro
+      - ./data:/data/experiments
+```
+
+## Monitoring & Logging
+
+### Health Checks
+- HTTP: `GET /health`
+- WebSocket: Connection test
+- Redis: Ping check
+
+### Metrics
+- Prometheus metrics at `/metrics`
+- Custom application metrics
+- Container resource usage
+
+### Logging
+- Structured JSON logging
+- Log levels: DEBUG, INFO, WARN, ERROR
+- Centralized logging via ELK stack
+
+## Security
+
+### TLS Configuration
+```bash
+# Generate self-signed cert (development)
+openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes
+
+# Production - use Let's Encrypt
+certbot certonly --standalone -d ml-experiments.example.com
+```
+
+### Network Security
+- Firewall rules (ports 9100, 9101, 6379)
+- VPN access for internal services
+- API key authentication
+- Rate limiting
+
+## Performance Tuning
+
+### Resource Allocation
+```yaml
+resources:
+  requests:
+    memory: "256Mi"
+    cpu: "250m"
+  limits:
+    memory: "1Gi"
+    cpu: "1000m"
+```
+
+### Scaling Strategies
+- Horizontal pod autoscaling
+- Redis clustering
+- Load balancing
+- CDN for static assets
+
+## Backup & Recovery
+
+### Data Backup
+```bash
+# Backup experiment data
+docker-compose exec redis redis-cli BGSAVE
+docker cp $(docker-compose ps -q redis):/data/dump.rdb ./redis-backup.rdb
+
+# Backup data volume
+docker run --rm -v ml-experiments_redis_data:/data -v $(pwd):/backup alpine tar czf /backup/redis-backup.tar.gz -C /data .
+```
+
+### Disaster Recovery
+1. Restore Redis data
+2. Restart services
+3. Verify experiment metadata
+4. Test API endpoints
+
+## Troubleshooting
+
+### Common Issues
+
+#### API Server Not Starting
+```bash
+# Check logs
+docker-compose logs api-server
+
+# Check configuration
+cat configs/config-local.yaml
+
+# Check Redis connection
+docker-compose exec redis redis-cli ping
+```
+
+#### WebSocket Connection Issues
+```bash
+# Test WebSocket
+wscat -c ws://localhost:9100/ws
+
+# Check TLS
+openssl s_client -connect localhost:9101 -servername localhost
+```
+
+#### Performance Issues
+```bash
+# Check resource usage
+docker-compose exec api-server ps aux
+
+# Check Redis memory
+docker-compose exec redis redis-cli info memory
+```
+
+### Debug Mode
+```bash
+# Enable debug logging
+export LOG_LEVEL=debug
+./bin/api-server -config configs/config-local.yaml
+```
+
+## CI/CD Integration
+
+### GitHub Actions
+- Automated testing on PR
+- Multi-platform builds
+- Security scanning
+- Automatic releases
+
+### Deployment Pipeline
+1. Code commit → GitHub
+2. CI/CD pipeline triggers
+3. Build and test
+4. Security scan
+5. Deploy to staging
+6. Run integration tests
+7. Deploy to production
+8. Post-deployment verification
+
+## Support
+
+For deployment issues:
+1. Check this guide
+2. Review logs
+3. Check GitHub Issues
+4. Contact maintainers

docs/src/development-setup.md

@@ -0,0 +1,55 @@
+# Development Setup
+
+Set up your local development environment for Fetch ML.
+
+## Prerequisites
+
+**Container Runtimes:**
+- **Docker Compose**: For testing and development only
+- **Podman**: For production experiment execution
+
+- Go 1.21+
+- Zig 0.11+
+- Docker Compose (testing only)
+- Redis (or use Docker)
+- Git
+
+## Quick Setup
+
+```bash
+# Clone repository
+git clone https://github.com/jfraeys/fetch_ml.git
+cd fetch_ml
+
+# Start dependencies
+see [Quick Start](quick-start.md) for Docker setup redis postgres
+
+# Build all components
+make build
+
+# Run tests
+see [Testing Guide](testing.md)
+```
+
+## Detailed Setup
+
+
+## Quick Start
+```bash
+git clone https://github.com/jfraeys/fetch_ml.git
+cd fetch_ml
+see [Quick Start](quick-start.md) for Docker setup
+make build
+see [Testing Guide](testing.md)
+```
+
+## Key Commands
+- `make build` - Build all components
+- `see [Testing Guide](testing.md)` - Run tests
+- `make dev` - Development build
+- `see [CLI Reference](cli-reference.md) and [Zig CLI](zig-cli.md)` - Build CLI
+
+## Common Issues
+- Build fails: `go mod tidy`
+- Zig errors: `cd cli && rm -rf zig-out zig-cache`
+- Port conflicts: `lsof -i :9101`

docs/src/environment-variables.md

@@ -0,0 +1,112 @@
+# Environment Variables
+
+Fetch ML supports environment variables for configuration, allowing you to override config file settings and deploy in different environments.
+
+## Priority Order
+
+1. Environment variables (highest priority)
+2. Configuration file values
+3. Default values (lowest priority)
+
+## Variable Prefixes
+
+### General Configuration
+- `FETCH_ML_*` - General server and application settings
+
+### CLI Configuration  
+- `FETCH_ML_CLI_*` - CLI-specific settings (overrides `~/.ml/config.toml`)
+
+### TUI Configuration
+- `FETCH_ML_TUI_*` - TUI-specific settings (overrides TUI config file)
+
+## CLI Environment Variables
+
+| Variable | Config Field | Example |
+|----------|-------------|---------|
+| `FETCH_ML_CLI_HOST` | `worker_host` | `localhost` |
+| `FETCH_ML_CLI_USER` | `worker_user` | `mluser` |
+| `FETCH_ML_CLI_BASE` | `worker_base` | `/opt/ml` |
+| `FETCH_ML_CLI_PORT` | `worker_port` | `22` |
+| `FETCH_ML_CLI_API_KEY` | `api_key` | `your-api-key-here` |
+
+## TUI Environment Variables
+
+| Variable | Config Field | Example |
+|----------|-------------|---------|
+| `FETCH_ML_TUI_HOST` | `host` | `localhost` |
+| `FETCH_ML_TUI_USER` | `user` | `mluser` |
+| `FETCH_ML_TUI_SSH_KEY` | `ssh_key` | `~/.ssh/id_rsa` |
+| `FETCH_ML_TUI_PORT` | `port` | `22` |
+| `FETCH_ML_TUI_BASE_PATH` | `base_path` | `/opt/ml` |
+| `FETCH_ML_TUI_TRAIN_SCRIPT` | `train_script` | `train.py` |
+| `FETCH_ML_TUI_REDIS_ADDR` | `redis_addr` | `localhost:6379` |
+| `FETCH_ML_TUI_REDIS_PASSWORD` | `redis_password` | `` |
+| `FETCH_ML_TUI_REDIS_DB` | `redis_db` | `0` |
+| `FETCH_ML_TUI_KNOWN_HOSTS` | `known_hosts` | `~/.ssh/known_hosts` |
+
+## Server Environment Variables (Auth & Debug)
+
+These variables control **server-side** authentication behavior and are intended **only for local development and debugging**.
+
+| Variable | Purpose | Allowed In Production? |
+|----------|---------|------------------------|
+| `FETCH_ML_ALLOW_INSECURE_AUTH` | When set to `1` *and* `FETCH_ML_DEBUG=1`, allows the API server to run with `auth.enabled: false` by injecting a default admin user. | **No**. Must never be set in production. |
+| `FETCH_ML_DEBUG` | Enables additional debug behaviors. Required (set to `1`) to activate the insecure auth bypass above. | **No**. Must never be set in production. |
+
+When both variables are set to `1` and `auth.enabled` is `false`, the server logs a clear warning and treats all requests as coming from a default admin user. This mode is convenient for local homelab experiments but is **insecure by design** and must not be used on any shared or internet-facing environment.
+
+## Usage Examples
+
+### Development Environment
+```bash
+export FETCH_ML_CLI_HOST=localhost
+export FETCH_ML_CLI_USER=devuser
+export FETCH_ML_CLI_API_KEY=dev-key-123456789012
+./ml status
+```
+
+### Production Environment
+```bash
+export FETCH_ML_CLI_HOST=prod-server.example.com
+export FETCH_ML_CLI_USER=mluser
+export FETCH_ML_CLI_API_KEY=prod-key-abcdef1234567890
+./ml status
+```
+
+### Docker/Kubernetes
+```yaml
+env:
+  - name: FETCH_ML_CLI_HOST
+    value: "ml-server.internal"
+  - name: FETCH_ML_CLI_USER
+    value: "mluser"
+  - name: FETCH_ML_CLI_API_KEY
+    valueFrom:
+      secretKeyRef:
+        name: ml-secrets
+        key: api-key
+```
+
+### Using .env file
+```bash
+# Copy the example file
+cp .env.example .env
+
+# Edit with your values
+vim .env
+
+# Load in your shell
+export $(cat .env | xargs)
+```
+
+## Backward Compatibility
+
+The CLI also supports the legacy `ML_*` prefix for backward compatibility, but `FETCH_ML_CLI_*` takes priority if both are set.
+
+| Legacy Variable | New Variable |
+|-----------------|--------------|
+| `ML_HOST` | `FETCH_ML_CLI_HOST` |
+| `ML_USER` | `FETCH_ML_CLI_USER` |
+| `ML_BASE` | `FETCH_ML_CLI_BASE` |
+| `ML_PORT` | `FETCH_ML_CLI_PORT` |
+| `ML_API_KEY` | `FETCH_ML_CLI_API_KEY` |

docs/src/first-experiment.md

@@ -0,0 +1,235 @@
+# First Experiment
+
+Run your first machine learning experiment with Fetch ML.
+
+## Prerequisites
+
+**Container Runtimes:**
+- **Docker Compose**: For testing and development only
+- **Podman**: For production experiment execution
+
+- Fetch ML installed and running
+- API key (see [Security](security.md) and [API Key Process](api-key-process.md))
+- Basic ML knowledge
+
+## Experiment Workflow
+
+### 1. Prepare Your ML Code
+
+Create a simple Python script:
+
+```python
+# experiment.py
+import argparse
+import json
+import sys
+import time
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--epochs', type=int, default=10)
+    parser.add_argument('--lr', type=float, default=0.001)
+    parser.add_argument('--output', default='results.json')
+    
+    args = parser.parse_args()
+    
+    # Simulate training
+    results = {
+        'epochs': args.epochs,
+        'learning_rate': args.lr,
+        'accuracy': 0.85 + (args.lr * 0.1),
+        'loss': 0.5 - (args.epochs * 0.01),
+        'training_time': args.epochs * 0.1
+    }
+    
+    # Save results
+    with open(args.output, 'w') as f:
+        json.dump(results, f, indent=2)
+    
+    print(f"Training completed: {results}")
+    return results
+
+if __name__ == '__main__':
+    main()
+```
+
+### 2. Submit Job via API
+
+```bash
+# Submit experiment
+curl -X POST http://localhost:9101/api/v1/jobs \
+  -H "Content-Type: application/json" \
+  -H "X-API-Key: your-api-key" \
+  -d '{
+    "job_name": "first-experiment",
+    "args": "--epochs 20 --lr 0.01 --output experiment_results.json",
+    "priority": 1,
+    "metadata": {
+      "experiment_type": "training",
+      "dataset": "sample_data"
+    }
+  }'
+```
+
+### 3. Monitor Progress
+
+```bash
+# Check job status
+curl -H "X-API-Key: your-api-key" \
+  http://localhost:9101/api/v1/jobs/first-experiment
+
+# List all jobs
+curl -H "X-API-Key: your-api-key" \
+  http://localhost:9101/api/v1/jobs
+
+# Get job metrics
+curl -H "X-API-Key: your-api-key" \
+  http://localhost:9101/api/v1/jobs/first-experiment/metrics
+```
+
+### 4. Use CLI
+
+```bash
+# Submit with CLI
+cd cli && zig build dev
+./cli/zig-out/dev/ml submit \
+  --name "cli-experiment" \
+  --args "--epochs 15 --lr 0.005" \
+  --server http://localhost:9101
+
+# Monitor with CLI
+./cli/zig-out/dev/ml list-jobs --server http://localhost:9101
+./cli/zig-out/dev/ml job-status cli-experiment --server http://localhost:9101
+```
+
+## Advanced Experiment
+
+### Hyperparameter Tuning
+
+```bash
+# Submit multiple experiments
+for lr in 0.001 0.01 0.1; do
+  curl -X POST http://localhost:9101/api/v1/jobs \
+    -H "Content-Type: application/json" \
+    -H "X-API-Key: your-api-key" \
+    -d "{
+      \"job_name\": \"tune-lr-$lr\",
+      \"args\": \"--epochs 10 --lr $lr\",
+      \"metadata\": {\"learning_rate\": $lr}
+    }"
+done
+```
+
+### Batch Processing
+
+```bash
+# Submit batch job
+curl -X POST http://localhost:9101/api/v1/jobs \
+  -H "Content-Type: application/json" \
+  -H "X-API-Key: your-api-key" \
+  -d '{
+    "job_name": "batch-processing",
+    "args": "--input data/ --output results/ --batch-size 32",
+    "priority": 2,
+    "datasets": ["training_data", "validation_data"]
+  }'
+```
+
+## Results and Output
+
+### Access Results
+
+```bash
+# Download results
+curl -H "X-API-Key: your-api-key" \
+  http://localhost:9101/api/v1/jobs/first-experiment/results
+
+# View job details
+curl -H "X-API-Key: your-api-key" \
+  http://localhost:9101/api/v1/jobs/first-experiment | jq .
+```
+
+### Result Format
+
+```json
+{
+  "job_id": "first-experiment",
+  "status": "completed",
+  "results": {
+    "epochs": 20,
+    "learning_rate": 0.01,
+    "accuracy": 0.86,
+    "loss": 0.3,
+    "training_time": 2.0
+  },
+  "metrics": {
+    "gpu_utilization": "85%",
+    "memory_usage": "2GB",
+    "execution_time": "120s"
+  }
+}
+```
+
+## Best Practices
+
+### Job Naming
+
+- Use descriptive names: `model-training-v2`, `data-preprocessing`
+- Include version numbers: `experiment-v1`, `experiment-v2`
+- Add timestamps: `daily-batch-2024-01-15`
+
+### Metadata Usage
+
+```json
+{
+  "metadata": {
+    "experiment_type": "training",
+    "model_version": "v2.1",
+    "dataset": "imagenet-2024",
+    "environment": "gpu",
+    "team": "ml-team"
+  }
+}
+```
+
+### Error Handling
+
+```bash
+# Check failed jobs
+curl -H "X-API-Key: your-api-key" \
+  "http://localhost:9101/api/v1/jobs?status=failed"
+
+# Retry failed job
+curl -X POST http://localhost:9101/api/v1/jobs \
+  -H "Content-Type: application/json" \
+  -H "X-API-Key: your-api-key" \
+  -d '{
+    "job_name": "retry-experiment",
+    "args": "--epochs 20 --lr 0.01",
+    "metadata": {"retry_of": "first-experiment"}
+  }'
+```
+
+## ## Related Documentation
+
+- [Development Setup (see [Development Setup](development-setup.md))](development-setup.md) - Local development environment
+- [Testing Guide (see [Testing Guide](testing.md))](testing.md) - Test your experiments
+- [Production Deployment (see [Deployment](deployment.md))](deployment.md) - Scale to production
+- [Monitoring](production-monitoring.md) - Track experiment performance
+
+## Troubleshooting
+
+**Job stuck in pending?**
+- Check worker status: `curl /api/v1/workers`
+- Verify resources: `docker stats`
+- Check logs: `docker-compose logs api-server`
+
+**Job failed?**
+- Check error message: `curl /api/v1/jobs/job-id`
+- Review job arguments
+- Verify input data
+
+**No results?**
+- Check job completion status
+- Verify output file paths
+- Check storage permissions

docs/src/index.md

@@ -0,0 +1,89 @@
+---
+layout: default
+title: Fetch ML Documentation
+---
+
+# Fetch ML - Secure Machine Learning Platform
+
+A secure, containerized platform for running machine learning experiments with role-based access control and comprehensive audit trails.
+
+## Quick Start
+
+**New to the project?** Start here!
+
+```bash
+# Clone the repository
+git clone https://github.com/your-username/fetch_ml.git
+cd fetch_ml
+
+# Quick setup (builds everything, creates test user)
+make quick-start
+
+# Create your API key
+./bin/user_manager --config configs/config_dev.yaml --cmd generate-key --username your_name --role data_scientist
+
+# Run your first experiment
+./bin/worker --config configs/config_dev.yaml --api-key YOUR_GENERATED_KEY
+```
+
+## Quick Navigation
+
+### 🚀 Getting Started
+- [**Getting Started Guide**](quick-start.md) - Complete setup instructions
+- [**Simple Install**](installation.md) - Quick installation guide
+
+### 🔒 Security & Authentication
+- [**Security Overview**](security.md) - Security best practices
+- [**API Key Process**](api-key-process.md) - Generate and manage API keys
+- [**User Permissions**](user-permissions.md) - Role-based access control
+
+### ⚙️ Configuration
+- [**Environment Variables**](environment-variables.md) - Configuration options
+- [**Smart Defaults**](smart-defaults.md) - Default configuration settings
+
+### 🛠️ Development
+- [**Architecture**](architecture.md) - System architecture and design
+- [**CLI Reference**](cli-reference.md) - Command-line interface documentation
+- [**Testing Guide**](testing.md) - Testing procedures and guidelines
+- [**Queue System**](queue.md) - Job queue implementation
+
+### 🏭 Production Deployment
+- [**Deployment Guide**](deployment.md) - Production deployment instructions
+- [**Production Monitoring**](production-monitoring.md) - Monitoring and observability
+- [**Operations Guide**](operations.md) - Production operations
+
+## Features
+
+- **🔐 Secure Authentication** - RBAC with API keys, roles, and permissions
+- **🐳 Containerized** - Podman-based secure execution environments
+- **🗄️ Database Storage** - SQLite backend for user management (optional)
+- **📋 Audit Trail** - Complete logging of all actions
+- **🚀 Production Ready** - Security audits, systemd services, log rotation
+
+## Available Commands
+
+```bash
+# Core commands
+make help                    # See all available commands
+make build                   # Build all binaries
+make test-unit              # Run tests
+
+# User management
+./bin/user_manager --config configs/config_dev.yaml --cmd generate-key --username new_user --role data_scientist
+./bin/user_manager --config configs/config_dev.yaml --cmd list-users
+
+# Run services
+./bin/worker --config configs/config_dev.yaml --api-key YOUR_KEY
+./bin/tui --config configs/config_dev.yaml
+./bin/data_manager --config configs/config_dev.yaml
+```
+
+## Need Help?
+
+- **📖 Documentation**: Use the navigation menu on the left
+- **⚡ Quick help**: `make help`
+- **🧪 Tests**: `make test-unit`
+
+---
+
+**Happy ML experimenting!**

docs/src/installation.md

@@ -0,0 +1,63 @@
+# Simple Installation Guide
+
+## Quick Start (5 minutes)
+
+```bash
+# 1. Install
+git clone https://github.com/jfraeys/fetch_ml.git
+cd fetch_ml
+make install
+
+# 2. Setup (auto-configures)
+./bin/ml setup
+
+# 3. Run experiments
+./bin/ml run my-experiment.py
+```
+
+That's it. Everything else is optional.
+
+---
+
+## What If I Want More Control?
+
+### Manual Configuration (Optional)
+```bash
+# Edit settings if defaults don't work
+nano ~/.ml/config.toml
+```
+
+### Monitoring Dashboard (Optional)
+```bash
+# Real-time monitoring
+./bin/tui
+```
+
+---
+
+## Senior Developer Feedback
+
+**"Keep it simple"** - Most data scientists want:
+1. One installation command
+2. Sensible defaults  
+3. Works without configuration
+4. Advanced features available when needed
+
+**Current plan is too complex** because it asks users to decide between:
+- CLI vs TUI vs Both
+- Zig vs Go build tools
+- Manual vs auto config
+- Multiple environment variables
+
+**Better approach:** Start simple, add complexity gradually.
+
+---
+
+## Recommended Simplified Workflow
+
+1. **Single Binary** - Combine CLI + basic TUI functionality
+2. **Auto-Discovery** - Detect common ML environments automatically  
+3. **Progressive Disclosure** - Show advanced options only when needed
+4. **Zero Config** - Work out-of-the-box with localhost defaults
+
+The goal: "It just works" for 80% of use cases.

docs/src/operations.md

@@ -0,0 +1,310 @@
+---
+layout: page
+title: "Operations Runbook"
+permalink: /operations/
+nav_order: 6
+---
+
+# Operations Runbook
+
+Operational guide for troubleshooting and maintaining the ML experiment system.
+
+## Task Queue Operations
+
+### Monitoring Queue Health
+
+```redis
+# Check queue depth
+ZCARD task:queue
+
+# List pending tasks
+ZRANGE task:queue 0 -1 WITHSCORES
+
+# Check dead letter queue
+KEYS task:dlq:*
+```
+
+### Handling Stuck Tasks
+
+**Symptom:** Tasks stuck in "running" status
+
+**Diagnosis:**
+```bash
+# Check for expired leases
+redis-cli GET task:{task-id}
+# Look for LeaseExpiry in past
+```
+
+**Rem
+
+ediation:**
+Tasks with expired leases are automatically reclaimed every 1 minute. To force immediate reclamation:
+```bash
+# Restart worker to trigger reclaim cycle
+systemctl restart ml-worker
+```
+
+### Dead Letter Queue Management
+
+**View failed tasks:**
+```redis
+KEYS task:dlq:*
+```
+
+**Inspect failed task:**
+```redis
+GET task:dlq:{task-id}
+```
+
+**Retry from DLQ:**
+```bash
+# Manual retry (requires custom script)
+# 1. Get task from DLQ
+# 2. Reset retry count
+# 3. Re-queue task
+```
+
+### Worker Crashes
+
+**Symptom:** Worker disappeared mid-task
+
+**What Happens:**
+1. Lease expires after 30 minutes (default)
+2. Background reclaim job detects expired lease
+3. Task is retried (up to 3 attempts)
+4. After max retries → Dead Letter Queue
+
+**Prevention:**
+- Monitor worker heartbeats
+- Set up alerts for worker down
+- Use process manager (systemd, supervisor)
+
+## Worker Operations
+
+### Graceful Shutdown
+
+```bash
+# Send SIGTERM for graceful shutdown
+kill -TERM $(pgrep ml-worker)
+
+# Worker will:
+# 1. Stop accepting new tasks
+# 2. Finish active tasks (up to 5min timeout)
+# 3. Release all leases
+# 4. Exit cleanly
+```
+
+### Force Shutdown
+
+```bash
+# Force kill (leases will be reclaimed automatically)
+kill -9 $(pgrep ml-worker)
+```
+
+### Worker Heartbeat Monitoring
+
+```redis
+# Check worker heartbeats
+HGETALL worker:heartbeat
+
+# Example output:
+# worker-abc123 1701234567
+# worker-def456 1701234580
+```
+
+**Alert if:** Heartbeat timestamp > 5 minutes old
+
+## Redis Operations
+
+### Backup
+
+```bash
+# Manual backup
+redis-cli SAVE
+cp /var/lib/redis/dump.rdb /backup/redis-$(date +%Y%m%d).rdb
+```
+
+### Restore
+
+```bash
+# Stop Redis
+systemctl stop redis
+
+#  Restore snapshot
+cp /backup/redis-20231201.rdb /var/lib/redis/dump.rdb
+
+# Start Redis
+systemctl start redis
+```
+
+### Memory Management
+
+```redis
+# Check memory usage
+INFO memory
+
+# Evict old data if needed
+FLUSHDB  # DANGER: Clears all data!
+```
+
+## Common Issues
+
+### Issue: Queue Growing Unbounded
+
+**Symptoms:**
+- `ZCARD task:queue` keeps increasing
+- No workers processing tasks
+
+**Diagnosis:**
+```bash
+# Check worker status
+systemctl status ml-worker
+
+# Check logs
+journalctl -u ml-worker -n 100
+```
+
+**Resolution:**
+1. Verify workers are running
+2. Check Redis connectivity
+3. Verify lease configuration
+
+### Issue: High Retry Rate
+
+**Symptoms:**
+- Many tasks in DLQ
+- `retry_count` field high on tasks
+
+**Diagnosis:**
+```bash
+# Check worker logs for errors
+journalctl -u ml-worker | grep "retry"
+
+# Look for patterns (network issues, resource limits, etc)
+```
+
+**Resolution:**
+- Fix underlying issue (network, resources, etc)
+- Adjust retry limits if permanent failures
+- Increase task timeout if jobs are slow
+
+### Issue: Leases Expiring Prematurely
+
+**Symptoms:**
+- Tasks retried even though worker is healthy
+- Logs show "lease expired" frequently
+
+**Diagnosis:**
+```yaml
+# Check worker config
+cat configs/worker-config.yaml | grep -A3 "lease"
+
+task_lease_duration: 30m  # Too short?
+heartbeat_interval: 1m    # Too infrequent?
+```
+
+**Resolution:**
+```yaml
+# Increase lease duration for long-running jobs
+task_lease_duration: 60m
+heartbeat_interval: 30s  # More frequent heartbeats
+```
+
+## Performance Tuning
+
+### Worker Concurrency
+
+```yaml
+# worker-config.yaml
+max_workers: 4  # Number of parallel tasks
+
+# Adjust based on:
+# - CPU cores available
+# - Memory per task
+# - GPU availability
+```
+
+### Redis Configuration
+
+```conf
+# /etc/redis/redis.conf
+
+# Persistence
+save 900 1
+save 300 10
+
+# Memory
+maxmemory 2gb
+maxmemory-policy noeviction
+
+# Performance
+tcp-keepalive 300
+timeout 0
+```
+
+## Alerting Rules
+
+### Critical Alerts
+
+1. **Worker Down** (no heartbeat > 5min)
+2. **Queue Depth** > 1000 tasks
+3. **DLQ Growth** > 100 tasks/hour
+4. **Redis Down** (connection failed)
+
+### Warning Alerts
+
+1. **High Retry Rate** > 10% of tasks
+2. **Slow Queue Drain** (depth increasing over 1 hour)
+3. **Worker Memory** > 80% usage
+
+## Health Checks
+
+```bash
+#!/bin/bash
+# health-check.sh
+
+# Check Redis
+redis-cli PING || echo "Redis DOWN"
+
+# Check worker heartbeat
+WORKER_ID=$(cat /var/run/ml-worker.pid)
+LAST_HB=$(redis-cli HGET worker:heartbeat "$WORKER_ID")
+NOW=$(date +%s)
+if [ $((NOW - LAST_HB)) -gt 300 ]; then
+  echo "Worker heartbeat stale"
+fi
+
+# Check queue depth
+DEPTH=$(redis-cli ZCARD task:queue)
+if [ "$DEPTH" -gt 1000 ]; then
+  echo "Queue depth critical: $DEPTH"
+fi
+```
+
+## Runbook Checklist
+
+### Daily Operations
+1. Check queue depth
+2. Verify worker heartbeats
+3. Review DLQ for patterns
+4. Check Redis memory usage
+
+### Weekly Operations
+1. Review retry rates
+2. Analyze failed task patterns
+3. Backup Redis snapshot
+4. Review worker logs
+
+### Monthly Operations
+1. Performance tuning review
+2. Capacity planning
+3. Update documentation
+4. Test disaster recovery
+
+---
+
+**For homelab setups:**
+Most of these operations can be simplified. Focus on:
+-  Basic monitoring (queue depth, worker status)
+- Periodic Redis backups
+- Graceful shutdowns for maintenance

docs/src/production-monitoring.md

@@ -0,0 +1,217 @@
+# Production Monitoring Deployment Guide (Linux)
+
+This guide covers deploying the monitoring stack (Prometheus, Grafana, Loki, Promtail) on Linux production servers.
+
+## Architecture
+
+**Testing**: Docker Compose (macOS/Linux)
+**Production**: Podman + systemd (Linux)
+
+**Important**: Docker is for testing only. Podman is used for running actual ML experiments in production.
+
+**Dev (Testing)**: Docker Compose  
+**Prod (Experiments)**: Podman + systemd
+
+Each service runs as a separate Podman container managed by systemd for automatic restarts and proper lifecycle management.
+
+## Prerequisites
+
+**Container Runtimes:**
+- **Docker Compose**: For testing and development only
+- **Podman**: For production experiment execution
+
+- Linux distribution with systemd (Rocky/RHEL/CentOS, Ubuntu/Debian, Arch, SUSE, etc.)
+- Production app already deployed (see `scripts/setup-prod.sh`)
+- Root or sudo access
+- Ports 3000, 9090, 3100 available
+
+## Quick Setup
+
+### 1. Run Setup Script
+```bash
+cd /path/to/fetch_ml
+sudo ./scripts/setup-monitoring-prod.sh /data/monitoring ml-user ml-group
+```
+
+This will:
+- Create directory structure at `/data/monitoring`
+- Copy configuration files to `/etc/fetch_ml/monitoring`
+- Create systemd services for each component
+- Set up firewall rules
+
+### 2. Start Services
+```bash
+# Start all monitoring services
+sudo systemctl start prometheus
+sudo systemctl start loki
+sudo systemctl start promtail
+sudo systemctl start grafana
+
+# Enable on boot
+sudo systemctl enable prometheus loki promtail grafana
+```
+
+### 3. Access Grafana
+- URL: `http://YOUR_SERVER_IP:3000`
+- Username: `admin`
+- Password: `admin` (change on first login)
+
+Dashboards will auto-load:
+- **ML Task Queue Monitoring** (metrics)
+- **Application Logs** (Loki logs)
+
+## Service Details
+
+### Prometheus
+- **Port**: 9090
+- **Config**: `/etc/fetch_ml/monitoring/prometheus.yml`
+- **Data**: `/data/monitoring/prometheus`
+- **Purpose**: Scrapes metrics from API server
+
+### Loki
+- **Port**: 3100
+- **Config**: `/etc/fetch_ml/monitoring/loki-config.yml`
+- **Data**: `/data/monitoring/loki`
+- **Purpose**: Log aggregation
+
+### Promtail
+- **Config**: `/etc/fetch_ml/monitoring/promtail-config.yml`
+- **Log Source**: `/var/log/fetch_ml/*.log`
+- **Purpose**: Ships logs to Loki
+
+### Grafana
+- **Port**: 3000
+- **Config**: `/etc/fetch_ml/monitoring/grafana/provisioning`
+- **Data**: `/data/monitoring/grafana`
+- **Dashboards**: `/var/lib/grafana/dashboards`
+
+## Management Commands
+
+```bash
+# Check status
+sudo systemctl status prometheus grafana loki promtail
+
+# View logs
+sudo journalctl -u prometheus -f
+sudo journalctl -u grafana -f
+sudo journalctl -u loki -f
+sudo journalctl -u promtail -f
+
+# Restart services
+sudo systemctl restart prometheus
+sudo systemctl restart grafana
+
+# Stop all monitoring
+sudo systemctl stop prometheus grafana loki promtail
+```
+
+## Data Retention
+
+### Prometheus
+Default: 15 days. Edit `/etc/fetch_ml/monitoring/prometheus.yml`:
+```yaml
+storage:
+  tsdb:
+    retention.time: 30d
+```
+
+### Loki
+Default: 30 days. Edit `/etc/fetch_ml/monitoring/loki-config.yml`:
+```yaml
+limits_config:
+  retention_period: 30d
+```
+
+## Security
+
+### Firewall
+The setup script automatically configures firewall rules using the detected firewall manager (firewalld or ufw).
+
+For manual firewall configuration:
+
+**RHEL/Rocky/Fedora (firewalld)**:
+```bash
+# Remove public access
+sudo firewall-cmd --permanent --remove-port=3000/tcp
+sudo firewall-cmd --permanent --remove-port=9090/tcp
+
+# Add specific source
+sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="10.0.0.0/24" port port="3000" protocol="tcp" accept'
+sudo firewall-cmd --reload
+```
+
+**Ubuntu/Debian (ufw)**:
+```bash
+# Remove public access
+sudo ufw delete allow 3000/tcp
+sudo ufw delete allow 9090/tcp
+
+# Add specific source
+sudo ufw allow from 10.0.0.0/24 to any port 3000 proto tcp
+```
+
+### Authentication
+Change Grafana admin password:
+1. Login to Grafana
+2. User menu → Profile → Change Password
+
+### TLS (Optional)
+For HTTPS, configure reverse proxy (nginx/Apache) in front of Grafana.
+
+## Troubleshooting
+
+### Grafana shows no data
+```bash
+# Check if Prometheus is reachable
+curl http://localhost:9090/-/healthy
+
+# Check datasource in Grafana
+# Settings → Data Sources → Prometheus → Save & Test
+```
+
+### Loki not receiving logs
+```bash
+# Check Promtail is running
+sudo systemctl status promtail
+
+# Verify log file exists
+ls -l /var/log/fetch_ml/
+
+# Check Promtail can reach Loki
+curl http://localhost:3100/ready
+```
+
+### Podman containers not starting
+```bash
+# Check pod status
+sudo -u ml-user podman pod ps
+sudo -u ml-user podman ps -a
+
+# Remove and recreate
+sudo -u ml-user podman pod stop monitoring
+sudo -u ml-user podman pod rm monitoring
+sudo systemctl restart prometheus
+```
+
+## Backup
+
+```bash
+# Backup Grafana dashboards and data
+sudo tar -czf grafana-backup.tar.gz /data/monitoring/grafana
+
+# Backup Prometheus data
+sudo tar -czf prometheus-backup.tar.gz /data/monitoring/prometheus
+```
+
+## Updates
+
+```bash
+# Pull latest images
+sudo -u ml-user podman pull docker.io/grafana/grafana:latest
+sudo -u ml-user podman pull docker.io/prom/prometheus:latest
+sudo -u ml-user podman pull docker.io/grafana/loki:latest
+sudo -u ml-user podman pull docker.io/grafana/promtail:latest
+
+# Restart services to use new images
+sudo systemctl restart grafana prometheus loki promtail
+```

docs/src/queue.md

@@ -0,0 +1,322 @@
+---
+layout: page
+title: "Task Queue Architecture"
+permalink: /queue/
+nav_order: 3
+---
+
+# Task Queue Architecture
+
+The task queue system enables reliable job processing between the API server and workers using Redis.
+
+## Overview
+
+```mermaid
+graph LR
+    CLI[CLI/Client] -->|WebSocket| API[API Server]
+    API -->|Enqueue| Redis[(Redis)]
+    Redis -->|Dequeue| Worker[Worker]
+    Worker -->|Update Status| Redis
+```
+
+## Components
+
+### TaskQueue (`internal/queue`)
+
+Shared package used by both API server and worker for job management.
+
+#### Task Structure
+
+```go
+type Task struct {
+    ID        string            // Unique task ID (UUID)
+    JobName   string            // User-defined job name  
+    Args      string            // Job arguments
+    Status    string            // queued, running, completed, failed
+    Priority  int64             // Higher = executed first
+    CreatedAt time.Time         
+    StartedAt *time.Time        
+    EndedAt   *time.Time        
+    WorkerID  string            
+    Error     string            
+    Datasets  []string          
+    Metadata  map[string]string // commit_id, user, etc
+}
+```
+
+#### TaskQueue Interface
+
+```go
+// Initialize queue
+queue, err := queue.NewTaskQueue(queue.Config{
+    RedisAddr:     "localhost:6379",
+    RedisPassword: "",
+    RedisDB:       0,
+})
+
+// Add task (API server)
+task := &queue.Task{
+    ID:       uuid.New().String(),
+    JobName:  "train-model",
+    Status:   "queued",
+    Priority: 5,
+    Metadata: map[string]string{
+        "commit_id": commitID,
+        "user":      username,
+    },
+}
+err = queue.AddTask(task)
+
+// Get next task (Worker)
+task, err := queue.GetNextTask()
+
+// Update task status
+task.Status = "running"
+err = queue.UpdateTask(task)
+```
+
+## Data Flow
+
+### Job Submission Flow
+
+```mermaid
+sequenceDiagram
+    participant CLI
+    participant API
+    participant Redis
+    participant Worker
+    
+    CLI->>API: Queue Job (WebSocket)
+    API->>API: Create Task (UUID)
+    API->>Redis: ZADD task:queue
+    API->>Redis: SET task:{id}
+    API->>CLI: Success Response
+    
+    Worker->>Redis: ZPOPMAX task:queue
+    Redis->>Worker: Task ID
+    Worker->>Redis: GET task:{id}
+    Redis->>Worker: Task Data
+    Worker->>Worker: Execute Job
+    Worker->>Redis: Update Status
+```
+
+### Protocol
+
+**CLI → API** (Binary WebSocket):
+```
+[opcode:1][api_key_hash:64][commit_id:64][priority:1][job_name_len:1][job_name:var]
+```
+
+**API → Redis**:
+- Priority queue: `ZADD task:queue {priority} {task_id}`
+- Task data: `SET task:{id} {json}`
+- Status: `HSET task:status:{job_name} ...`
+
+**Worker ← Redis**:
+- Poll: `ZPOPMAX task:queue 1` (highest priority first)
+- Fetch: `GET task:{id}`
+
+## Redis Data Structures
+
+### Keys
+
+```
+task:queue                    # ZSET: priority queue
+task:{uuid}                  # STRING: task JSON data
+task:status:{job_name}       # HASH: job status
+worker:heartbeat             # HASH: worker health
+job:metrics:{job_name}       # HASH: job metrics
+```
+
+### Priority Queue (ZSET)
+
+```redis
+ZADD task:queue 10 "uuid-1"   # Priority 10
+ZADD task:queue 5  "uuid-2"   # Priority 5
+ZPOPMAX task:queue 1          # Returns uuid-1 (highest)
+```
+
+## API Server Integration
+
+### Initialization
+
+```go
+// cmd/api-server/main.go
+queueCfg := queue.Config{
+    RedisAddr:     cfg.Redis.Addr,
+    RedisPassword: cfg.Redis.Password,
+    RedisDB:       cfg.Redis.DB,
+}
+taskQueue, err := queue.NewTaskQueue(queueCfg)
+```
+
+### WebSocket Handler
+
+```go
+// internal/api/ws.go
+func (h *WSHandler) handleQueueJob(conn *websocket.Conn, payload []byte) error {
+    // Parse request
+    apiKeyHash, commitID, priority, jobName := parsePayload(payload)
+    
+    // Create task with unique ID
+    taskID := uuid.New().String()
+    task := &queue.Task{
+        ID:       taskID,
+        JobName:  jobName,
+        Status:   "queued",
+        Priority: int64(priority),
+        Metadata: map[string]string{
+            "commit_id": commitID,
+            "user":      user,
+        },
+    }
+    
+    // Enqueue
+    if err := h.queue.AddTask(task); err != nil {
+        return h.sendErrorPacket(conn, ErrorCodeDatabaseError, ...)
+    }
+    
+    return h.sendSuccessPacket(conn, "Job queued")
+}
+```
+
+## Worker Integration
+
+### Task Polling
+
+```go
+// cmd/worker/worker_server.go
+func (w *Worker) Start() error {
+    for {
+        task, err := w.queue.WaitForNextTask(ctx, 5*time.Second)
+        if task != nil {
+            go w.executeTask(task)
+        }
+    }
+}
+```
+
+### Task Execution
+
+```go
+func (w *Worker) executeTask(task *queue.Task) {
+    // Update status
+    task.Status = "running"
+    task.StartedAt = &now
+    w.queue.UpdateTaskWithMetrics(task, "start")
+    
+    // Execute
+    err := w.runJob(task)
+    
+    // Finalize
+    task.Status = "completed" // or "failed"
+    task.EndedAt = &endTime
+    task.Error = err.Error() // if err != nil
+    w.queue.UpdateTaskWithMetrics(task, "final")
+}
+```
+
+## Configuration
+
+### API Server (`configs/config.yaml`)
+
+```yaml
+redis:
+  addr: "localhost:6379"
+  password: ""
+  db: 0
+```
+
+### Worker (`configs/worker-config.yaml`)
+
+```yaml
+redis:
+  addr: "localhost:6379"
+  password: ""
+  db: 0
+  
+metrics_flush_interval: 500ms
+```
+
+## Monitoring
+
+### Queue Depth
+
+```go
+depth, err := queue.QueueDepth()
+fmt.Printf("Pending tasks: %d\n", depth)
+```
+
+### Worker Heartbeat
+
+```go
+// Worker sends heartbeat every 30s
+err := queue.Heartbeat(workerID)
+```
+
+### Metrics
+
+```redis
+HGETALL job:metrics:{job_name}
+# Returns: timestamp, tasks_start, tasks_final, etc
+```
+
+## Error Handling
+
+### Task Failures
+
+```go
+if err := w.runJob(task); err != nil {
+    task.Status = "failed"
+    task.Error = err.Error()
+    w.queue.UpdateTask(task)
+}
+```
+
+### Redis Connection Loss
+
+```go
+// TaskQueue automatically reconnects
+// Workers should implement retry logic
+for retries := 0; retries < 3; retries++ {
+    task, err := queue.GetNextTask()
+    if err == nil {
+        break
+    }
+    time.Sleep(backoff)
+}
+```
+
+## Testing
+
+```go
+// tests using miniredis
+s, _ := miniredis.Run()
+defer s.Close()
+
+tq, _ := queue.NewTaskQueue(queue.Config{
+    RedisAddr: s.Addr(),
+})
+
+task := &queue.Task{ID: "test-1", JobName: "test"}
+tq.AddTask(task)
+
+fetched, _ := tq.GetNextTask()
+// assert fetched.ID == "test-1"
+```
+
+## Best Practices
+
+1. **Unique Task IDs**: Always use UUIDs to avoid conflicts
+2. **Metadata**: Store commit_id and user in task metadata
+3. **Priority**: Higher values execute first (0-255 range)
+4. **Status Updates**: Update status at each lifecycle stage
+5. **Error Logging**: Store detailed errors in task.Error
+6. **Heartbeats**: Workers should send heartbeats regularly
+7. **Metrics**: Use UpdateTaskWithMetrics for atomic updates
+
+---
+
+For implementation details, see:
+- [internal/queue/task.go](https://github.com/jfraeys/fetch_ml/blob/main/internal/queue/task.go)
+- [internal/queue/queue.go](https://github.com/jfraeys/fetch_ml/blob/main/internal/queue/queue.go)

docs/src/quick-start.md

@@ -0,0 +1,93 @@
+# Quick Start
+
+Get Fetch ML running in minutes with Docker Compose.
+
+## Prerequisites
+
+**Container Runtimes:**
+- **Docker Compose**: For testing and development only
+- **Podman**: For production experiment execution
+
+- Docker Compose (testing only)
+- 4GB+ RAM
+- 2GB+ disk space
+
+## One-Command Setup
+
+```bash
+# Clone and start
+git clone https://github.com/jfraeys/fetch_ml.git
+cd fetch_ml
+docker-compose up -d (testing only)
+
+# Wait for services (30 seconds)
+sleep 30
+
+# Verify setup
+curl http://localhost:9101/health
+```
+
+## First Experiment
+
+```bash
+# Submit a simple ML job (see [First Experiment](first-experiment.md) for details)
+curl -X POST http://localhost:9101/api/v1/jobs \
+  -H "Content-Type: application/json" \
+  -H "X-API-Key: admin" \
+  -d '{
+    "job_name": "hello-world",
+    "args": "--echo Hello World",
+    "priority": 1
+  }'
+
+# Check job status
+curl http://localhost:9101/api/v1/jobs \
+  -H "X-API-Key: admin"
+```
+
+## CLI Access
+
+```bash
+# Build CLI
+cd cli && zig build dev
+
+# List jobs
+./cli/zig-out/dev/ml --server http://localhost:9101 list-jobs
+
+# Submit new job
+./cli/zig-out/dev/ml --server http://localhost:9101 submit \
+  --name "test-job" --args "--epochs 10"
+```
+
+## Related Documentation
+
+- [Installation Guide](installation.md) - Detailed setup options
+- [First Experiment](first-experiment.md) - Complete ML workflow
+- [Development Setup](development-setup.md) - Local development
+- [Security](security.md) - Authentication and permissions
+
+## Troubleshooting
+
+**Services not starting?**
+```bash
+# Check logs
+docker-compose logs
+
+# Restart services
+docker-compose down && docker-compose up -d (testing only)
+```
+
+**API not responding?**
+```bash
+# Check health
+curl http://localhost:9101/health
+
+# Verify ports
+docker-compose ps
+```
+
+**Permission denied?**
+```bash
+# Check API key
+curl -H "X-API-Key: admin" http://localhost:9101/api/v1/jobs
+```

docs/src/redis-ha.md

@@ -0,0 +1,95 @@
+---
+layout: page
+title: "Redis High Availability (Optional)"
+permalink: /redis-ha/
+nav_order: 7
+---
+
+# Redis High Availability
+
+**Note:** This is optional for homelab setups. Single Redis instance is sufficient for most use cases.
+
+## When You Need HA
+
+Consider Redis HA if:
+- Running production workloads
+- Uptime > 99.9% required
+- Can't afford to lose queued tasks
+- Multiple workers across machines
+
+## Redis Sentinel (Recommended)
+
+### Setup
+
+```yaml
+# docker-compose.yml
+version: '3.8'
+services:
+  redis-master:
+    image: redis:7-alpine
+    command: redis-server --maxmemory 2gb
+    
+  redis-replica:
+    image: redis:7-alpine
+    command: redis-server --slaveof redis-master 6379
+    
+  redis-sentinel-1:
+    image: redis:7-alpine
+    command: redis-sentinel /etc/redis/sentinel.conf
+    volumes:
+      - ./sentinel.conf:/etc/redis/sentinel.conf
+```
+
+**sentinel.conf:**
+```conf
+sentinel monitor mymaster redis-master 6379 2
+sentinel down-after-milliseconds mymaster 5000
+sentinel parallel-syncs mymaster 1
+sentinel failover-timeout mymaster 10000
+```
+
+### Application Configuration
+
+```yaml
+# worker-config.yaml
+redis_addr: "redis-sentinel-1:26379,redis-sentinel-2:26379"
+redis_master_name: "mymaster"
+```
+
+## Redis Cluster (Advanced)
+
+For larger deployments with sharding needs.
+
+```yaml
+# Minimum 3 masters + 3 replicas
+services:
+  redis-1:
+    image: redis:7-alpine
+    command: redis-server --cluster-enabled yes
+    
+  redis-2:
+    # ... similar config
+```
+
+## Homelab Alternative: Persistence Only
+
+**For most homelabs, just enable persistence:**
+
+```yaml
+# docker-compose.yml
+services:
+  redis:
+    image: redis:7-alpine
+    command: redis-server --appendonly yes
+    volumes:
+      - redis_data:/data
+      
+volumes:
+  redis_data:
+```
+
+This ensures tasks survive Redis restarts without full HA complexity.
+
+---
+
+**Recommendation:** Start simple. Add HA only if you experience actual downtime issues.

docs/src/release-checklist.md

@@ -0,0 +1,37 @@
+# Release Checklist
+
+This checklist captures the work required before cutting a release that includes
+the graceful worker shutdown feature.
+
+## 1. Code Hygiene / Compilation
+1. Merge the graceful-shutdown helpers into the canonical worker type to avoid
+     `Worker redeclared` errors (see `cmd/worker/worker_graceful_shutdown.go` and
+     `cmd/worker/worker_server.go`).
+2. Ensure the worker struct exposes the fields referenced by the new helpers
+     (`logger`, `queue`, `cfg`, `metrics`).
+3. `go build ./cmd/worker` succeeds without undefined-field errors.
+
+## 2. Graceful Shutdown Logic
+1. Initialize `shutdownCh`, `activeTasks`, and `gracefulWait` during worker start-up.
+2. Confirm the heartbeat/lease helpers compile and handle queue errors gracefully
+     (`heartbeatLoop`, `releaseAllLeases`).
+3. Add tests (unit or integration) that simulate SIGINT/SIGTERM and verify leases
+     are released or tasks complete.
+
+## 3. Task Execution Flow
+1. Align `executeTaskWithLease` with the real `executeTask` signature so the
+     "no value used as value" compile error disappears.
+2. Double-check retry/metrics paths still match existing worker behavior after
+     the new wrapper is added.
+
+## 4. Server Wiring
+1. Ensure worker construction in `cmd/worker/worker_server.go` wires up config,
+     queue, metrics, and logger instances used by the shutdown logic.
+2. Re-run worker unit tests plus any queue/lease e2e tests.
+
+## 5. Validation Before Tagging
+1. `go test ./cmd/worker/...` and `make test` (or equivalent) pass locally.
+2. Manual smoke test: start worker, queue jobs, send SIGTERM, confirm tasks finish
+     or leases are released and the process exits cleanly.
+3. Update release notes describing the new shutdown capability and any config
+     changes required (e.g., graceful timeout settings).

docs/src/security.md

@@ -0,0 +1,297 @@
+# Security Guide
+
+This document outlines security features, best practices, and hardening procedures for FetchML.
+
+## Security Features
+
+### Authentication & Authorization
+- **API Keys**: SHA256-hashed with role-based access control (RBAC)
+- **Permissions**: Granular read/write/delete permissions per user
+- **IP Whitelisting**: Network-level access control
+- **Rate Limiting**: Per-user request quotas
+
+### Communication Security
+- **TLS/HTTPS**: End-to-end encryption for API traffic
+- **WebSocket Auth**: API key required before upgrade
+- **Redis Auth**: Password-protected task queue
+
+### Data Privacy
+- **Log Sanitization**: Automatically redacts API keys, passwords, tokens
+- **Experiment Isolation**: User-specific experiment directories
+- **No Anonymous Access**: All services require authentication
+
+### Network Security
+- **Internal Networks**: Backend services (Redis, Loki) not exposed publicly
+- **Firewall Rules**: Restrictive port access
+- **Container Isolation**: Services run in separate containers/pods
+
+## Security Checklist
+
+### Initial Setup
+
+1. **Generate Strong Passwords**
+  ```bash
+  # Grafana admin password
+  openssl rand -base64 32 > .grafana-password
+  
+  # Redis password
+  openssl rand -base64 32
+  ```
+
+2. **Configure Environment Variables**
+  ```bash
+  cp .env.example .env
+  # Edit .env and set:
+  # - GRAFANA_ADMIN_PASSWORD
+  ```
+
+3. **Enable TLS** (Production only)
+  ```yaml
+  # configs/config-prod.yaml
+  server:
+    tls:
+      enabled: true
+      cert_file: "/secrets/cert.pem"
+      key_file: "/secrets/key.pem"
+  ```
+
+4. **Configure Firewall**
+  ```bash
+  # Allow only necessary ports
+  sudo ufw allow 22/tcp    # SSH
+  sudo ufw allow 443/tcp   # HTTPS
+  sudo ufw allow 80/tcp    # HTTP (redirect to HTTPS)
+  sudo ufw enable
+  ```
+
+### Production Hardening
+
+5. **Restrict IP Access**
+  ```yaml
+  # configs/config-prod.yaml
+  auth:
+    ip_whitelist:
+      - "10.0.0.0/8"
+      - "192.168.0.0/16"
+      - "127.0.0.1"
+  ```
+
+6. **Enable Audit Logging**
+  ```yaml
+  logging:
+    level: "info"
+    audit: true
+    file: "/var/log/fetch_ml/audit.log"
+  ```
+
+7. **Harden Redis**
+  ```bash
+  # Redis security
+  redis-cli CONFIG SET requirepass "your-strong-password"
+  redis-cli CONFIG SET rename-command FLUSHDB ""
+  redis-cli CONFIG SET rename-command FLUSHALL ""
+  ```
+
+8. **Secure Grafana**
+  ```bash
+  # Change default admin password
+  docker-compose exec grafana grafana-cli admin reset-admin-password new-strong-password
+  ```
+
+9. **Regular Updates**
+  ```bash
+  # Update system packages
+  sudo apt update && sudo apt upgrade -y
+  
+  # Update containers
+  docker-compose pull
+  docker-compose up -d (testing only)
+  ```
+
+## Password Management
+
+### Generate Secure Passwords
+
+```bash
+# Method 1: OpenSSL
+openssl rand -base64 32
+
+# Method 2: pwgen (if installed)
+pwgen -s 32 1
+
+# Method 3: /dev/urandom
+head -c 32 /dev/urandom | base64
+```
+
+### Store Passwords Securely
+
+**Development**: Use `.env` file (gitignored)
+```bash
+echo "REDIS_PASSWORD=$(openssl rand -base64 32)" >> .env
+echo "GRAFANA_ADMIN_PASSWORD=$(openssl rand -base64 32)" >> .env
+```
+
+**Production**: Use systemd environment files
+```bash
+sudo mkdir -p /etc/fetch_ml/secrets
+sudo chmod 700 /etc/fetch_ml/secrets
+echo "REDIS_PASSWORD=..." | sudo tee /etc/fetch_ml/secrets/redis.env
+sudo chmod 600 /etc/fetch_ml/secrets/redis.env
+```
+
+## API Key Management
+
+### Generate API Keys
+
+```bash
+# Generate random API key
+openssl rand -hex 32
+
+# Hash for storage
+echo -n "your-api-key" | sha256sum
+```
+
+### Rotate API Keys
+
+1. Generate new API key
+2. Update `config-local.yaml` with new hash
+3. Distribute new key to users
+4. Remove old key after grace period
+
+### Revoke API Keys
+
+Remove user entry from `config-local.yaml`:
+```yaml
+auth:
+  apikeys:
+    # user_to_revoke:  # Comment out or delete
+```
+
+## Network Security
+
+### Production Network Topology
+
+```
+Internet
+    ↓
+[Firewall] (ports 3000, 9102)
+    ↓
+[Reverse Proxy] (nginx/Apache) - TLS termination
+    ↓
+┌─────────────────────┐
+│   Application Pod   │
+│                     │
+│  ┌──────────────┐   │
+│  │ API Server   │   │  ← Public (via reverse proxy)
+│  └──────────────┘   │
+│                     │
+│  ┌──────────────┐   │
+│  │   Redis      │   │  ← Internal only
+│  └──────────────┘   │
+│                     │
+│  ┌──────────────┐   │
+│  │   Grafana    │   │  ← Public (via reverse proxy)
+│  └──────────────┘   │
+│                     │
+│  ┌──────────────┐   │
+│  │ Prometheus   │   │  ← Internal only
+│  └──────────────┘   │
+│                     │
+│  ┌──────────────┐   │
+│  │    Loki      │   │  ← Internal only
+│  └──────────────┘   │
+└─────────────────────┘
+```
+
+### Recommended Firewall Rules
+
+```bash
+# Allow only necessary inbound connections
+sudo firewall-cmd --permanent --zone=public --add-rich-rule='
+  rule family="ipv4"
+  source address="YOUR_NETWORK"
+  port port="3000" protocol="tcp" accept'
+
+sudo firewall-cmd --permanent --zone=public --add-rich-rule='
+  rule family="ipv4"
+  source address="YOUR_NETWORK"
+  port port="9102" protocol="tcp" accept'
+
+# Block all other traffic
+sudo firewall-cmd --permanent --set-default-zone=drop
+sudo firewall-cmd --reload
+```
+
+## Incident Response
+
+### Suspected Breach
+
+1. **Immediate Actions**
+2. **Investigation** 
+3. **Recovery** 
+   - Rotate all API keys
+   - Stop affected services
+   - Review audit logs
+   
+2. **Investigation**
+   ```bash
+   # Check recent logins
+   sudo journalctl -u fetchml-api --since "1 hour ago"
+   
+   # Review failed auth attempts
+   grep "authentication failed" /var/log/fetch_ml/*.log
+   
+   # Check active connections
+   ss -tnp | grep :9102
+   ```
+
+3. **Recovery**
+   - Rotate all passwords and API keys
+   - Update firewall rules
+   - Patch vulnerabilities
+   - Resume services
+
+### Security Monitoring
+
+```bash
+# Monitor failed authentication
+tail -f /var/log/fetch_ml/api.log | grep "auth.*failed"
+
+# Monitor unusual activity
+journalctl -u fetchml-api -f | grep -E "(ERROR|WARN)"
+
+# Check open ports
+nmap -p- localhost
+```
+
+## Security Best Practices
+
+1. **Principle of Least Privilege**: Grant minimum necessary permissions
+2. **Defense in Depth**: Multiple security layers (firewall + auth + TLS)
+3. **Regular Updates**: Keep all components patched
+4. **Audit Regularly**: Review logs and access patterns
+5. **Secure Secrets**: Never commit passwords/keys to git
+6. **Network Segmentation**: Isolate services with internal networks
+7. **Monitor Everything**: Enable comprehensive logging and alerting
+8. **Test Security**: Regular penetration testing and vulnerability scans
+
+## Compliance
+
+### Data Privacy
+- Logs are sanitized (no passwords/API keys)
+- Experiment data is user-isolated
+- No telemetry or external data sharing
+
+### Audit Trail
+All API access is logged with:
+- Timestamp
+- User/API key
+- Action performed
+- Source IP
+- Result (success/failure)
+
+## Getting Help
+
+- **Security Issues**: Report privately via email
+- **Questions**: See documentation or create issue
+- **Updates**: Monitor releases for security patches

docs/src/smart-defaults.md

@@ -0,0 +1,187 @@
+# Smart Defaults
+
+This document describes Fetch ML's smart defaults system, which automatically adapts configuration based on the runtime environment.
+
+## Overview
+
+Smart defaults eliminate the need for manual configuration tweaks when running in different environments:
+
+- **Local Development**: Optimized for developer machines with sensible paths and localhost services
+- **Container Environments**: Uses container-friendly hostnames and paths
+- **CI/CD**: Optimized for automated testing with fast polling and minimal resource usage
+- **Production**: Uses production-ready defaults with proper security and scaling
+
+## Environment Detection
+
+The system automatically detects the environment based on:
+
+1. **CI Detection**: Checks for `CI`, `GITHUB_ACTIONS`, `GITLAB_CI` environment variables
+2. **Container Detection**: Looks for `/.dockerenv`, `KUBERNETES_SERVICE_HOST`, or `CONTAINER` variables
+3. **Production Detection**: Checks `FETCH_ML_ENV=production` or `ENV=production`
+4. **Default**: Falls back to local development
+
+## Default Values by Environment
+
+### Host Configuration
+- **Local**: `localhost`
+- **Container/CI**: `host.docker.internal` (Docker Desktop/Colima)
+- **Production**: `0.0.0.0`
+
+### Base Paths
+- **Local**: `~/ml-experiments`
+- **Container/CI**: `/workspace/ml-experiments`
+- **Production**: `/var/lib/fetch_ml/experiments`
+
+### Data Directory
+- **Local**: `~/ml-data`
+- **Container/CI**: `/workspace/data`
+- **Production**: `/var/lib/fetch_ml/data`
+
+### Redis Address
+- **Local**: `localhost:6379`
+- **Container/CI**: `redis:6379` (service name)
+- **Production**: `redis:6379`
+
+### SSH Configuration
+- **Local**: `~/.ssh/id_rsa` and `~/.ssh/known_hosts`
+- **Container/CI**: `/workspace/.ssh/id_rsa` and `/workspace/.ssh/known_hosts`
+- **Production**: `/etc/fetch_ml/ssh/id_rsa` and `/etc/fetch_ml/ssh/known_hosts`
+
+### Worker Configuration
+- **Local**: 2 workers, 5-second poll interval
+- **CI**: 1 worker, 1-second poll interval (fast testing)
+- **Production**: CPU core count workers, 10-second poll interval
+
+### Log Levels
+- **Local**: `info`
+- **CI**: `debug` (verbose for debugging)
+- **Production**: `info`
+
+## Usage
+
+### In Configuration Loaders
+
+```go
+// Get smart defaults for current environment
+smart := config.GetSmartDefaults()
+
+// Use smart defaults
+if cfg.Host == "" {
+    cfg.Host = smart.Host()
+}
+if cfg.BasePath == "" {
+    cfg.BasePath = smart.BasePath()
+}
+```
+
+### Environment Overrides
+
+Smart defaults can be overridden with environment variables:
+
+- `FETCH_ML_HOST` - Override host
+- `FETCH_ML_BASE_PATH` - Override base path
+- `FETCH_ML_REDIS_ADDR` - Override Redis address
+- `FETCH_ML_ENV` - Force environment profile
+
+### Manual Environment Selection
+
+You can force a specific environment:
+
+```bash
+# Force production mode
+export FETCH_ML_ENV=production
+
+# Force container mode
+export CONTAINER=true
+```
+
+## Implementation Details
+
+The smart defaults system is implemented in `internal/config/smart_defaults.go`:
+
+- `DetectEnvironment()` - Determines current environment profile
+- `SmartDefaults` struct - Provides environment-aware defaults
+- Helper methods for each configuration value
+
+## Migration Guide
+
+### For Users
+
+No changes required - existing configurations continue to work. Smart defaults only apply when values are not explicitly set.
+
+### For Developers
+
+When adding new configuration options:
+
+1. Add a method to `SmartDefaults` struct
+2. Use the smart default in config loaders
+3. Document the environment-specific values
+
+Example:
+
+```go
+// Add to SmartDefaults struct
+func (s *SmartDefaults) NewFeature() string {
+    switch s.Profile {
+    case ProfileContainer, ProfileCI:
+        return "/workspace/new-feature"
+    case ProfileProduction:
+        return "/var/lib/fetch_ml/new-feature"
+    default:
+        return "./new-feature"
+    }
+}
+
+// Use in config loader
+if cfg.NewFeature == "" {
+    cfg.NewFeature = smart.NewFeature()
+}
+```
+
+## Testing
+
+To test different environments:
+
+```bash
+# Test local defaults (default)
+./bin/worker
+
+# Test container defaults
+export CONTAINER=true
+./bin/worker
+
+# Test CI defaults
+export CI=true
+./bin/worker
+
+# Test production defaults
+export FETCH_ML_ENV=production
+./bin/worker
+```
+
+## Troubleshooting
+
+### Wrong Environment Detection
+
+Check environment variables:
+
+```bash
+echo "CI: $CI"
+echo "CONTAINER: $CONTAINER"
+echo "FETCH_ML_ENV: $FETCH_ML_ENV"
+```
+
+### Path Issues
+
+Smart defaults expand `~` and environment variables automatically. If paths don't work as expected:
+
+1. Check the detected environment: `config.GetSmartDefaults().GetEnvironmentDescription()`
+2. Verify the path exists in the target environment
+3. Override with environment variable if needed
+
+### Container Networking
+
+For container environments, ensure:
+- Redis service is named `redis` in docker-compose
+- Host networking is configured properly
+- `host.docker.internal` resolves (Docker Desktop/Colima)

docs/src/testing.md

@@ -0,0 +1,46 @@
+# Testing Guide
+
+How to run and write tests for FetchML.
+
+## Running Tests
+
+### Quick Test
+
+```bash
+# All tests
+make test
+
+# Unit tests only
+make test-unit
+
+# Integration tests
+make test-integration
+
+# With coverage
+make test-coverage
+
+
+## Quick Test
+```bash
+make test          # All tests
+make test-unit     # Unit only
+.
+make test.
+make test$
+make test; make test  # Coverage
+  # E2E tests
+```
+
+## Docker Testing
+```bash
+docker-compose up -d (testing only)
+make test
+docker-compose down
+```
+
+## CLI Testing
+```bash
+cd cli && zig build dev
+./cli/zig-out/dev/ml --help
+zig build test
+``` 

docs/src/troubleshooting.md

@@ -0,0 +1,94 @@
+# Troubleshooting
+
+Common issues and solutions for Fetch ML.
+
+## Quick Fixes
+
+### Services Not Starting
+
+```bash
+# Check Docker status
+docker-compose ps
+
+# Restart services
+docker-compose down && docker-compose up -d (testing only)
+
+# Check logs
+docker-compose logs -f
+```
+
+### API Not Responding
+
+```bash
+# Check health endpoint
+curl http://localhost:9101/health
+
+# Check if port is in use
+lsof -i :9101
+
+# Kill process on port
+kill -9 $(lsof -ti :9101)
+```
+
+### Database Issues
+
+```bash
+# Check database connection
+docker-compose exec postgres psql -U postgres -d fetch_ml
+
+# Reset database
+docker-compose down postgres
+docker-compose up -d (testing only) postgres
+
+# Check Redis
+docker-compose exec redis redis-cli ping
+```
+
+## Common Errors
+
+### Authentication Errors
+
+- **Invalid API key**: Check config and regenerate hash
+- **JWT expired**: Check `jwt_expiry` setting
+
+### Database Errors
+- **Connection failed**: Verify database type and connection params
+- **No such table**: Run migrations with `--migrate` (see [Development Setup](development-setup.md))
+
+### Container Errors
+- **Runtime not found**: Set `runtime: docker (testing only)` in config
+- **Image pull failed**: Check registry access
+
+## Performance Issues
+- **High memory**: Adjust `resources.memory_limit`
+- **Slow jobs**: Check worker count and queue size
+
+## Development Issues
+- **Build fails**: `go mod tidy` and `cd cli && rm -rf zig-out zig-cache`
+- **Tests fail**: Start test dependencies with `docker-compose -f docker-compose.test.yml up -d`
+
+## CLI Issues
+- **Not found**: `cd cli && zig build dev`
+- **Connection errors**: Check `--server` and `--api-key`
+
+## Network Issues
+- **Port conflicts**: `lsof -i :9101` and kill processes
+- **Firewall**: Allow ports 9101, 6379, 5432
+
+## Configuration Issues
+- **Invalid YAML**: `python3 -c "import yaml; yaml.safe_load(open('config.yaml'))"`
+- **Missing fields**: Run `see [Configuration Schema](configuration-schema.md)`
+
+## Debug Information
+```bash
+./bin/api-server --version
+docker-compose ps
+docker-compose logs api-server | grep ERROR
+```
+
+## Emergency Reset
+```bash
+docker-compose down -v
+rm -rf data/ results/ *.db
+docker-compose up -d (testing only)
+```