Fetch ML CLI Reference

Comprehensive command-line tools for managing ML experiments in your homelab with Zig-based high-performance CLI.

Overview

Fetch ML provides a comprehensive CLI toolkit built with performance and security in mind:

• Zig CLI - High-performance experiment management written in Zig
• Go Commands - API server, TUI, and data management utilities
• Management Scripts - Service orchestration and deployment
• Setup Scripts - One-command installation and configuration

Zig CLI (./cli/zig-out/bin/ml)

High-performance command-line interface for experiment management, written in Zig for speed and efficiency.

Available Commands

Command	Description	Example
init	Interactive configuration setup	ml init
sync	Sync project to worker with deduplication	ml sync ./project --name myjob --queue
queue	Queue job for execution	ml queue myjob --commit abc123 --priority 8
status	Get system and worker status	ml status
monitor	Launch TUI monitoring via SSH	ml monitor
cancel	Cancel running job	ml cancel job123
prune	Clean up old experiments	ml prune --keep 10
watch	Auto-sync directory on changes	ml watch ./project --queue

Command Details

init - Configuration Setup

ml init

Creates a configuration template at ~/.ml/config.toml with: - Worker connection details - API authentication - Base paths and ports

sync - Project Synchronization

# Basic sync
ml sync ./my-project

# Sync with custom name and queue
ml sync ./my-project --name "experiment-1" --queue

# Sync with priority
ml sync ./my-project --priority 9

Features: - Content-addressed storage for deduplication - SHA256 commit ID generation - Rsync-based file transfer - Automatic queuing (with --queue flag)

queue - Job Management

# Queue with commit ID
ml queue my-job --commit abc123def456

# Queue with priority (1-10, default 5)
ml queue my-job --commit abc123 --priority 8

Features: - WebSocket-based communication - Priority queuing system - API key authentication

watch - Auto-Sync Monitoring

# Watch directory for changes
ml watch ./project

# Watch and auto-queue on changes
ml watch ./project --name "dev-exp" --queue

Features: - Real-time file system monitoring - Automatic re-sync on changes - Configurable polling interval (2 seconds) - Commit ID comparison for efficiency

prune - Cleanup Management

# Keep last N experiments
ml prune --keep 20

# Remove experiments older than N days
ml prune --older-than 30

monitor - Remote Monitoring

ml monitor

Launches TUI interface via SSH for real-time monitoring.

cancel - Job Cancellation

ml cancel running-job-id

Cancels currently running jobs by ID.

Configuration

The Zig CLI reads configuration from ~/.ml/config.toml:

worker_host = "worker.local"
worker_user = "mluser"
worker_base = "/data/ml-experiments"
worker_port = 22
api_key = "your-api-key"

Performance Features

• Content-Addressed Storage: Automatic deduplication of identical files
• Incremental Sync: Only transfers changed files
• SHA256 Hashing: Reliable commit ID generation
• WebSocket Communication: Efficient real-time messaging
• Multi-threaded: Concurrent operations where applicable

Go Commands

API Server (./cmd/api-server/main.go)

Main HTTPS API server for experiment management.

# Build and run
go run ./cmd/api-server/main.go

# With configuration
./bin/api-server --config configs/config-local.yaml

Features: - HTTPS-only communication - API key authentication - Rate limiting and IP whitelisting - WebSocket support for real-time updates - Redis integration for caching

TUI (./cmd/tui/main.go)

Terminal User Interface for monitoring experiments.

# Launch TUI
go run ./cmd/tui/main.go

# With custom config
./tui --config configs/config-local.yaml

Features: - Real-time experiment monitoring - Interactive job management - Status visualization - Log viewing

Data Manager (./cmd/data_manager/)

Utilities for data synchronization and management.

# Sync data
./data_manager --sync ./data

# Clean old data
./data_manager --cleanup --older-than 30d

Config Lint (./cmd/configlint/main.go)

Configuration validation and linting tool.

# Validate configuration
./configlint configs/config-local.yaml

# Check schema compliance
./configlint --schema configs/schema/config_schema.yaml

Management Script (./tools/manage.sh)

Simple service management for your homelab.

Commands

./tools/manage.sh start          # Start all services
./tools/manage.sh stop           # Stop all services
./tools/manage.sh status         # Check service status
./tools/manage.sh logs           # View logs
./tools/manage.sh monitor        # Basic monitoring
./tools/manage.sh security       # Security status
./tools/manage.sh cleanup        # Clean project artifacts

Setup Script (./setup.sh)

One-command homelab setup.

Usage

# Full setup
./setup.sh

# Setup includes:
# - SSL certificate generation
# - Configuration creation
# - Build all components
# - Start Redis
# - Setup Fail2Ban (if available)

API Testing

Test the API with curl:

# Health check
curl -k -H 'X-API-Key: password' https://localhost:9101/health

# List experiments
curl -k -H 'X-API-Key: password' https://localhost:9101/experiments

# Submit experiment
curl -k -X POST -H 'X-API-Key: password' \
     -H 'Content-Type: application/json' \
     -d '{"name":"test","config":{"type":"basic"}}' \
     https://localhost:9101/experiments

Zig CLI Architecture

The Zig CLI is designed for performance and reliability:

Core Components

• Commands (cli/src/commands/): Individual command implementations
• Config (cli/src/config.zig): Configuration management
• Network (cli/src/net/ws.zig): WebSocket client implementation
• Utils (cli/src/utils/): Cryptography, storage, and rsync utilities
• Errors (cli/src/errors.zig): Centralized error handling

Performance Optimizations

• Content-Addressed Storage: Deduplicates identical files across experiments
• SHA256 Hashing: Fast, reliable commit ID generation
• Rsync Integration: Efficient incremental file transfers
• WebSocket Protocol: Low-latency communication with worker
• Memory Management: Efficient allocation with Zig's allocator system

Security Features

• API Key Hashing: Secure authentication token handling
• SSH Integration: Secure file transfers
• Input Validation: Comprehensive argument checking
• Error Handling: Secure error reporting without information leakage

Configuration

Main configuration file: configs/config-local.yaml

Key Settings

auth:
  enabled: true
  api_keys:
    homelab_user:
      hash: "5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8"
      admin: true

server:
  address: ":9101"
  tls:
    enabled: true
    cert_file: "./ssl/cert.pem"
    key_file: "./ssl/key.pem"

security:
  rate_limit:
    enabled: true
    requests_per_minute: 30
  ip_whitelist:
    - "127.0.0.1"
    - "::1"
    - "192.168.0.0/16"
    - "10.0.0.0/8"

Docker Commands

If using Docker Compose:

# Start services
docker-compose up -d (testing only)

# View logs
docker-compose logs -f

# Stop services
docker-compose down

# Check status
docker-compose ps

Troubleshooting

Common Issues

Zig CLI not found:

# Build the CLI
cd cli && make build

# Check binary exists
ls -la ./cli/zig-out/bin/ml

Configuration not found:

# Create configuration
./cli/zig-out/bin/ml init

# Check config file
ls -la ~/.ml/config.toml

Worker connection failed:

# Test SSH connection
ssh -p 22 mluser@worker.local

# Check configuration
cat ~/.ml/config.toml

Sync not working:

# Check rsync availability
rsync --version

# Test manual sync
rsync -avz ./project/ mluser@worker.local:/tmp/test/

WebSocket connection failed:

# Check worker WebSocket port
telnet worker.local 9100

# Verify API key
./cli/zig-out/bin/ml status

API not responding:

./tools/manage.sh status
./tools/manage.sh logs

Authentication failed:

# Check API key in config-local.yaml
grep -A 5 "api_keys:" configs/config-local.yaml

Redis connection failed:

# Check Redis status
redis-cli ping

# Start Redis
redis-server

Getting Help

# CLI help
./cli/zig-out/bin/ml help

# Management script help
./tools/manage.sh help

# Check all available commands
make help

---

That's it for the CLI reference! For complete setup instructions, see the main index.