jfraeysd/fetch_ml
0
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
fetch_ml/docs/_site/zig-cli/index.html
Jeremie Fraeys ea15af1833 Fix multi-user authentication and clean up debug code 
- Fix YAML tags in auth config struct (json -> yaml)
- Update CLI configs to use pre-hashed API keys
- Remove double hashing in WebSocket client
- Fix port mapping (9102 -> 9103) in CLI commands
- Update permission keys to use jobs:read, jobs:create, etc.
- Clean up all debug logging from CLI and server
- All user roles now authenticate correctly:
  * Admin: Can queue jobs and see all jobs
  * Researcher: Can queue jobs and see own jobs
  * Analyst: Can see status (read-only access)

Multi-user authentication is now fully functional.
2025-12-06 12:35:32 -05:00

2679 lines
No EOL
64 KiB
HTML
Raw Blame History

<!doctype html>
<html lang="en" class="no-js">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<meta name="description" content="Secure Machine Learning Platform">
<link rel="prev" href="../cli-reference/">
<link rel="next" href="../queue/">
<link rel="icon" href="../assets/images/favicon.png">
<meta name="generator" content="mkdocs-1.6.1, mkdocs-material-9.7.0">
<title>Zig CLI Guide - Fetch ML Documentation</title>
<link rel="stylesheet" href="../assets/stylesheets/main.618322db.min.css">
<link rel="stylesheet" href="../assets/stylesheets/palette.ab4e12ef.min.css">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,300i,400,400i,700,700i%7CRoboto+Mono:400,400i,700,700i&display=fallback">
<style>:root{--md-text-font:"Roboto";--md-code-font:"Roboto Mono"}</style>
<script>__md_scope=new URL("..",location),__md_hash=e=>[...e].reduce(((e,_)=>(e<<5)-e+_.charCodeAt(0)),0),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script>
</head>
<body dir="ltr" data-md-color-scheme="default" data-md-color-primary="blue" data-md-color-accent="blue">
<input class="md-toggle" data-md-toggle="drawer" type="checkbox" id="__drawer" autocomplete="off">
<input class="md-toggle" data-md-toggle="search" type="checkbox" id="__search" autocomplete="off">
<label class="md-overlay" for="__drawer"></label>
<div data-md-component="skip">
<a href="#zig-cli-guide" class="md-skip">
Skip to content
</a>
</div>
<div data-md-component="announce">
</div>
<header class="md-header" data-md-component="header">
<nav class="md-header__inner md-grid" aria-label="Header">
<a href=".." title="Fetch ML Documentation" class="md-header__button md-logo" aria-label="Fetch ML Documentation" data-md-component="logo">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
</a>
<label class="md-header__button md-icon" for="__drawer">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M3 6h18v2H3zm0 5h18v2H3zm0 5h18v2H3z"/></svg>
</label>
<div class="md-header__title" data-md-component="header-title">
<div class="md-header__ellipsis">
<div class="md-header__topic">
<span class="md-ellipsis">
Fetch ML Documentation
</span>
</div>
<div class="md-header__topic" data-md-component="header-topic">
<span class="md-ellipsis">
Zig CLI Guide
</span>
</div>
</div>
</div>
<form class="md-header__option" data-md-component="palette">
<input class="md-option" data-md-color-media="" data-md-color-scheme="default" data-md-color-primary="blue" data-md-color-accent="blue" aria-hidden="true" type="radio" name="__palette" id="__palette_0">
<input class="md-option" data-md-color-media="" data-md-color-scheme="slate" data-md-color-primary="blue" data-md-color-accent="blue" aria-hidden="true" type="radio" name="__palette" id="__palette_1">
</form>
<script>var palette=__md_get("__palette");if(palette&&palette.color){if("(prefers-color-scheme)"===palette.color.media){var media=matchMedia("(prefers-color-scheme: light)"),input=document.querySelector(media.matches?"[data-md-color-media='(prefers-color-scheme: light)']":"[data-md-color-media='(prefers-color-scheme: dark)']");palette.color.media=input.getAttribute("data-md-color-media"),palette.color.scheme=input.getAttribute("data-md-color-scheme"),palette.color.primary=input.getAttribute("data-md-color-primary"),palette.color.accent=input.getAttribute("data-md-color-accent")}for(var[key,value]of Object.entries(palette.color))document.body.setAttribute("data-md-color-"+key,value)}</script>
<label class="md-header__button md-icon" for="__search">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M9.5 3A6.5 6.5 0 0 1 16 9.5c0 1.61-.59 3.09-1.56 4.23l.27.27h.79l5 5-1.5 1.5-5-5v-.79l-.27-.27A6.52 6.52 0 0 1 9.5 16 6.5 6.5 0 0 1 3 9.5 6.5 6.5 0 0 1 9.5 3m0 2C7 5 5 7 5 9.5S7 14 9.5 14 14 12 14 9.5 12 5 9.5 5"/></svg>
</label>
<div class="md-search" data-md-component="search" role="dialog">
<label class="md-search__overlay" for="__search"></label>
<div class="md-search__inner" role="search">
<form class="md-search__form" name="search">
<input type="text" class="md-search__input" name="query" aria-label="Search" placeholder="Search" autocapitalize="off" autocorrect="off" autocomplete="off" spellcheck="false" data-md-component="search-query" required>
<label class="md-search__icon md-icon" for="__search">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M9.5 3A6.5 6.5 0 0 1 16 9.5c0 1.61-.59 3.09-1.56 4.23l.27.27h.79l5 5-1.5 1.5-5-5v-.79l-.27-.27A6.52 6.52 0 0 1 9.5 16 6.5 6.5 0 0 1 3 9.5 6.5 6.5 0 0 1 9.5 3m0 2C7 5 5 7 5 9.5S7 14 9.5 14 14 12 14 9.5 12 5 9.5 5"/></svg>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20 11v2H8l5.5 5.5-1.42 1.42L4.16 12l7.92-7.92L13.5 5.5 8 11z"/></svg>
</label>
<nav class="md-search__options" aria-label="Search">
<a href="javascript:void(0)" class="md-search__icon md-icon" title="Share" aria-label="Share" data-clipboard data-clipboard-text="" data-md-component="search-share" tabindex="-1">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M18 16.08c-.76 0-1.44.3-1.96.77L8.91 12.7c.05-.23.09-.46.09-.7s-.04-.47-.09-.7l7.05-4.11c.54.5 1.25.81 2.04.81a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3c0 .24.04.47.09.7L8.04 9.81C7.5 9.31 6.79 9 6 9a3 3 0 0 0-3 3 3 3 0 0 0 3 3c.79 0 1.5-.31 2.04-.81l7.12 4.15c-.05.21-.08.43-.08.66 0 1.61 1.31 2.91 2.92 2.91s2.92-1.3 2.92-2.91A2.92 2.92 0 0 0 18 16.08"/></svg>
</a>
<button type="reset" class="md-search__icon md-icon" title="Clear" aria-label="Clear" tabindex="-1">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M19 6.41 17.59 5 12 10.59 6.41 5 5 6.41 10.59 12 5 17.59 6.41 19 12 13.41 17.59 19 19 17.59 13.41 12z"/></svg>
</button>
</nav>
</form>
<div class="md-search__output">
<div class="md-search__scrollwrap" tabindex="0" data-md-scrollfix>
<div class="md-search-result" data-md-component="search-result">
<div class="md-search-result__meta">
Initializing search
</div>
<ol class="md-search-result__list" role="presentation"></ol>
</div>
</div>
</div>
</div>
</div>
<div class="md-header__source">
<a href="https://github.com/jfraeys/fetch_ml" title="Go to repository" class="md-source" data-md-component="source">
<div class="md-source__icon md-icon">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 7.1.0 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2025 Fonticons, Inc.--><path d="M439.6 236.1 244 40.5c-5.4-5.5-12.8-8.5-20.4-8.5s-15 3-20.4 8.4L162.5 81l51.5 51.5c27.1-9.1 52.7 16.8 43.4 43.7l49.7 49.7c34.2-11.8 61.2 31 35.5 56.7-26.5 26.5-70.2-2.9-56-37.3L240.3 199v121.9c25.3 12.5 22.3 41.8 9.1 55-6.4 6.4-15.2 10.1-24.3 10.1s-17.8-3.6-24.3-10.1c-17.6-17.6-11.1-46.9 11.2-56v-123c-20.8-8.5-24.6-30.7-18.6-45L142.6 101 8.5 235.1C3 240.6 0 247.9 0 255.5s3 15 8.5 20.4l195.6 195.7c5.4 5.4 12.7 8.4 20.4 8.4s15-3 20.4-8.4l194.7-194.7c5.4-5.4 8.4-12.8 8.4-20.4s-3-15-8.4-20.4"/></svg>
</div>
<div class="md-source__repository">
GitHub
</div>
</a>
</div>
</nav>
</header>
<div class="md-container" data-md-component="container">
<nav class="md-tabs" aria-label="Tabs" data-md-component="tabs">
<div class="md-grid">
<ul class="md-tabs__list">
<li class="md-tabs__item">
<a href=".." class="md-tabs__link">
Home
</a>
</li>
<li class="md-tabs__item">
<a href="../quick-start/" class="md-tabs__link">
Getting Started
</a>
</li>
<li class="md-tabs__item md-tabs__item--active">
<a href="../development-setup/" class="md-tabs__link">
Development
</a>
</li>
<li class="md-tabs__item">
<a href="../deployment/" class="md-tabs__link">
Operations & Production
</a>
</li>
<li class="md-tabs__item">
<a href="../security/" class="md-tabs__link">
Security
</a>
</li>
<li class="md-tabs__item">
<a href="../configuration-schema/" class="md-tabs__link">
Reference
</a>
</li>
<li class="md-tabs__item">
<a href="../adr/" class="md-tabs__link">
Architecture Decisions
</a>
</li>
</ul>
</div>
</nav>
<main class="md-main" data-md-component="main">
<div class="md-main__inner md-grid">
<div class="md-sidebar md-sidebar--primary" data-md-component="sidebar" data-md-type="navigation" >
<div class="md-sidebar__scrollwrap">
<div class="md-sidebar__inner">
<nav class="md-nav md-nav--primary md-nav--lifted md-nav--integrated" aria-label="Navigation" data-md-level="0">
<label class="md-nav__title" for="__drawer">
<a href=".." title="Fetch ML Documentation" class="md-nav__button md-logo" aria-label="Fetch ML Documentation" data-md-component="logo">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
</a>
Fetch ML Documentation
</label>
<div class="md-nav__source">
<a href="https://github.com/jfraeys/fetch_ml" title="Go to repository" class="md-source" data-md-component="source">
<div class="md-source__icon md-icon">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 7.1.0 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2025 Fonticons, Inc.--><path d="M439.6 236.1 244 40.5c-5.4-5.5-12.8-8.5-20.4-8.5s-15 3-20.4 8.4L162.5 81l51.5 51.5c27.1-9.1 52.7 16.8 43.4 43.7l49.7 49.7c34.2-11.8 61.2 31 35.5 56.7-26.5 26.5-70.2-2.9-56-37.3L240.3 199v121.9c25.3 12.5 22.3 41.8 9.1 55-6.4 6.4-15.2 10.1-24.3 10.1s-17.8-3.6-24.3-10.1c-17.6-17.6-11.1-46.9 11.2-56v-123c-20.8-8.5-24.6-30.7-18.6-45L142.6 101 8.5 235.1C3 240.6 0 247.9 0 255.5s3 15 8.5 20.4l195.6 195.7c5.4 5.4 12.7 8.4 20.4 8.4s15-3 20.4-8.4l194.7-194.7c5.4-5.4 8.4-12.8 8.4-20.4s-3-15-8.4-20.4"/></svg>
</div>
<div class="md-source__repository">
GitHub
</div>
</a>
</div>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href=".." class="md-nav__link">
<span class="md-ellipsis">
Home
</span>
</a>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle md-toggle--indeterminate" type="checkbox" id="__nav_2" >
<label class="md-nav__link" for="__nav_2" id="__nav_2_label" tabindex="0">
<span class="md-ellipsis">
Getting Started
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_2">
<span class="md-nav__icon md-icon"></span>
Getting Started
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../quick-start/" class="md-nav__link">
<span class="md-ellipsis">
Quick Start
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../installation/" class="md-nav__link">
<span class="md-ellipsis">
Simple Installation Guide
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../first-experiment/" class="md-nav__link">
<span class="md-ellipsis">
First Experiment
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--active md-nav__item--section md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3" checked>
<label class="md-nav__link" for="__nav_3" id="__nav_3_label" tabindex="">
<span class="md-ellipsis">
Development
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_3_label" aria-expanded="true">
<label class="md-nav__title" for="__nav_3">
<span class="md-nav__icon md-icon"></span>
Development
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../development-setup/" class="md-nav__link">
<span class="md-ellipsis">
Development Setup
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../testing/" class="md-nav__link">
<span class="md-ellipsis">
Testing Guide
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../architecture/" class="md-nav__link">
<span class="md-ellipsis">
Homelab Architecture
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../cli-reference/" class="md-nav__link">
<span class="md-ellipsis">
CLI Reference
</span>
</a>
</li>
<li class="md-nav__item md-nav__item--active">
<input class="md-nav__toggle md-toggle" type="checkbox" id="__toc">
<label class="md-nav__link md-nav__link--active" for="__toc">
<span class="md-ellipsis">
Zig CLI Guide
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<a href="./" class="md-nav__link md-nav__link--active">
<span class="md-ellipsis">
Zig CLI Guide
</span>
</a>
<nav class="md-nav md-nav--secondary" aria-label="Table of contents">
<label class="md-nav__title" for="__toc">
<span class="md-nav__icon md-icon"></span>
Table of contents
</label>
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
<li class="md-nav__item">
<a href="#overview" class="md-nav__link">
<span class="md-ellipsis">
Overview
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#installation" class="md-nav__link">
<span class="md-ellipsis">
Installation
</span>
</a>
<nav class="md-nav" aria-label="Installation">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#pre-built-binaries-recommended" class="md-nav__link">
<span class="md-ellipsis">
Pre-built Binaries (Recommended)
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#build-from-source" class="md-nav__link">
<span class="md-ellipsis">
Build from Source
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#verify-installation" class="md-nav__link">
<span class="md-ellipsis">
Verify Installation
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="#quick-start" class="md-nav__link">
<span class="md-ellipsis">
Quick Start
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#command-reference" class="md-nav__link">
<span class="md-ellipsis">
Command Reference
</span>
</a>
<nav class="md-nav" aria-label="Command Reference">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#init-configuration-setup" class="md-nav__link">
<span class="md-ellipsis">
init - Configuration Setup
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#sync-project-synchronization" class="md-nav__link">
<span class="md-ellipsis">
sync - Project Synchronization
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#queue-job-management" class="md-nav__link">
<span class="md-ellipsis">
queue - Job Management
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#watch-auto-sync-monitoring" class="md-nav__link">
<span class="md-ellipsis">
watch - Auto-Sync Monitoring
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#status-system-status" class="md-nav__link">
<span class="md-ellipsis">
status - System Status
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#monitor-remote-monitoring" class="md-nav__link">
<span class="md-ellipsis">
monitor - Remote Monitoring
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#cancel-job-cancellation" class="md-nav__link">
<span class="md-ellipsis">
cancel - Job Cancellation
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#prune-cleanup-management" class="md-nav__link">
<span class="md-ellipsis">
prune - Cleanup Management
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="#architecture" class="md-nav__link">
<span class="md-ellipsis">
Architecture
</span>
</a>
<nav class="md-nav" aria-label="Architecture">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#core-components" class="md-nav__link">
<span class="md-ellipsis">
Core Components
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#performance-features" class="md-nav__link">
<span class="md-ellipsis">
Performance Features
</span>
</a>
<nav class="md-nav" aria-label="Performance Features">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#content-addressed-storage" class="md-nav__link">
<span class="md-ellipsis">
Content-Addressed Storage
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#sha256-commit-ids" class="md-nav__link">
<span class="md-ellipsis">
SHA256 Commit IDs
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#websocket-protocol" class="md-nav__link">
<span class="md-ellipsis">
WebSocket Protocol
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#memory-management" class="md-nav__link">
<span class="md-ellipsis">
Memory Management
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="#security-features" class="md-nav__link">
<span class="md-ellipsis">
Security Features
</span>
</a>
<nav class="md-nav" aria-label="Security Features">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#authentication" class="md-nav__link">
<span class="md-ellipsis">
Authentication
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#secure-communication" class="md-nav__link">
<span class="md-ellipsis">
Secure Communication
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#error-handling" class="md-nav__link">
<span class="md-ellipsis">
Error Handling
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="#advanced-usage" class="md-nav__link">
<span class="md-ellipsis">
Advanced Usage
</span>
</a>
<nav class="md-nav" aria-label="Advanced Usage">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#workflow-integration" class="md-nav__link">
<span class="md-ellipsis">
Workflow Integration
</span>
</a>
<nav class="md-nav" aria-label="Workflow Integration">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#development-workflow" class="md-nav__link">
<span class="md-ellipsis">
Development Workflow
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#batch-processing" class="md-nav__link">
<span class="md-ellipsis">
Batch Processing
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#priority-management" class="md-nav__link">
<span class="md-ellipsis">
Priority Management
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="#configuration-management" class="md-nav__link">
<span class="md-ellipsis">
Configuration Management
</span>
</a>
<nav class="md-nav" aria-label="Configuration Management">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#multiple-workers" class="md-nav__link">
<span class="md-ellipsis">
Multiple Workers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#security-settings" class="md-nav__link">
<span class="md-ellipsis">
Security Settings
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="#troubleshooting" class="md-nav__link">
<span class="md-ellipsis">
Troubleshooting
</span>
</a>
<nav class="md-nav" aria-label="Troubleshooting">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#common-issues" class="md-nav__link">
<span class="md-ellipsis">
Common Issues
</span>
</a>
<nav class="md-nav" aria-label="Common Issues">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#build-problems" class="md-nav__link">
<span class="md-ellipsis">
Build Problems
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#connection-issues" class="md-nav__link">
<span class="md-ellipsis">
Connection Issues
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#sync-failures" class="md-nav__link">
<span class="md-ellipsis">
Sync Failures
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#performance-issues" class="md-nav__link">
<span class="md-ellipsis">
Performance Issues
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="#debug-mode" class="md-nav__link">
<span class="md-ellipsis">
Debug Mode
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="#performance-benchmarks" class="md-nav__link">
<span class="md-ellipsis">
Performance Benchmarks
</span>
</a>
<nav class="md-nav" aria-label="Performance Benchmarks">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#file-operations" class="md-nav__link">
<span class="md-ellipsis">
File Operations
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#memory-usage" class="md-nav__link">
<span class="md-ellipsis">
Memory Usage
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#network-performance" class="md-nav__link">
<span class="md-ellipsis">
Network Performance
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="#contributing" class="md-nav__link">
<span class="md-ellipsis">
Contributing
</span>
</a>
<nav class="md-nav" aria-label="Contributing">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#development-setup" class="md-nav__link">
<span class="md-ellipsis">
Development Setup
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#testing" class="md-nav__link">
<span class="md-ellipsis">
Testing
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#code-style" class="md-nav__link">
<span class="md-ellipsis">
Code Style
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="../queue/" class="md-nav__link">
<span class="md-ellipsis">
Task Queue Architecture
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../smart-defaults/" class="md-nav__link">
<span class="md-ellipsis">
Smart Defaults
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../cicd/" class="md-nav__link">
<span class="md-ellipsis">
CI/CD Pipeline
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle md-toggle--indeterminate" type="checkbox" id="__nav_4" >
<label class="md-nav__link" for="__nav_4" id="__nav_4_label" tabindex="0">
<span class="md-ellipsis">
Operations & Production
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_4_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_4">
<span class="md-nav__icon md-icon"></span>
Operations & Production
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../deployment/" class="md-nav__link">
<span class="md-ellipsis">
ML Experiment Manager - Deployment Guide
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../environment-variables/" class="md-nav__link">
<span class="md-ellipsis">
Environment Variables
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../production-monitoring/" class="md-nav__link">
<span class="md-ellipsis">
Production Monitoring Deployment Guide (Linux)
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../operations/" class="md-nav__link">
<span class="md-ellipsis">
Operations Runbook
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../redis-ha/" class="md-nav__link">
<span class="md-ellipsis">
Redis High Availability (Optional)
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../release-checklist/" class="md-nav__link">
<span class="md-ellipsis">
Release Checklist
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle md-toggle--indeterminate" type="checkbox" id="__nav_5" >
<label class="md-nav__link" for="__nav_5" id="__nav_5_label" tabindex="0">
<span class="md-ellipsis">
Security
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_5_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_5">
<span class="md-nav__icon md-icon"></span>
Security
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../security/" class="md-nav__link">
<span class="md-ellipsis">
Security Guide
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../api-key-process/" class="md-nav__link">
<span class="md-ellipsis">
FetchML API Key Process
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../user-permissions/" class="md-nav__link">
<span class="md-ellipsis">
User Permissions in Fetch ML
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle md-toggle--indeterminate" type="checkbox" id="__nav_6" >
<label class="md-nav__link" for="__nav_6" id="__nav_6_label" tabindex="0">
<span class="md-ellipsis">
Reference
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_6_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_6">
<span class="md-nav__icon md-icon"></span>
Reference
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../configuration-schema/" class="md-nav__link">
<span class="md-ellipsis">
Configuration Schema
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../troubleshooting/" class="md-nav__link">
<span class="md-ellipsis">
Troubleshooting
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle md-toggle--indeterminate" type="checkbox" id="__nav_7" >
<div class="md-nav__link md-nav__container">
<a href="../adr/" class="md-nav__link ">
<span class="md-ellipsis">
Architecture Decisions
</span>
</a>
<label class="md-nav__link " for="__nav_7" id="__nav_7_label" tabindex="0">
<span class="md-nav__icon md-icon"></span>
</label>
</div>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_7_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_7">
<span class="md-nav__icon md-icon"></span>
Architecture Decisions
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../adr/ADR-001-use-go-for-api-server/" class="md-nav__link">
<span class="md-ellipsis">
ADR-001: Use Go for API Server
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../adr/ADR-002-use-sqlite-for-local-development/" class="md-nav__link">
<span class="md-ellipsis">
ADR-002: Use SQLite for Local Development
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../adr/ADR-003-use-redis-for-job-queue/" class="md-nav__link">
<span class="md-ellipsis">
ADR-003: Use Redis for Job Queue
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</div>
</div>
</div>
<div class="md-content" data-md-component="content">
<article class="md-content__inner md-typeset">
<h1 id="zig-cli-guide">Zig CLI Guide<a class="headerlink" href="#zig-cli-guide" title="Permanent link">&para;</a></h1>
<p>High-performance command-line interface for ML experiment management, written in Zig for maximum speed and efficiency.</p>
<h2 id="overview">Overview<a class="headerlink" href="#overview" title="Permanent link">&para;</a></h2>
<p>The Zig CLI (<code>ml</code>) is the primary interface for managing ML experiments in your homelab. Built with Zig, it provides exceptional performance for file operations, network communication, and experiment management.</p>
<h2 id="installation">Installation<a class="headerlink" href="#installation" title="Permanent link">&para;</a></h2>
<h3 id="pre-built-binaries-recommended">Pre-built Binaries (Recommended)<a class="headerlink" href="#pre-built-binaries-recommended" title="Permanent link">&para;</a></h3>
<p>Download from <a href="https://github.com/jfraeys/fetch_ml/releases">GitHub Releases</a>:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># Download for your platform</span>
curl<span class="w"> </span>-LO<span class="w"> </span>https://github.com/jfraeys/fetch_ml/releases/latest/download/ml-&lt;platform&gt;.tar.gz
<span class="c1"># Extract</span>
tar<span class="w"> </span>-xzf<span class="w"> </span>ml-&lt;platform&gt;.tar.gz
<span class="c1"># Install</span>
chmod<span class="w"> </span>+x<span class="w"> </span>ml-&lt;platform&gt;
sudo<span class="w"> </span>mv<span class="w"> </span>ml-&lt;platform&gt;<span class="w"> </span>/usr/local/bin/ml
<span class="c1"># Verify</span>
ml<span class="w"> </span>--help
</code></pre></div>
<p><strong>Platforms:</strong>
- <code>ml-linux-x86_64.tar.gz</code> - Linux (fully static, zero dependencies)
- <code>ml-macos-x86_64.tar.gz</code> - macOS Intel<br />
- <code>ml-macos-arm64.tar.gz</code> - macOS Apple Silicon</p>
<p>All release binaries include <strong>embedded static rsync</strong> for complete independence.</p>
<h3 id="build-from-source">Build from Source<a class="headerlink" href="#build-from-source" title="Permanent link">&para;</a></h3>
<p><strong>Development Build</strong> (uses system rsync):
<div class="highlight"><pre><span></span><code><span class="nb">cd</span><span class="w"> </span>cli
zig<span class="w"> </span>build<span class="w"> </span>dev
./zig-out/dev/ml-dev<span class="w"> </span>--help
</code></pre></div></p>
<p><strong>Production Build</strong> (embedded rsync):
<div class="highlight"><pre><span></span><code><span class="nb">cd</span><span class="w"> </span>cli
<span class="c1"># For testing: uses rsync wrapper</span>
zig<span class="w"> </span>build<span class="w"> </span>prod
<span class="c1"># For release with static rsync:</span>
<span class="c1"># 1. Place static rsync binary at src/assets/rsync_release.bin</span>
<span class="c1"># 2. Build</span>
zig<span class="w"> </span>build<span class="w"> </span>prod
strip<span class="w"> </span>zig-out/prod/ml<span class="w"> </span><span class="c1"># Optional: reduce size</span>
<span class="c1"># Verify</span>
./zig-out/prod/ml<span class="w"> </span>--help
ls<span class="w"> </span>-lh<span class="w"> </span>zig-out/prod/ml
</code></pre></div></p>
<p>See <a href="https://github.com/jfraeys/fetch_ml/blob/main/cli/src/assets/README.md">cli/src/assets/README.md</a> for details on obtaining static rsync binaries.</p>
<h3 id="verify-installation">Verify Installation<a class="headerlink" href="#verify-installation" title="Permanent link">&para;</a></h3>
<div class="highlight"><pre><span></span><code>ml<span class="w"> </span>--help
ml<span class="w"> </span>--version<span class="w"> </span><span class="c1"># Shows build config</span>
</code></pre></div>
<h2 id="quick-start">Quick Start<a class="headerlink" href="#quick-start" title="Permanent link">&para;</a></h2>
<ol>
<li>
<p><strong>Initialize Configuration</strong>
<div class="highlight"><pre><span></span><code>./cli/zig-out/bin/ml<span class="w"> </span>init
</code></pre></div></p>
</li>
<li>
<p><strong>Sync Your First Project</strong>
<div class="highlight"><pre><span></span><code>./cli/zig-out/bin/ml<span class="w"> </span>sync<span class="w"> </span>./my-project<span class="w"> </span>--queue
</code></pre></div></p>
</li>
<li>
<p><strong>Monitor Progress</strong>
<div class="highlight"><pre><span></span><code>./cli/zig-out/bin/ml<span class="w"> </span>status
</code></pre></div></p>
</li>
</ol>
<h2 id="command-reference">Command Reference<a class="headerlink" href="#command-reference" title="Permanent link">&para;</a></h2>
<h3 id="init-configuration-setup"><code>init</code> - Configuration Setup<a class="headerlink" href="#init-configuration-setup" title="Permanent link">&para;</a></h3>
<p>Initialize the CLI configuration file.</p>
<div class="highlight"><pre><span></span><code>ml<span class="w"> </span>init
</code></pre></div>
<p><strong>Creates:</strong> <code>~/.ml/config.toml</code></p>
<p><strong>Configuration Template:</strong>
<div class="highlight"><pre><span></span><code><span class="n">worker_host</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&quot;worker.local&quot;</span>
<span class="n">worker_user</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&quot;mluser&quot;</span>
<span class="n">worker_base</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&quot;/data/ml-experiments&quot;</span>
<span class="n">worker_port</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">22</span>
<span class="n">api_key</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&quot;your-api-key&quot;</span>
</code></pre></div></p>
<h3 id="sync-project-synchronization"><code>sync</code> - Project Synchronization<a class="headerlink" href="#sync-project-synchronization" title="Permanent link">&para;</a></h3>
<p>Sync project files to the worker with intelligent deduplication.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># Basic sync</span>
ml<span class="w"> </span>sync<span class="w"> </span>./project
<span class="c1"># Sync with custom name and auto-queue</span>
ml<span class="w"> </span>sync<span class="w"> </span>./project<span class="w"> </span>--name<span class="w"> </span><span class="s2">&quot;experiment-1&quot;</span><span class="w"> </span>--queue
<span class="c1"># Sync with priority</span>
ml<span class="w"> </span>sync<span class="w"> </span>./project<span class="w"> </span>--priority<span class="w"> </span><span class="m">8</span>
</code></pre></div>
<p><strong>Options:</strong>
- <code>--name &lt;name&gt;</code>: Custom experiment name
- <code>--queue</code>: Automatically queue after sync
- <code>--priority N</code>: Set priority (1-10, default 5)</p>
<p><strong>Features:</strong>
- <strong>Content-Addressed Storage</strong>: Automatic deduplication
- <strong>SHA256 Commit IDs</strong>: Reliable change detection
- <strong>Incremental Transfer</strong>: Only sync changed files
- <strong>Rsync Backend</strong>: Efficient file transfer</p>
<h3 id="queue-job-management"><code>queue</code> - Job Management<a class="headerlink" href="#queue-job-management" title="Permanent link">&para;</a></h3>
<p>Queue experiments for execution on the worker.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># Queue with commit ID</span>
ml<span class="w"> </span>queue<span class="w"> </span>my-job<span class="w"> </span>--commit<span class="w"> </span>abc123def456
<span class="c1"># Queue with priority</span>
ml<span class="w"> </span>queue<span class="w"> </span>my-job<span class="w"> </span>--commit<span class="w"> </span>abc123<span class="w"> </span>--priority<span class="w"> </span><span class="m">8</span>
</code></pre></div>
<p><strong>Options:</strong>
- <code>--commit &lt;id&gt;</code>: Commit ID from sync output
- <code>--priority N</code>: Execution priority (1-10)</p>
<p><strong>Features:</strong>
- <strong>WebSocket Communication</strong>: Real-time job submission
- <strong>Priority Queuing</strong>: Higher priority jobs run first
- <strong>API Authentication</strong>: Secure job submission</p>
<h3 id="watch-auto-sync-monitoring"><code>watch</code> - Auto-Sync Monitoring<a class="headerlink" href="#watch-auto-sync-monitoring" title="Permanent link">&para;</a></h3>
<p>Monitor directories for changes and auto-sync.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># Watch for changes</span>
ml<span class="w"> </span>watch<span class="w"> </span>./project
<span class="c1"># Watch and auto-queue on changes</span>
ml<span class="w"> </span>watch<span class="w"> </span>./project<span class="w"> </span>--name<span class="w"> </span><span class="s2">&quot;dev-exp&quot;</span><span class="w"> </span>--queue
</code></pre></div>
<p><strong>Options:</strong>
- <code>--name &lt;name&gt;</code>: Custom experiment name
- <code>--queue</code>: Auto-queue on changes
- <code>--priority N</code>: Set priority for queued jobs</p>
<p><strong>Features:</strong>
- <strong>Real-time Monitoring</strong>: 2-second polling interval
- <strong>Change Detection</strong>: File modification time tracking
- <strong>Commit Comparison</strong>: Only sync when content changes
- <strong>Automatic Queuing</strong>: Seamless development workflow</p>
<h3 id="status-system-status"><code>status</code> - System Status<a class="headerlink" href="#status-system-status" title="Permanent link">&para;</a></h3>
<p>Check system and worker status.</p>
<div class="highlight"><pre><span></span><code>ml<span class="w"> </span>status
</code></pre></div>
<p><strong>Displays:</strong>
- Worker connectivity
- Queue status
- Running jobs
- System health</p>
<h3 id="monitor-remote-monitoring"><code>monitor</code> - Remote Monitoring<a class="headerlink" href="#monitor-remote-monitoring" title="Permanent link">&para;</a></h3>
<p>Launch TUI interface via SSH for real-time monitoring.</p>
<div class="highlight"><pre><span></span><code>ml<span class="w"> </span>monitor
</code></pre></div>
<p><strong>Features:</strong>
- <strong>Real-time Updates</strong>: Live experiment status
- <strong>Interactive Interface</strong>: Browse and manage experiments
- <strong>SSH Integration</strong>: Secure remote access</p>
<h3 id="cancel-job-cancellation"><code>cancel</code> - Job Cancellation<a class="headerlink" href="#cancel-job-cancellation" title="Permanent link">&para;</a></h3>
<p>Cancel running or queued jobs.</p>
<div class="highlight"><pre><span></span><code>ml<span class="w"> </span>cancel<span class="w"> </span>job-id
</code></pre></div>
<p><strong>Options:</strong>
- <code>job-id</code>: Job identifier from status output</p>
<h3 id="prune-cleanup-management"><code>prune</code> - Cleanup Management<a class="headerlink" href="#prune-cleanup-management" title="Permanent link">&para;</a></h3>
<p>Clean up old experiments to save space.</p>
<div class="highlight"><pre><span></span><code><span class="c1"># Keep last N experiments</span>
ml<span class="w"> </span>prune<span class="w"> </span>--keep<span class="w"> </span><span class="m">20</span>
<span class="c1"># Remove experiments older than N days</span>
ml<span class="w"> </span>prune<span class="w"> </span>--older-than<span class="w"> </span><span class="m">30</span>
</code></pre></div>
<p><strong>Options:</strong>
- <code>--keep N</code>: Keep N most recent experiments
- <code>--older-than N</code>: Remove experiments older than N days</p>
<h2 id="architecture">Architecture<a class="headerlink" href="#architecture" title="Permanent link">&para;</a></h2>
<p><strong>Testing</strong>: Docker Compose (macOS/Linux)
<strong>Production</strong>: Podman + systemd (Linux)</p>
<p><strong>Important</strong>: Docker is for testing only. Podman is used for running actual ML experiments in production.</p>
<h3 id="core-components">Core Components<a class="headerlink" href="#core-components" title="Permanent link">&para;</a></h3>
<div class="highlight"><pre><span></span><code>cli/src/
├── commands/ # Command implementations
│ ├── init.zig # Configuration setup
│ ├── sync.zig # Project synchronization
│ ├── queue.zig # Job management
│ ├── watch.zig # Auto-sync monitoring
│ ├── status.zig # System status
│ ├── monitor.zig # Remote monitoring
│ ├── cancel.zig # Job cancellation
│ └── prune.zig # Cleanup operations
├── config.zig # Configuration management
├── errors.zig # Error handling
├── net/ # Network utilities
│ └── ws.zig # WebSocket client
└── utils/ # Utility functions
├── crypto.zig # Hashing and encryption
├── storage.zig # Content-addressed storage
└── rsync.zig # File synchronization
</code></pre></div>
<h3 id="performance-features">Performance Features<a class="headerlink" href="#performance-features" title="Permanent link">&para;</a></h3>
<h4 id="content-addressed-storage">Content-Addressed Storage<a class="headerlink" href="#content-addressed-storage" title="Permanent link">&para;</a></h4>
<ul>
<li><strong>Deduplication</strong>: Identical files shared across experiments</li>
<li><strong>Hash-based Storage</strong>: Files stored by SHA256 hash</li>
<li><strong>Space Efficiency</strong>: Reduces storage by up to 90%</li>
</ul>
<h4 id="sha256-commit-ids">SHA256 Commit IDs<a class="headerlink" href="#sha256-commit-ids" title="Permanent link">&para;</a></h4>
<ul>
<li><strong>Reliable Detection</strong>: Cryptographic change detection</li>
<li><strong>Collision Resistance</strong>: Guaranteed unique identifiers</li>
<li><strong>Fast Computation</strong>: Optimized for large directories</li>
</ul>
<h4 id="websocket-protocol">WebSocket Protocol<a class="headerlink" href="#websocket-protocol" title="Permanent link">&para;</a></h4>
<ul>
<li><strong>Low Latency</strong>: Real-time communication</li>
<li><strong>Binary Protocol</strong>: Efficient message format</li>
<li><strong>Connection Pooling</strong>: Reused connections</li>
</ul>
<h4 id="memory-management">Memory Management<a class="headerlink" href="#memory-management" title="Permanent link">&para;</a></h4>
<ul>
<li><strong>Arena Allocators</strong>: Efficient memory allocation</li>
<li><strong>Zero-copy Operations</strong>: Minimized memory usage</li>
<li><strong>Resource Cleanup</strong>: Automatic resource management</li>
</ul>
<h3 id="security-features">Security Features<a class="headerlink" href="#security-features" title="Permanent link">&para;</a></h3>
<h4 id="authentication">Authentication<a class="headerlink" href="#authentication" title="Permanent link">&para;</a></h4>
<ul>
<li><strong>API Key Hashing</strong>: Secure token storage</li>
<li><strong>SHA256 Hashes</strong>: Irreversible token protection</li>
<li><strong>Config Validation</strong>: Input sanitization</li>
</ul>
<h4 id="secure-communication">Secure Communication<a class="headerlink" href="#secure-communication" title="Permanent link">&para;</a></h4>
<ul>
<li><strong>SSH Integration</strong>: Encrypted file transfers</li>
<li><strong>WebSocket Security</strong>: TLS-protected communication</li>
<li><strong>Input Validation</strong>: Comprehensive argument checking</li>
</ul>
<h4 id="error-handling">Error Handling<a class="headerlink" href="#error-handling" title="Permanent link">&para;</a></h4>
<ul>
<li><strong>Secure Reporting</strong>: No sensitive information leakage</li>
<li><strong>Graceful Degradation</strong>: Safe error recovery</li>
<li><strong>Audit Logging</strong>: Operation tracking</li>
</ul>
<h2 id="advanced-usage">Advanced Usage<a class="headerlink" href="#advanced-usage" title="Permanent link">&para;</a></h2>
<h3 id="workflow-integration">Workflow Integration<a class="headerlink" href="#workflow-integration" title="Permanent link">&para;</a></h3>
<h4 id="development-workflow">Development Workflow<a class="headerlink" href="#development-workflow" title="Permanent link">&para;</a></h4>
<div class="highlight"><pre><span></span><code><span class="c1"># 1. Initialize project</span>
ml<span class="w"> </span>sync<span class="w"> </span>./project<span class="w"> </span>--name<span class="w"> </span><span class="s2">&quot;dev&quot;</span><span class="w"> </span>--queue
<span class="c1"># 2. Auto-sync during development</span>
ml<span class="w"> </span>watch<span class="w"> </span>./project<span class="w"> </span>--name<span class="w"> </span><span class="s2">&quot;dev&quot;</span><span class="w"> </span>--queue
<span class="c1"># 3. Monitor progress</span>
ml<span class="w"> </span>status
</code></pre></div>
<h4 id="batch-processing">Batch Processing<a class="headerlink" href="#batch-processing" title="Permanent link">&para;</a></h4>
<div class="highlight"><pre><span></span><code><span class="c1"># Process multiple experiments</span>
<span class="k">for</span><span class="w"> </span>dir<span class="w"> </span><span class="k">in</span><span class="w"> </span>experiments/*/<span class="p">;</span><span class="w"> </span><span class="k">do</span>
<span class="w"> </span>ml<span class="w"> </span>sync<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$dir</span><span class="s2">&quot;</span><span class="w"> </span>--queue
<span class="k">done</span>
</code></pre></div>
<h4 id="priority-management">Priority Management<a class="headerlink" href="#priority-management" title="Permanent link">&para;</a></h4>
<div class="highlight"><pre><span></span><code><span class="c1"># High priority experiment</span>
ml<span class="w"> </span>sync<span class="w"> </span>./urgent<span class="w"> </span>--priority<span class="w"> </span><span class="m">10</span><span class="w"> </span>--queue
<span class="c1"># Background processing</span>
ml<span class="w"> </span>sync<span class="w"> </span>./background<span class="w"> </span>--priority<span class="w"> </span><span class="m">1</span><span class="w"> </span>--queue
</code></pre></div>
<h3 id="configuration-management">Configuration Management<a class="headerlink" href="#configuration-management" title="Permanent link">&para;</a></h3>
<h4 id="multiple-workers">Multiple Workers<a class="headerlink" href="#multiple-workers" title="Permanent link">&para;</a></h4>
<div class="highlight"><pre><span></span><code><span class="c1"># ~/.ml/config.toml</span>
<span class="n">worker_host</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&quot;worker.local&quot;</span>
<span class="n">worker_user</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&quot;mluser&quot;</span>
<span class="n">worker_base</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&quot;/data/ml-experiments&quot;</span>
<span class="n">worker_port</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">22</span>
<span class="n">api_key</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&quot;your-api-key&quot;</span>
</code></pre></div>
<h4 id="security-settings">Security Settings<a class="headerlink" href="#security-settings" title="Permanent link">&para;</a></h4>
<div class="highlight"><pre><span></span><code><span class="c1"># Set restrictive permissions</span>
chmod<span class="w"> </span><span class="m">600</span><span class="w"> </span>~/.ml/config.toml
<span class="c1"># Verify configuration</span>
ml<span class="w"> </span>status
</code></pre></div>
<h2 id="troubleshooting">Troubleshooting<a class="headerlink" href="#troubleshooting" title="Permanent link">&para;</a></h2>
<h3 id="common-issues">Common Issues<a class="headerlink" href="#common-issues" title="Permanent link">&para;</a></h3>
<h4 id="build-problems">Build Problems<a class="headerlink" href="#build-problems" title="Permanent link">&para;</a></h4>
<div class="highlight"><pre><span></span><code><span class="c1"># Check Zig installation</span>
zig<span class="w"> </span>version
<span class="c1"># Clean build</span>
<span class="nb">cd</span><span class="w"> </span>cli<span class="w"> </span><span class="o">&amp;&amp;</span><span class="w"> </span>make<span class="w"> </span>clean<span class="w"> </span><span class="o">&amp;&amp;</span><span class="w"> </span>make<span class="w"> </span>build
</code></pre></div>
<h4 id="connection-issues">Connection Issues<a class="headerlink" href="#connection-issues" title="Permanent link">&para;</a></h4>
<div class="highlight"><pre><span></span><code><span class="c1"># Test SSH connectivity</span>
ssh<span class="w"> </span>-p<span class="w"> </span><span class="nv">$worker_port</span><span class="w"> </span><span class="nv">$worker_user</span>@<span class="nv">$worker_host</span>
<span class="c1"># Verify configuration</span>
cat<span class="w"> </span>~/.ml/config.toml
</code></pre></div>
<h4 id="sync-failures">Sync Failures<a class="headerlink" href="#sync-failures" title="Permanent link">&para;</a></h4>
<div class="highlight"><pre><span></span><code><span class="c1"># Check rsync</span>
rsync<span class="w"> </span>--version
<span class="c1"># Manual sync test</span>
rsync<span class="w"> </span>-avz<span class="w"> </span>./test/<span class="w"> </span><span class="nv">$worker_user</span>@<span class="nv">$worker_host</span>:/tmp/
</code></pre></div>
<h4 id="performance-issues">Performance Issues<a class="headerlink" href="#performance-issues" title="Permanent link">&para;</a></h4>
<div class="highlight"><pre><span></span><code><span class="c1"># Monitor resource usage</span>
top<span class="w"> </span>-p<span class="w"> </span><span class="k">$(</span>pgrep<span class="w"> </span>ml<span class="k">)</span>
<span class="c1"># Check disk space</span>
df<span class="w"> </span>-h<span class="w"> </span><span class="nv">$worker_base</span>
</code></pre></div>
<h3 id="debug-mode">Debug Mode<a class="headerlink" href="#debug-mode" title="Permanent link">&para;</a></h3>
<p>Enable verbose logging:
<div class="highlight"><pre><span></span><code><span class="c1"># Environment variable</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">ML_DEBUG</span><span class="o">=</span><span class="m">1</span>
ml<span class="w"> </span>sync<span class="w"> </span>./project
<span class="c1"># Or use debug build</span>
<span class="nb">cd</span><span class="w"> </span>cli<span class="w"> </span><span class="o">&amp;&amp;</span><span class="w"> </span>make<span class="w"> </span>debug
</code></pre></div></p>
<h2 id="performance-benchmarks">Performance Benchmarks<a class="headerlink" href="#performance-benchmarks" title="Permanent link">&para;</a></h2>
<h3 id="file-operations">File Operations<a class="headerlink" href="#file-operations" title="Permanent link">&para;</a></h3>
<ul>
<li><strong>Sync Speed</strong>: 100MB/s+ (network limited)</li>
<li><strong>Hash Computation</strong>: 500MB/s+ (CPU limited)</li>
<li><strong>Deduplication</strong>: 90%+ space savings</li>
</ul>
<h3 id="memory-usage">Memory Usage<a class="headerlink" href="#memory-usage" title="Permanent link">&para;</a></h3>
<ul>
<li><strong>Base Memory</strong>: ~10MB</li>
<li><strong>Large Projects</strong>: ~50MB (1GB+ projects)</li>
<li><strong>Memory Efficiency</strong>: Constant per-file overhead</li>
</ul>
<h3 id="network-performance">Network Performance<a class="headerlink" href="#network-performance" title="Permanent link">&para;</a></h3>
<ul>
<li><strong>WebSocket Latency</strong>: &lt;10ms (local network)</li>
<li><strong>Connection Setup</strong>: &lt;100ms</li>
<li><strong>Throughput</strong>: Network limited</li>
</ul>
<h2 id="contributing">Contributing<a class="headerlink" href="#contributing" title="Permanent link">&para;</a></h2>
<h3 id="development-setup">Development Setup<a class="headerlink" href="#development-setup" title="Permanent link">&para;</a></h3>
<div class="highlight"><pre><span></span><code><span class="nb">cd</span><span class="w"> </span>cli
zig<span class="w"> </span>build-exe<span class="w"> </span>src/main.zig
</code></pre></div>
<h3 id="testing">Testing<a class="headerlink" href="#testing" title="Permanent link">&para;</a></h3>
<div class="highlight"><pre><span></span><code><span class="c1"># Run tests</span>
<span class="nb">cd</span><span class="w"> </span>cli<span class="w"> </span><span class="o">&amp;&amp;</span><span class="w"> </span>zig<span class="w"> </span><span class="nb">test</span><span class="w"> </span>src/
<span class="c1"># Integration tests</span>
zig<span class="w"> </span><span class="nb">test</span><span class="w"> </span>tests/
</code></pre></div>
<h3 id="code-style">Code Style<a class="headerlink" href="#code-style" title="Permanent link">&para;</a></h3>
<ul>
<li>Follow Zig style guidelines</li>
<li>Use explicit error handling</li>
<li>Document public APIs</li>
<li>Add comprehensive tests</li>
</ul>
<hr />
<p><strong>For more information, see the <a href="../cli-reference/">CLI Reference</a> and <a href="../architecture/">Architecture</a> pages.</strong></p>
</article>
</div>
<script>var target=document.getElementById(location.hash.slice(1));target&&target.name&&(target.checked=target.name.startsWith("__tabbed_"))</script>
</div>
</main>
<footer class="md-footer">
<div class="md-footer-meta md-typeset">
<div class="md-footer-meta__inner md-grid">
<div class="md-copyright">
Made with
<a href="https://squidfunk.github.io/mkdocs-material/" target="_blank" rel="noopener">
Material for MkDocs
</a>
</div>
</div>
</div>
</footer>
</div>
<div class="md-dialog" data-md-component="dialog">
<div class="md-dialog__inner md-typeset"></div>
</div>
<script id="__config" type="application/json">{"annotate": null, "base": "..", "features": ["navigation.instant", "navigation.tracking", "navigation.tabs", "navigation.sections", "navigation.expand", "navigation.indexes", "toc.integrate", "search.highlight", "search.share"], "search": "../assets/javascripts/workers/search.7a47a382.min.js", "tags": null, "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}, "version": null}</script>
<script src="../assets/javascripts/bundle.e71a0d61.min.js"></script>
</body>
</html>
Reference in a new issue View git blame Copy permalink