59 commits

Author	SHA1	Message	Date
Jeremie Fraeys	2d66a85abc	ci: add SHA256 verification for SQLite and fix CLI build ... - Add SHA256 verification to build_sqlite.sh (SQLite 3.48.0) - Add build-sqlite and CLI build steps to ci.yml build job - Remove cache from benchmark-metrics.yml	2026-02-21 20:58:08 -05:00
Jeremie Fraeys	f6c3e650b5	fix(build): disable openssl in rsync build to fix missing headers ... rsync configure requires --disable-openssl when OpenSSL dev headers unavailable Also removes dependency on openssl/md4.h and openssl/md5.h	2026-02-21 20:48:13 -05:00
Jeremie Fraeys	2e2bba47a2	fix(build): disable pipefail around tar head pipe to prevent SIGPIPE ... The tar | head -n 1 | cut pipeline causes SIGPIPE (error 141) when head closes the pipe before tar finishes writing	2026-02-21 20:46:15 -05:00
Jeremie Fraeys	7bc1c8a982	fix(build): avoid SIGPIPE in rsync SHA256 verification ... Replace echo | sha256sum -c - pipeline with direct comparison Fixes error 141 (SIGPIPE) caused by pipefail with early pipe close	2026-02-21 20:44:27 -05:00
Jeremie Fraeys	97c066af4f	fix(build): add rsync SHA256 hash to skip GPG verification ... Add SHA256 for rsync 3.3.0: 7399e9a6708c32d678a72a63219e96f23be0be2336e50fd1348498d07041df90 This allows the build to proceed without requiring GPG keyring setup in CI	2026-02-21 20:42:35 -05:00
Jeremie Fraeys	be39b37aec	feat: native GPU detection and NVML bridge for macOS and Linux ... - Add dynamic NVML loading for Linux GPU detection - Add macOS GPU detection via IOKit framework - Add Zig NVML wrapper for cross-platform GPU queries - Update native bridge to support platform-specific GPU libs - Add CMake support for NVML dynamic library	2026-02-21 17:59:59 -05:00
Jeremie Fraeys	1a1844e9e9	fix(cli): remaining ArrayList API fixes in dataset and queue commands	2026-02-21 17:59:51 -05:00
Jeremie Fraeys	b1c9bc97fc	fix(cli): CLI structure, manifest, and asset fixes ... - Fix commands.zig imports (logs.zig → log.zig, remove missing modules) - Fix manifest.writeManifest to accept allocator param - Add db.Stmt type alias for sqlite3_stmt - Fix rsync placeholder to be valid shell script (#!/bin/sh)	2026-02-21 17:59:20 -05:00
Jeremie Fraeys	382c67edfc	fix(cli): WebSocket protocol and sync command fixes ... - Add sendSyncRun method for run synchronization - Add sendRerunRequest method for queue rerun - Add sync_run (0x26) and rerun_request (0x27) opcodes - Fix protocol import path to relative path - Fix db.Stmt type alias usage in sync.zig	2026-02-21 17:59:14 -05:00
Jeremie Fraeys	ccd1dd7a4d	fix(cli): Zig 0.15 core API changes ... - ArrayList: .init(allocator) → .empty, add allocator param to append/deinit/toOwnedSlice - Atomic: std.atomic.Atomic → std.atomic.Value, lowercase order names (.seq_cst) - Process: execvp instead of execvpe, inline wait status macros for macOS - Time: std.time.sleep → std.Thread.sleep - Error handling: fix isProcessRunning error union comparison	2026-02-21 17:59:05 -05:00
Jeremie Fraeys	d1ac558107	perf: implement context reuse ... Go Worker (internal/worker/native_bridge_libs.go): - Add global hashCtx with sync.Once for lazy initialization - Eliminates 5-20ms fh_init/fh_cleanup per hash operation - Uses runtime.NumCPU() for optimal thread count - Log initialization time for observability Zig CLI (cli/src/native/hash.zig): - Add global_ctx with atomic flag and mutex - Thread-safe initialization with double-check pattern - Idempotent init() callable from multiple threads - Log init time for debugging	2026-02-21 14:19:14 -05:00
Jeremie Fraeys	25ae791b5c	refactor: make dataset hash automatic in verify command ... - Remove separate 'hash' subcommand - Integrate native SHA256 hash into 'dataset verify' - Hash is now computed automatically when verifying datasets - Shows hash in output (JSON, CSV, and text formats) - Help text updated to indicate auto-hashing	2026-02-21 14:09:44 -05:00
Jeremie Fraeys	1a35c54300	feat: integrate native library into Zig CLI ... - Add cli/src/native/hash.zig - C ABI wrapper for dataset_hash - Update cli/src/commands/dataset.zig - Add 'hash' subcommand - Update cli/build.zig - Link against libdataset_hash.so - Fix pre-existing CLI errors in experiment.zig (errorMsg signatures, columnInt64) Usage: ml dataset hash <path> Note: Additional pre-existing CLI errors remain in sync.zig	2026-02-21 14:08:07 -05:00
Jeremie Fraeys	7efe8bbfbf	native: security hardening, research trustworthiness, and CVE mitigations ... Security Fixes: - CVE-2024-45339: Add O_EXCL flag to temp file creation in storage_write_entries() Prevents symlink attacks on predictable .tmp file paths - CVE-2025-47290: Use openat_nofollow() in storage_open() Closes TOCTOU race condition via path_sanitizer infrastructure - CVE-2025-0838: Add MAX_BATCH_SIZE=10000 to add_tasks() Prevents integer overflow in batch operations Research Trustworthiness (dataset_hash): - Deterministic file ordering: std::sort after collect_files() - Recursive directory traversal: depth-limited with cycle detection - Documented exclusions: hidden files and special files noted in API Bug Fixes: - R1: storage_init path validation for non-existent directories - R2: safe_strncpy return value check before strcat - R3: parallel_hash 256-file cap replaced with std::vector - R4: wire qi_compact_index/qi_rebuild_index stubs - R5: CompletionLatch race condition fix (hold mutex during decrement) - R6: ARMv8 SHA256 transform fix (save abcd_pre before vsha256hq_u32) - R7: fuzz_index_storage header format fix - R8: enforce null termination in add_tasks/update_tasks - R9: use 64 bytes (not 65) in combined hash to exclude null terminator - R10: status field persistence in save() New Tests: - test_recursive_dataset.cpp: Verify deterministic recursive hashing - test_storage_symlink_resistance.cpp: Verify CVE-2024-45339 fix - test_queue_index_batch_limit.cpp: Verify CVE-2025-0838 fix - test_sha256_arm_kat.cpp: ARMv8 known-answer tests - test_storage_init_new_dir.cpp: F1 verification - test_parallel_hash_large_dir.cpp: F3 verification - test_queue_index_compact.cpp: F4 verification All 8 native tests passing. Library ready for research lab deployment.	2026-02-21 13:33:45 -05:00
Jeremie Fraeys	201cb66f56	fix(cli): Standardize WebSocket client imports ... - Change from deps.zig indirect imports to direct @import() calls - Improves build compatibility and clarity - Aligns with Zig idiomatic import style	2026-02-20 21:41:51 -05:00
Jeremie Fraeys	a3b957dcc0	refactor(cli): Update build system and core infrastructure ... - Makefile: Update build targets for native library integration - build.zig: Add SQLite linking and native hash library support - scripts/build_rsync.sh: Update rsync embedded binary build process - scripts/build_sqlite.sh: Add SQLite constants generation script - src/assets/README.md: Document embedded asset structure - src/utils/rsync_embedded_binary.zig: Update for new build layout	2026-02-20 21:39:51 -05:00
Jeremie Fraeys	04ac745b01	refactor(cli): Rename note to annotate and re-add experiment command ... - Renamed note.zig to annotate.zig (preserves user's preferred naming) - Updated all references from 'ml note' to 'ml annotate' - Re-added experiment.zig with create/list/show subcommands - Updated main.zig dispatch: 'a' for annotate, 'e' for experiment - Updated printUsage and test block to reflect changes	2026-02-20 21:32:01 -05:00
Jeremie Fraeys	adf4c2a834	refactor(cli): Update main.zig and remove deprecated commands ... - main.zig: Update command dispatch and usage text - Wire up new commands: note, logs, sync, cancel, watch - Remove deprecated command references - Updated usage reflects unified command structure - Delete deprecated command files: - annotate.zig (replaced by note.zig) - experiment.zig (functionality in run/note/logs) - logs.zig (old version, replaced) - monitor.zig (unused) - narrative.zig (replaced by note --hypothesis/context) - outcome.zig (replaced by note --outcome) - privacy.zig (replaced by note --privacy) - requeue.zig (functionality merged into queue --rerun)	2026-02-20 21:28:42 -05:00
Jeremie Fraeys	d3461cd07f	feat(cli): Update server integration commands ... - queue.zig: Add --rerun <run_id> flag to re-queue completed local runs - Requires server connection, rejects in offline mode with clear error - HandleRerun function sends rerun request via WebSocket - sync.zig: Rewrite for WebSocket experiment sync protocol - Queries unsynced runs from SQLite ml_runs table - Builds sync JSON with metrics and params - Sends sync_run message, waits for sync_ack response - MarkRunSynced updates synced flag in database - watch.zig: Add --sync flag for continuous experiment sync - Auto-sync runs to server every 30 seconds when online - Mode detection with offline error handling	2026-02-20 21:28:34 -05:00
Jeremie Fraeys	f5b68cca49	feat(cli): Add metadata commands and update cancel ... - note.zig: New unified metadata annotation command - Supports --text, --hypothesis, --outcome, --confidence, --privacy, --author - Stores metadata as tags in SQLite ml_tags table - log.zig: Simplified to unified logs command (fetch/stream only) - Removed metric/param/tag subcommands (now in run wrapper) - Supports --follow for live log streaming from server - cancel.zig: Add local process termination support - Sends SIGTERM first, waits 5s, then SIGKILL if needed - Updates run status to CANCELLED in SQLite - Also supports server job cancellation via WebSocket	2026-02-20 21:28:23 -05:00
Jeremie Fraeys	d0c68772ea	feat(cli): Implement unified run wrapper command ... - Fork child process and capture stdout/stderr via pipe - Parse FETCHML_METRIC key=value [step=N] lines from output - Write run_manifest.json with run metadata - Insert/update ml_runs table in SQLite with PID tracking - Stream output to output.log file - Support entrypoint from config or explicit command after --	2026-02-20 21:28:16 -05:00
Jeremie Fraeys	551597b5df	feat(cli): Add core infrastructure for local mode support ... - mode.zig: Automatic online/offline mode detection with API ping - manifest.zig: Run manifest read/write/update operations - core/: Common flags, output formatting, and context management - local.zig + local/: Local mode experiment operations - server.zig + server/: Server mode API client - db.zig: Add pid column to ml_runs table for process tracking - config.zig: Add force_local, [experiment] section with name/entrypoint - utils/native_bridge.zig: Native library integration	2026-02-20 21:28:06 -05:00
Jeremie Fraeys	7583932897	feat(cli): add progress UI and rsync assets ... - Add progress.zig for sync progress display - Add rsync placeholder and release binaries to assets/rsync/	2026-02-20 15:51:17 -05:00
Jeremie Fraeys	2258f60ade	feat(cli): add utility modules for local mode ... - Add hash_cache.zig for efficient file hash caching - Add ignore.zig for .gitignore-style pattern matching - Add native_hash.zig for C dataset_hash library integration	2026-02-20 15:51:10 -05:00
Jeremie Fraeys	7ce0fd251e	feat(cli): unified commands and local mode support ... - Update experiment.zig with unified commands (local + server modes) - Add init.zig for local project initialization - Update sync.zig for project synchronization - Update main.zig to route new local mode commands (experiment, run, log) - Support automatic mode detection from config (sqlite:// vs wss://)	2026-02-20 15:51:04 -05:00
Jeremie Fraeys	2c596038b5	refactor(cli): update build system and config for local mode ... - Update Makefile with build-sqlite target matching rsync pattern - Fix build.zig to handle SQLite assets and dataset_hash linking - Add SQLite asset detection mirroring rsync binary detection - Update CLI README with local mode documentation - Restructure rsync assets into rsync/ subdirectory - Remove obsolete files (fix_arraylist.sh, old rsync_placeholder.bin) - Add build_rsync.sh script to fetch/build rsync from source	2026-02-20 15:50:52 -05:00
Jeremie Fraeys	ff542b533f	feat(cli): embed SQLite and unify commands for local mode ... - Add SQLite amalgamation fetch script (make build-sqlite) - Embed SQLite in release builds, link system lib in dev - Create sqlite_embedded.zig utility module - Unify experiment/run/log commands with auto mode detection - Add Forgejo CI workflow for building with embedded SQLite - Update READMEs for local mode and build instructions SQLite follows rsync embedding pattern: assets/sqlite_release_<os>_<arch>/ Zero external dependencies for release builds.	2026-02-20 15:50:04 -05:00
Jeremie Fraeys	6028779239	feat: update CLI, TUI, and security documentation ... - Add safety checks to Zig build - Add TUI with job management and narrative views - Add WebSocket support and export services - Add smart configuration defaults - Update API routes with security headers - Update SECURITY.md with comprehensive policy - Add Makefile security scanning targets	2026-02-19 15:35:05 -05:00
Jeremie Fraeys	27c8b08a16	test: Reorganize and add unit tests ... Reorganize tests for better structure and coverage: - Move container/security_test.go from internal/ to tests/unit/container/ - Move related tests to proper unit test locations - Delete orphaned test files (startup_blacklist_test.go) - Add privacy middleware unit tests - Add worker config unit tests - Update E2E tests for homelab and websocket scenarios - Update test fixtures with utility functions - Add CLI helper script for arraylist fixes	2026-02-18 21:28:13 -05:00
Jeremie Fraeys	cb826b74a3	feat: WebSocket API infrastructure improvements ... Enhance WebSocket client and server components: - Add new WebSocket opcodes (CompareRuns, FindRuns, ExportRun, SetRunOutcome) - Improve WebSocket client with additional response handlers - Add crypto utilities for secure WebSocket communications - Add I/O utilities for WebSocket payload handling - Enhance validation for WebSocket message payloads - Update routes for new WebSocket endpoints - Improve monitor and validate command WebSocket integrations	2026-02-18 21:27:48 -05:00
Jeremie Fraeys	b2eba75f09	feat: CLI shell completion for new commands ... Update bash and zsh completion scripts to include: - compare, find, export, outcome commands - privacy command and subcommands - All new narrative field flags (--hypothesis, --context, etc.) - Sandboxing options (--network, --read-only, --secret)	2026-02-18 21:27:38 -05:00
Jeremie Fraeys	aaeef69bab	feat: Privacy and PII detection ... Add privacy protection features to prevent accidental PII leakage: - PII detection engine supporting emails, phone numbers, SSNs, credit cards - CLI privacy command for scanning files and text - Privacy middleware for API request/response filtering - Suggestion utility for privacy-preserving alternatives Integrates PII scanning into manifest validation for narrative fields.	2026-02-18 21:27:23 -05:00
Jeremie Fraeys	260e18499e	feat: Research features - narrative fields and outcome tracking ... Add comprehensive research context tracking to jobs: - Narrative fields: hypothesis, context, intent, expected_outcome - Experiment groups and tags for organization - Run comparison (compare command) for diff analysis - Run search (find command) with criteria filtering - Run export (export command) for data portability - Outcome setting (outcome command) for experiment validation Update queue and requeue commands to support narrative fields. Add narrative validation to manifest validator. Add WebSocket handlers for compare, find, export, and outcome operations. Includes E2E tests for phase 2 features.	2026-02-18 21:27:05 -05:00
Jeremie Fraeys	a64233d4f6	fix: CLI now properly rejects unknown commands ... - Add handleUnknownCommand() helper function - Add else clauses to 'i' and 's' switch cases - Commands like 'invalid_command', 'infoooo', 'syncccc' now properly rejected - Prevents silent acceptance of invalid commands - Test TestCLICommandsE2E/CLIErrorHandling now passes	2026-02-18 16:04:37 -05:00
Jeremie Fraeys	cd2908181c	refactor(cli): reduce code duplication in WebSocket client ... Add MessageBuilder struct and validation helpers to reduce duplication: - MessageBuilder: Helper for constructing binary WebSocket messages - writeOpcode, writeBytes, writeU8/16/32/64 - writeStringU8, writeStringU16 for length-prefixed strings - Validation helpers: validateApiKeyHash, validateCommitId, validateJobName - getStream(): Extracted common stream check pattern Refactored 4 representative send methods to use new helpers: - sendValidateRequestCommit, sendListJupyterPackages - sendCancelJob, sendStatusRequest Consolidated disconnect/close: close() now calls disconnect() Updated response handlers to use packet.deinit(): - receiveAndHandleResponse - receiveAndHandleDatasetResponse Reduces ~100 lines of boilerplate duplication. Build verified: zig build --release=fast	2026-02-18 13:56:30 -05:00
Jeremie Fraeys	14eba436bf	feat(cli): include command name in error messages and crash reports ... Implement TODO in handleCommandError: - Log command name with error for crash report context - Display 'Error in \'{s}': {s}' instead of generic 'Error: {s}' - Helps users identify which command failed Build verified: zig build --release=fast	2026-02-18 13:51:13 -05:00
Jeremie Fraeys	32e65545ae	feat(cli): implement proper JSON parsing for sync progress ... Use utils/json.zig helpers to parse sync progress messages: - Add json module import - Rename json variable to json_mode to avoid shadowing - Parse status, progress, total fields from JSON response - Show percentage completion for in-progress syncs - Handle 'complete' and 'error' status codes Build verified: zig build --release=fast	2026-02-18 13:49:01 -05:00
Jeremie Fraeys	55b4b23645	feat(cli): integrate embedded rsync for remote sync ... Replace local file copy with embedded rsync binary for actual remote synchronization to the server. The embedded rsync is extracted from assets/ at runtime - no external dependencies required. Changes: - Re-add rsync_embedded.zig import - Replace manual file copying with rsync.sync() call - Construct remote path as api_key@host:worker_base/commit_id/files/ - Update JSON output to include commit_id Build verified: zig build --release=fast	2026-02-18 13:45:50 -05:00
Jeremie Fraeys	c31055be30	refactor(cli): remove unused imports from sync.zig ... Remove unused imports: - rsync (commented as 'Use embedded rsync' but never used) - storage (never referenced) Build verified: zig build --release=fast	2026-02-18 13:39:16 -05:00
Jeremie Fraeys	999af75f39	refactor(cli): use config.getWebSocketUrl() in jupyter.zig ... Replace 6 instances of inline WebSocket URL construction with the config.getWebSocketUrl() helper method. This ensures consistent URL formatting and reduces code duplication. Functions updated: - startJupyter - stopJupyter - removeJupyter - listServices - packageCommands Build verified: zig build --release=fast	2026-02-18 13:36:03 -05:00
Jeremie Fraeys	f2f645fe21	refactor(cli): use packet.deinit() in remaining command files ... Replace manual defer blocks with packet.deinit() for consistent cleanup: - jupyter.zig: 5 locations updated (restoreJupyter, startJupyter, stopJupyter, removeJupyter, listServices, packageCommands) - experiment.zig: 3 locations updated (executeLog, executeShow, executeDelete) - queue.zig: 1 location updated (queueSingleJob) - logs.zig: 1 location updated (run) Eliminates 44 manual cleanup blocks across 4 files. Build verified: zig build --release=fast	2026-02-18 13:33:10 -05:00
Jeremie Fraeys	29d130dcdb	refactor(cli): standardize error handling and use packet.deinit() ... - Replace std.process.exit(1) with error.InvalidArgs in sync command - Replace std.process.exit(1) with error.ValidationFailed in validate command - Update validate.zig to use protocol.ResponsePacket.deinit() for cleanup - Build verified: zig build --release=fast	2026-02-18 13:29:53 -05:00
Jeremie Fraeys	9517271bbb	fix(cli): remove unreachable code and unused Command enum ... - Fix duplicate 'i' case in Command.fromString that was unreachable - Remove unused Command enum entirely (dispatch uses inline comparisons) - Build verified: zig build --release=fast	2026-02-18 13:26:18 -05:00
Jeremie Fraeys	1597c20b73	refactor(cli): consolidate shared utilities and remove code duplication ... Extract common helper functions from multiple command files into shared utility modules: - Create cli/src/utils/json.zig with json.getString(), getInt(), getFloat(), getBool() - Create cli/src/utils/manifest.zig with readFileAlloc(), resolvePathWithBase(), resolvePathById(), readJobNameFromManifest() - Add ResponsePacket.deinit() method to net/protocol.zig for consistent cleanup - Update info.zig, annotate.zig, narrative.zig, requeue.zig to use shared utilities - Update utils.zig exports for new modules Eliminates duplicate implementations of: - jsonGetString() and jsonGetInt() in 4 files - readFileAlloc() in 4 files - resolveManifestPath*() functions in 4 files - ResponsePacket cleanup defer blocks (replaced with .deinit()) Builds cleanly with zig build --release=fast	2026-02-18 13:19:40 -05:00
Jeremie Fraeys	34186675dc	refactor(cli): use config.getWebSocketUrl() helper across all commands ... Replace inline WebSocket URL construction with Config.getWebSocketUrl() helper method in all command files. This eliminates code duplication and ensures consistent URL formatting across the CLI. Files updated: - annotate.zig, dataset.zig, experiment.zig, logs.zig - narrative.zig, prune.zig, queue.zig, requeue.zig - sync.zig, validate.zig, watch.zig The helper properly handles ws:// vs wss:// based on port (443).	2026-02-18 13:05:43 -05:00
Jeremie Fraeys	c85575048f	refactor(cli): consolidate shared types and reduce code duplication ... Extract common UserContext and authentication logic from cancel.zig and status.zig into new utils/auth.zig module. Add CommonFlags struct to utils/flags.zig for shared CLI flags. Add getWebSocketUrl() helper to Config to eliminate URL construction duplication. Changes: - Create cli/src/utils/auth.zig with UserContext and authenticateUser() - Create cli/src/utils/flags.zig with CommonFlags struct - Update cancel.zig and status.zig to use shared modules - Add getWebSocketUrl() helper to config.zig - Export new modules from utils.zig Reduces code duplication and improves separation of concerns in the Zig CLI codebase.	2026-02-18 13:00:48 -05:00
Jeremie Fraeys	f9bba8a0fc	refactor(cli): reorganize queue commands and add logs test ... - Reorganize queue command handlers - Remove debug logs test file - Add proper logs command test	2026-02-18 12:45:54 -05:00
Jeremie Fraeys	fb4c91f4c5	chore: add review workflow and test updates ... - Add Windsurf review workflow configuration - Add logs debug test for CLI - Update main test file	2026-02-16 20:39:09 -05:00
Jeremie Fraeys	1147958e15	feat: enhance CLI with improved commands and WebSocket handling ... - Refactor command structure for better organization - Improve WebSocket client frame handling - Add response handler improvements - Update queue, requeue, and status commands - Add security module for CLI authentication	2026-02-16 20:38:08 -05:00
Jeremie Fraeys	8e3fa94322	feat(cli): enhance Zig CLI with new commands and improved networking ... - Add new commands: annotate, narrative, requeue - Refactor WebSocket client into modular components (net/ws/) - Add rsync embedded binary support - Improve error handling and response packet processing - Update build.zig and completions	2026-02-12 12:05:10 -05:00