Remove obsolete native hash implementation files:
- Delete native/hash.zig (superseded by utils/hash.zig)
- Delete utils/native_bridge.zig (replaced by direct TLS)
- Delete utils/native_hash.zig (consolidated into utils/hash.zig)
Cleanup as part of CLI hardening.
Update command structure with improved implementations:
- exec.zig: consolidated command execution
- queue.zig: improved job queuing with narrative support
- run.zig: enhanced local run execution
- dataset.zig, dataset_hash.zig: improved dataset management
Part of CLI hardening for better UX and reliability.
Add progress reporting and offline sync infrastructure:
- progress.zig: progress bars and status reporting
- sync_manager.zig: offline run synchronization manager
Supports resilient operation with server connectivity issues.
Add signal handling, environment detection, and secrets management:
- signals.zig: graceful Ctrl+C handling and signal management
- environment.zig: user environment detection for telemetry
- secrets.zig: secrets redaction for secure logging
Improves CLI reliability and security posture.
Update experiment creation to track sync status:
- Insert experiments with synced=0 (not synced to server)
- Align with schema update in db.zig for offline tracking
Part of offline run synchronization feature.
Add hardware-accelerated hash detection:
- Implement hasShaNi() using CPUID inline assembly for x86_64
- Detect SHA-NI support (bit 29 of EBX in leaf 7, subleaf 0)
- Cross-platform fallback for non-x86_64 architectures
- Enables hardware-accelerated SHA-256 when available
Improves hashing performance on modern Intel/AMD CPUs.
Add TLS transport abstraction for secure WebSocket connections:
- Create tls.zig module with TlsStream struct for TLS-encrypted sockets
- Implement Transport union in client.zig supporting both TCP and TLS
- Update frame.zig and handshake.zig to use Transport abstraction
- Add TLS handshake, read, write, flush, and close operations
- Support TLS 1.2/1.3 protocol versions with error handling
- Zig 0.15 compatible ArrayList API usage
Enables wss:// protocol support for encrypted server communication.
Remove colors dependency from output.zig
Add terminal.zig for TTY detection and terminal width
Update flags.zig with color flag support
Simplify colors.zig to basic ANSI codes
Update main.zig and utils.zig exports
Replace space-padding with consistent tab (\t) alignment in all printUsage() functions.
Add ligature-friendly ASCII symbols:
- => for results/outcomes (renders as ⇒ with ligatures)
- ~> for modifications/changes (renders as ~> with ligatures)
- -> for state transitions (renders as → with ligatures)
- [OK] / [FAIL] for status indicators
All symbols use ASCII 32-126 for xargs-safe, copy-pasteable output.
- Surface GPUDetectionInfo from parseGPUCountFromConfig for detection metadata
- Document FETCH_ML_TOTAL_CPU and FETCH_ML_GPU_SLOTS_PER_GPU env vars
- Add debug logging for all env var overrides to stderr
- Track config-layer auto-detection in GPUDetectionInfo.ConfigLayerAutoDetected
- Add --include-all flag to artifact scanner (includeAll parameter)
- Add AMD production mode enforcement (error in non-local mode)
- Add GPU detector unit tests for env overrides and AMD aliasing
- Add sendSyncRun method for run synchronization
- Add sendRerunRequest method for queue rerun
- Add sync_run (0x26) and rerun_request (0x27) opcodes
- Fix protocol import path to relative path
- Fix db.Stmt type alias usage in sync.zig
Go Worker (internal/worker/native_bridge_libs.go):
- Add global hashCtx with sync.Once for lazy initialization
- Eliminates 5-20ms fh_init/fh_cleanup per hash operation
- Uses runtime.NumCPU() for optimal thread count
- Log initialization time for observability
Zig CLI (cli/src/native/hash.zig):
- Add global_ctx with atomic flag and mutex
- Thread-safe initialization with double-check pattern
- Idempotent init() callable from multiple threads
- Log init time for debugging
- Remove separate 'hash' subcommand
- Integrate native SHA256 hash into 'dataset verify'
- Hash is now computed automatically when verifying datasets
- Shows hash in output (JSON, CSV, and text formats)
- Help text updated to indicate auto-hashing
- Renamed note.zig to annotate.zig (preserves user's preferred naming)
- Updated all references from 'ml note' to 'ml annotate'
- Re-added experiment.zig with create/list/show subcommands
- Updated main.zig dispatch: 'a' for annotate, 'e' for experiment
- Updated printUsage and test block to reflect changes
- queue.zig: Add --rerun <run_id> flag to re-queue completed local runs
- Requires server connection, rejects in offline mode with clear error
- HandleRerun function sends rerun request via WebSocket
- sync.zig: Rewrite for WebSocket experiment sync protocol
- Queries unsynced runs from SQLite ml_runs table
- Builds sync JSON with metrics and params
- Sends sync_run message, waits for sync_ack response
- MarkRunSynced updates synced flag in database
- watch.zig: Add --sync flag for continuous experiment sync
- Auto-sync runs to server every 30 seconds when online
- Mode detection with offline error handling
- note.zig: New unified metadata annotation command
- Supports --text, --hypothesis, --outcome, --confidence, --privacy, --author
- Stores metadata as tags in SQLite ml_tags table
- log.zig: Simplified to unified logs command (fetch/stream only)
- Removed metric/param/tag subcommands (now in run wrapper)
- Supports --follow for live log streaming from server
- cancel.zig: Add local process termination support
- Sends SIGTERM first, waits 5s, then SIGKILL if needed
- Updates run status to CANCELLED in SQLite
- Also supports server job cancellation via WebSocket
- Fork child process and capture stdout/stderr via pipe
- Parse FETCHML_METRIC key=value [step=N] lines from output
- Write run_manifest.json with run metadata
- Insert/update ml_runs table in SQLite with PID tracking
- Stream output to output.log file
- Support entrypoint from config or explicit command after --
- Update experiment.zig with unified commands (local + server modes)
- Add init.zig for local project initialization
- Update sync.zig for project synchronization
- Update main.zig to route new local mode commands (experiment, run, log)
- Support automatic mode detection from config (sqlite:// vs wss://)
- Add SQLite amalgamation fetch script (make build-sqlite)
- Embed SQLite in release builds, link system lib in dev
- Create sqlite_embedded.zig utility module
- Unify experiment/run/log commands with auto mode detection
- Add Forgejo CI workflow for building with embedded SQLite
- Update READMEs for local mode and build instructions
SQLite follows rsync embedding pattern: assets/sqlite_release_<os>_<arch>/
Zero external dependencies for release builds.
Add comprehensive research context tracking to jobs:
- Narrative fields: hypothesis, context, intent, expected_outcome
- Experiment groups and tags for organization
- Run comparison (compare command) for diff analysis
- Run search (find command) with criteria filtering
- Run export (export command) for data portability
- Outcome setting (outcome command) for experiment validation
Update queue and requeue commands to support narrative fields.
Add narrative validation to manifest validator.
Add WebSocket handlers for compare, find, export, and outcome operations.
Includes E2E tests for phase 2 features.
- Add handleUnknownCommand() helper function
- Add else clauses to 'i' and 's' switch cases
- Commands like 'invalid_command', 'infoooo', 'syncccc' now properly rejected
- Prevents silent acceptance of invalid commands
- Test TestCLICommandsE2E/CLIErrorHandling now passes
Implement TODO in handleCommandError:
- Log command name with error for crash report context
- Display 'Error in \'{s}': {s}' instead of generic 'Error: {s}'
- Helps users identify which command failed
Build verified: zig build --release=fast
Use utils/json.zig helpers to parse sync progress messages:
- Add json module import
- Rename json variable to json_mode to avoid shadowing
- Parse status, progress, total fields from JSON response
- Show percentage completion for in-progress syncs
- Handle 'complete' and 'error' status codes
Build verified: zig build --release=fast
Replace local file copy with embedded rsync binary for actual remote
synchronization to the server. The embedded rsync is extracted from
assets/ at runtime - no external dependencies required.
Changes:
- Re-add rsync_embedded.zig import
- Replace manual file copying with rsync.sync() call
- Construct remote path as api_key@host:worker_base/commit_id/files/
- Update JSON output to include commit_id
Build verified: zig build --release=fast
- Replace std.process.exit(1) with error.InvalidArgs in sync command
- Replace std.process.exit(1) with error.ValidationFailed in validate command
- Update validate.zig to use protocol.ResponsePacket.deinit() for cleanup
- Build verified: zig build --release=fast
