Jeremie Fraeys
54ddab887e
build: update Makefile and Zig build for new targets
...
**Makefile:**
- Add native build targets and test infrastructure
- Update benchmark and CI test commands
**cli/build.zig:**
- Build configuration updates for CLI compilation
2026-02-23 18:03:47 -05:00
Jeremie Fraeys
7d1ba75092
chore: Update security scan workflow and SQLite build script
2026-02-23 14:24:00 -05:00
Jeremie Fraeys
6faa13aabf
refactor(cli): Remove progress UI and update native/server code
...
Delete cli/src/ui/progress.zig (removing progress bars/spinners)
Update native GPU detection modules
Update server experiment API
2026-02-23 14:12:48 -05:00
Jeremie Fraeys
fd317c9791
refactor(cli): Update network handlers and info command
...
Replace Unicode symbols with ASCII in handshake.zig
Add [OK]/[FAIL] status indicators in response_handlers.zig
Simplify info.zig output formatting
2026-02-23 14:12:22 -05:00
Jeremie Fraeys
2b7319dc2e
refactor(cli): Simplify output system and add terminal utilities
...
Remove colors dependency from output.zig
Add terminal.zig for TTY detection and terminal width
Update flags.zig with color flag support
Simplify colors.zig to basic ANSI codes
Update main.zig and utils.zig exports
2026-02-23 14:11:59 -05:00
Jeremie Fraeys
a1988de8b1
style(cli): Standardize printUsage() formatting with tabs and ASCII symbols
...
Replace space-padding with consistent tab (\t) alignment in all printUsage() functions.
Add ligature-friendly ASCII symbols:
- => for results/outcomes (renders as ⇒ with ligatures)
- ~> for modifications/changes (renders as ~> with ligatures)
- -> for state transitions (renders as → with ligatures)
- [OK] / [FAIL] for status indicators
All symbols use ASCII 32-126 for xargs-safe, copy-pasteable output.
2026-02-23 14:09:49 -05:00
Jeremie Fraeys
3b194ff2e8
feat: GPU detection transparency and artifact scanner improvements
...
Build CLI with Embedded SQLite / build (arm64, aarch64-linux) (push) Waiting to run
Build CLI with Embedded SQLite / build (x86_64, x86_64-linux) (push) Waiting to run
Build CLI with Embedded SQLite / build-macos (arm64) (push) Waiting to run
Build CLI with Embedded SQLite / build-macos (x86_64) (push) Waiting to run
Security Scan / Security Analysis (push) Waiting to run
Security Scan / Native Library Security (push) Waiting to run
Checkout test / test (push) Successful in 6s
CI/CD Pipeline / Test (push) Failing after 1s
CI/CD Pipeline / Dev Compose Smoke Test (push) Has been skipped
CI/CD Pipeline / Build (push) Has been skipped
CI/CD Pipeline / Test Scripts (push) Has been skipped
CI/CD Pipeline / Test Native Libraries (push) Has been skipped
CI/CD Pipeline / GPU Golden Test Matrix (push) Has been skipped
Documentation / build-and-publish (push) Failing after 39s
CI/CD Pipeline / Docker Build (push) Has been skipped
- Surface GPUDetectionInfo from parseGPUCountFromConfig for detection metadata
- Document FETCH_ML_TOTAL_CPU and FETCH_ML_GPU_SLOTS_PER_GPU env vars
- Add debug logging for all env var overrides to stderr
- Track config-layer auto-detection in GPUDetectionInfo.ConfigLayerAutoDetected
- Add --include-all flag to artifact scanner (includeAll parameter)
- Add AMD production mode enforcement (error in non-local mode)
- Add GPU detector unit tests for env overrides and AMD aliasing
2026-02-23 12:29:34 -05:00
Jeremie Fraeys
ec568b4c85
fix(build): link libc for Zig tests using C imports
Documentation / build-and-publish (push) Waiting to run
Security Scan / Security Analysis (push) Waiting to run
Security Scan / Native Library Security (push) Waiting to run
Checkout test / test (push) Successful in 5s
CI/CD Pipeline / Test (push) Has been cancelled
CI/CD Pipeline / Dev Compose Smoke Test (push) Has been cancelled
CI/CD Pipeline / Build (push) Has been cancelled
CI/CD Pipeline / Test Scripts (push) Has been cancelled
CI/CD Pipeline / Test Native Libraries (push) Has been cancelled
CI/CD Pipeline / Docker Build (push) Has been cancelled
Build CLI with Embedded SQLite / build (arm64, aarch64-linux) (push) Has been cancelled
Build CLI with Embedded SQLite / build (x86_64, x86_64-linux) (push) Has been cancelled
Build CLI with Embedded SQLite / build-macos (arm64) (push) Has been cancelled
Build CLI with Embedded SQLite / build-macos (x86_64) (push) Has been cancelled
2026-02-21 21:19:09 -05:00
Jeremie Fraeys
03ed1a05bf
fix(build): add iterate permission for test directory
...
Build CLI with Embedded SQLite / build (arm64, aarch64-linux) (push) Waiting to run
Build CLI with Embedded SQLite / build (x86_64, x86_64-linux) (push) Waiting to run
Build CLI with Embedded SQLite / build-macos (arm64) (push) Waiting to run
Build CLI with Embedded SQLite / build-macos (x86_64) (push) Waiting to run
Security Scan / Security Analysis (push) Waiting to run
Security Scan / Native Library Security (push) Waiting to run
Checkout test / test (push) Successful in 5s
CI/CD Pipeline / Test (push) Failing after 5m50s
CI/CD Pipeline / Dev Compose Smoke Test (push) Has been skipped
CI/CD Pipeline / Build (push) Has been skipped
CI/CD Pipeline / Test Scripts (push) Has been skipped
CI/CD Pipeline / Test Native Libraries (push) Has been skipped
Documentation / build-and-publish (push) Failing after 41s
CI/CD Pipeline / Docker Build (push) Has been skipped
Zig 0.15 requires .iterate = true flag when opening directories for iteration
Fixes panic: reached unreachable code in std.fs.Dir.iterate
2026-02-21 21:10:48 -05:00
Jeremie Fraeys
39bf466737
refactor(build): fetch SHA256 from official sources
...
Build CLI with Embedded SQLite / build (arm64, aarch64-linux) (push) Waiting to run
Build CLI with Embedded SQLite / build (x86_64, x86_64-linux) (push) Waiting to run
Build CLI with Embedded SQLite / build-macos (arm64) (push) Waiting to run
Build CLI with Embedded SQLite / build-macos (x86_64) (push) Waiting to run
Documentation / build-and-publish (push) Waiting to run
Security Scan / Security Analysis (push) Waiting to run
Security Scan / Native Library Security (push) Waiting to run
Checkout test / test (push) Successful in 5s
CI/CD Pipeline / Test (push) Has been cancelled
CI/CD Pipeline / Dev Compose Smoke Test (push) Has been cancelled
CI/CD Pipeline / Build (push) Has been cancelled
CI/CD Pipeline / Test Scripts (push) Has been cancelled
CI/CD Pipeline / Test Native Libraries (push) Has been cancelled
CI/CD Pipeline / Docker Build (push) Has been cancelled
- SQLite: fetch from sqlite.org/<YEAR>/<file>.zip.sha256 with embedded fallback
- Rsync: fetch from download.samba.org/.../<file>.tar.gz.sha256 with embedded fallback
- Remove hardcoded SHA256 requirement when official checksums available
2026-02-21 21:00:23 -05:00
Jeremie Fraeys
2d66a85abc
ci: add SHA256 verification for SQLite and fix CLI build
...
Build CLI with Embedded SQLite / build (arm64, aarch64-linux) (push) Waiting to run
Build CLI with Embedded SQLite / build (x86_64, x86_64-linux) (push) Waiting to run
Build CLI with Embedded SQLite / build-macos (arm64) (push) Waiting to run
Build CLI with Embedded SQLite / build-macos (x86_64) (push) Waiting to run
Documentation / build-and-publish (push) Waiting to run
Security Scan / Security Analysis (push) Waiting to run
Security Scan / Native Library Security (push) Waiting to run
Checkout test / test (push) Successful in 5s
CI/CD Pipeline / Test (push) Has been cancelled
CI/CD Pipeline / Dev Compose Smoke Test (push) Has been cancelled
CI/CD Pipeline / Build (push) Has been cancelled
CI/CD Pipeline / Test Scripts (push) Has been cancelled
CI/CD Pipeline / Test Native Libraries (push) Has been cancelled
CI/CD Pipeline / Docker Build (push) Has been cancelled
- Add SHA256 verification to build_sqlite.sh (SQLite 3.48.0)
- Add build-sqlite and CLI build steps to ci.yml build job
- Remove cache from benchmark-metrics.yml
2026-02-21 20:58:08 -05:00
Jeremie Fraeys
f6c3e650b5
fix(build): disable openssl in rsync build to fix missing headers
...
Build CLI with Embedded SQLite / build (arm64, aarch64-linux) (push) Waiting to run
Build CLI with Embedded SQLite / build (x86_64, x86_64-linux) (push) Waiting to run
Build CLI with Embedded SQLite / build-macos (arm64) (push) Waiting to run
Build CLI with Embedded SQLite / build-macos (x86_64) (push) Waiting to run
Documentation / build-and-publish (push) Waiting to run
Security Scan / Security Analysis (push) Waiting to run
Security Scan / Native Library Security (push) Waiting to run
Checkout test / test (push) Successful in 5s
CI with Native Libraries / Check Build Environment (push) Successful in 11s
CI/CD Pipeline / Test (push) Has been cancelled
CI/CD Pipeline / Dev Compose Smoke Test (push) Has been cancelled
CI/CD Pipeline / Build (push) Has been cancelled
CI/CD Pipeline / Test Scripts (push) Has been cancelled
CI/CD Pipeline / Security Scan (push) Has been cancelled
CI/CD Pipeline / Docker Build (push) Has been cancelled
CI with Native Libraries / Build Release Libraries (push) Has been cancelled
CI with Native Libraries / Build and Test Native Libraries (push) Has been cancelled
rsync configure requires --disable-openssl when OpenSSL dev headers unavailable
Also removes dependency on openssl/md4.h and openssl/md5.h
2026-02-21 20:48:13 -05:00
Jeremie Fraeys
2e2bba47a2
fix(build): disable pipefail around tar head pipe to prevent SIGPIPE
...
Build CLI with Embedded SQLite / build (arm64, aarch64-linux) (push) Waiting to run
Build CLI with Embedded SQLite / build (x86_64, x86_64-linux) (push) Waiting to run
Build CLI with Embedded SQLite / build-macos (arm64) (push) Waiting to run
Build CLI with Embedded SQLite / build-macos (x86_64) (push) Waiting to run
CI with Native Libraries / Build and Test Native Libraries (push) Blocked by required conditions
CI with Native Libraries / Build Release Libraries (push) Blocked by required conditions
Documentation / build-and-publish (push) Waiting to run
Security Scan / Security Analysis (push) Waiting to run
Security Scan / Native Library Security (push) Waiting to run
Checkout test / test (push) Successful in 5s
CI with Native Libraries / Check Build Environment (push) Successful in 11s
CI/CD Pipeline / Test (push) Failing after 1m3s
CI/CD Pipeline / Dev Compose Smoke Test (push) Has been skipped
CI/CD Pipeline / Build (push) Has been skipped
CI/CD Pipeline / Test Scripts (push) Has been skipped
CI/CD Pipeline / Security Scan (push) Has been cancelled
CI/CD Pipeline / Docker Build (push) Has been cancelled
The tar | head -n 1 | cut pipeline causes SIGPIPE (error 141)
when head closes the pipe before tar finishes writing
2026-02-21 20:46:15 -05:00
Jeremie Fraeys
7bc1c8a982
fix(build): avoid SIGPIPE in rsync SHA256 verification
...
Build CLI with Embedded SQLite / build (arm64, aarch64-linux) (push) Waiting to run
Build CLI with Embedded SQLite / build (x86_64, x86_64-linux) (push) Waiting to run
Build CLI with Embedded SQLite / build-macos (arm64) (push) Waiting to run
Build CLI with Embedded SQLite / build-macos (x86_64) (push) Waiting to run
CI with Native Libraries / Build and Test Native Libraries (push) Blocked by required conditions
CI with Native Libraries / Build Release Libraries (push) Blocked by required conditions
Documentation / build-and-publish (push) Waiting to run
Security Scan / Security Analysis (push) Waiting to run
Security Scan / Native Library Security (push) Waiting to run
Checkout test / test (push) Successful in 5s
CI with Native Libraries / Check Build Environment (push) Successful in 11s
CI/CD Pipeline / Test (push) Failing after 1m3s
CI/CD Pipeline / Dev Compose Smoke Test (push) Has been skipped
CI/CD Pipeline / Build (push) Has been skipped
CI/CD Pipeline / Test Scripts (push) Has been skipped
CI/CD Pipeline / Security Scan (push) Has been cancelled
CI/CD Pipeline / Docker Build (push) Has been cancelled
Replace echo | sha256sum -c - pipeline with direct comparison
Fixes error 141 (SIGPIPE) caused by pipefail with early pipe close
2026-02-21 20:44:27 -05:00
Jeremie Fraeys
97c066af4f
fix(build): add rsync SHA256 hash to skip GPG verification
...
Build CLI with Embedded SQLite / build (arm64, aarch64-linux) (push) Waiting to run
Build CLI with Embedded SQLite / build (x86_64, x86_64-linux) (push) Waiting to run
Build CLI with Embedded SQLite / build-macos (arm64) (push) Waiting to run
Build CLI with Embedded SQLite / build-macos (x86_64) (push) Waiting to run
CI with Native Libraries / Build and Test Native Libraries (push) Blocked by required conditions
CI with Native Libraries / Build Release Libraries (push) Blocked by required conditions
Documentation / build-and-publish (push) Waiting to run
Security Scan / Security Analysis (push) Waiting to run
Security Scan / Native Library Security (push) Waiting to run
Checkout test / test (push) Successful in 5s
CI with Native Libraries / Check Build Environment (push) Successful in 11s
CI/CD Pipeline / Test (push) Failing after 1m3s
CI/CD Pipeline / Dev Compose Smoke Test (push) Has been skipped
CI/CD Pipeline / Build (push) Has been skipped
CI/CD Pipeline / Test Scripts (push) Has been skipped
CI/CD Pipeline / Security Scan (push) Has been cancelled
CI/CD Pipeline / Docker Build (push) Has been cancelled
Add SHA256 for rsync 3.3.0: 7399e9a6708c32d678a72a63219e96f23be0be2336e50fd1348498d07041df90
This allows the build to proceed without requiring GPG keyring setup in CI
2026-02-21 20:42:35 -05:00
Jeremie Fraeys
be39b37aec
feat: native GPU detection and NVML bridge for macOS and Linux
...
- Add dynamic NVML loading for Linux GPU detection
- Add macOS GPU detection via IOKit framework
- Add Zig NVML wrapper for cross-platform GPU queries
- Update native bridge to support platform-specific GPU libs
- Add CMake support for NVML dynamic library
2026-02-21 17:59:59 -05:00
Jeremie Fraeys
1a1844e9e9
fix(cli): remaining ArrayList API fixes in dataset and queue commands
2026-02-21 17:59:51 -05:00
Jeremie Fraeys
b1c9bc97fc
fix(cli): CLI structure, manifest, and asset fixes
...
- Fix commands.zig imports (logs.zig → log.zig, remove missing modules)
- Fix manifest.writeManifest to accept allocator param
- Add db.Stmt type alias for sqlite3_stmt
- Fix rsync placeholder to be valid shell script (#!/bin/sh)
2026-02-21 17:59:20 -05:00
Jeremie Fraeys
382c67edfc
fix(cli): WebSocket protocol and sync command fixes
...
- Add sendSyncRun method for run synchronization
- Add sendRerunRequest method for queue rerun
- Add sync_run (0x26) and rerun_request (0x27) opcodes
- Fix protocol import path to relative path
- Fix db.Stmt type alias usage in sync.zig
2026-02-21 17:59:14 -05:00
Jeremie Fraeys
ccd1dd7a4d
fix(cli): Zig 0.15 core API changes
...
- ArrayList: .init(allocator) → .empty, add allocator param to append/deinit/toOwnedSlice
- Atomic: std.atomic.Atomic → std.atomic.Value, lowercase order names (.seq_cst)
- Process: execvp instead of execvpe, inline wait status macros for macOS
- Time: std.time.sleep → std.Thread.sleep
- Error handling: fix isProcessRunning error union comparison
2026-02-21 17:59:05 -05:00
Jeremie Fraeys
d1ac558107
perf: implement context reuse
...
Go Worker (internal/worker/native_bridge_libs.go):
- Add global hashCtx with sync.Once for lazy initialization
- Eliminates 5-20ms fh_init/fh_cleanup per hash operation
- Uses runtime.NumCPU() for optimal thread count
- Log initialization time for observability
Zig CLI (cli/src/native/hash.zig):
- Add global_ctx with atomic flag and mutex
- Thread-safe initialization with double-check pattern
- Idempotent init() callable from multiple threads
- Log init time for debugging
2026-02-21 14:19:14 -05:00
Jeremie Fraeys
25ae791b5c
refactor: make dataset hash automatic in verify command
...
- Remove separate 'hash' subcommand
- Integrate native SHA256 hash into 'dataset verify'
- Hash is now computed automatically when verifying datasets
- Shows hash in output (JSON, CSV, and text formats)
- Help text updated to indicate auto-hashing
2026-02-21 14:09:44 -05:00
Jeremie Fraeys
1a35c54300
feat: integrate native library into Zig CLI
...
- Add cli/src/native/hash.zig - C ABI wrapper for dataset_hash
- Update cli/src/commands/dataset.zig - Add 'hash' subcommand
- Update cli/build.zig - Link against libdataset_hash.so
- Fix pre-existing CLI errors in experiment.zig (errorMsg signatures, columnInt64)
Usage: ml dataset hash <path>
Note: Additional pre-existing CLI errors remain in sync.zig
2026-02-21 14:08:07 -05:00
Jeremie Fraeys
7efe8bbfbf
native: security hardening, research trustworthiness, and CVE mitigations
...
Security Fixes:
- CVE-2024-45339: Add O_EXCL flag to temp file creation in storage_write_entries()
Prevents symlink attacks on predictable .tmp file paths
- CVE-2025-47290: Use openat_nofollow() in storage_open()
Closes TOCTOU race condition via path_sanitizer infrastructure
- CVE-2025-0838: Add MAX_BATCH_SIZE=10000 to add_tasks()
Prevents integer overflow in batch operations
Research Trustworthiness (dataset_hash):
- Deterministic file ordering: std::sort after collect_files()
- Recursive directory traversal: depth-limited with cycle detection
- Documented exclusions: hidden files and special files noted in API
Bug Fixes:
- R1: storage_init path validation for non-existent directories
- R2: safe_strncpy return value check before strcat
- R3: parallel_hash 256-file cap replaced with std::vector
- R4: wire qi_compact_index/qi_rebuild_index stubs
- R5: CompletionLatch race condition fix (hold mutex during decrement)
- R6: ARMv8 SHA256 transform fix (save abcd_pre before vsha256hq_u32)
- R7: fuzz_index_storage header format fix
- R8: enforce null termination in add_tasks/update_tasks
- R9: use 64 bytes (not 65) in combined hash to exclude null terminator
- R10: status field persistence in save()
New Tests:
- test_recursive_dataset.cpp: Verify deterministic recursive hashing
- test_storage_symlink_resistance.cpp: Verify CVE-2024-45339 fix
- test_queue_index_batch_limit.cpp: Verify CVE-2025-0838 fix
- test_sha256_arm_kat.cpp: ARMv8 known-answer tests
- test_storage_init_new_dir.cpp: F1 verification
- test_parallel_hash_large_dir.cpp: F3 verification
- test_queue_index_compact.cpp: F4 verification
All 8 native tests passing. Library ready for research lab deployment.
2026-02-21 13:33:45 -05:00
Jeremie Fraeys
201cb66f56
fix(cli): Standardize WebSocket client imports
...
- Change from deps.zig indirect imports to direct @import() calls
- Improves build compatibility and clarity
- Aligns with Zig idiomatic import style
2026-02-20 21:41:51 -05:00
Jeremie Fraeys
a3b957dcc0
refactor(cli): Update build system and core infrastructure
...
- Makefile: Update build targets for native library integration
- build.zig: Add SQLite linking and native hash library support
- scripts/build_rsync.sh: Update rsync embedded binary build process
- scripts/build_sqlite.sh: Add SQLite constants generation script
- src/assets/README.md: Document embedded asset structure
- src/utils/rsync_embedded_binary.zig: Update for new build layout
2026-02-20 21:39:51 -05:00
Jeremie Fraeys
04ac745b01
refactor(cli): Rename note to annotate and re-add experiment command
...
- Renamed note.zig to annotate.zig (preserves user's preferred naming)
- Updated all references from 'ml note' to 'ml annotate'
- Re-added experiment.zig with create/list/show subcommands
- Updated main.zig dispatch: 'a' for annotate, 'e' for experiment
- Updated printUsage and test block to reflect changes
2026-02-20 21:32:01 -05:00
Jeremie Fraeys
adf4c2a834
refactor(cli): Update main.zig and remove deprecated commands
...
- main.zig: Update command dispatch and usage text
- Wire up new commands: note, logs, sync, cancel, watch
- Remove deprecated command references
- Updated usage reflects unified command structure
- Delete deprecated command files:
- annotate.zig (replaced by note.zig)
- experiment.zig (functionality in run/note/logs)
- logs.zig (old version, replaced)
- monitor.zig (unused)
- narrative.zig (replaced by note --hypothesis/context)
- outcome.zig (replaced by note --outcome)
- privacy.zig (replaced by note --privacy)
- requeue.zig (functionality merged into queue --rerun)
2026-02-20 21:28:42 -05:00
Jeremie Fraeys
d3461cd07f
feat(cli): Update server integration commands
...
- queue.zig: Add --rerun <run_id> flag to re-queue completed local runs
- Requires server connection, rejects in offline mode with clear error
- HandleRerun function sends rerun request via WebSocket
- sync.zig: Rewrite for WebSocket experiment sync protocol
- Queries unsynced runs from SQLite ml_runs table
- Builds sync JSON with metrics and params
- Sends sync_run message, waits for sync_ack response
- MarkRunSynced updates synced flag in database
- watch.zig: Add --sync flag for continuous experiment sync
- Auto-sync runs to server every 30 seconds when online
- Mode detection with offline error handling
2026-02-20 21:28:34 -05:00
Jeremie Fraeys
f5b68cca49
feat(cli): Add metadata commands and update cancel
...
- note.zig: New unified metadata annotation command
- Supports --text, --hypothesis, --outcome, --confidence, --privacy, --author
- Stores metadata as tags in SQLite ml_tags table
- log.zig: Simplified to unified logs command (fetch/stream only)
- Removed metric/param/tag subcommands (now in run wrapper)
- Supports --follow for live log streaming from server
- cancel.zig: Add local process termination support
- Sends SIGTERM first, waits 5s, then SIGKILL if needed
- Updates run status to CANCELLED in SQLite
- Also supports server job cancellation via WebSocket
2026-02-20 21:28:23 -05:00
Jeremie Fraeys
d0c68772ea
feat(cli): Implement unified run wrapper command
...
- Fork child process and capture stdout/stderr via pipe
- Parse FETCHML_METRIC key=value [step=N] lines from output
- Write run_manifest.json with run metadata
- Insert/update ml_runs table in SQLite with PID tracking
- Stream output to output.log file
- Support entrypoint from config or explicit command after --
2026-02-20 21:28:16 -05:00
Jeremie Fraeys
551597b5df
feat(cli): Add core infrastructure for local mode support
...
- mode.zig: Automatic online/offline mode detection with API ping
- manifest.zig: Run manifest read/write/update operations
- core/: Common flags, output formatting, and context management
- local.zig + local/: Local mode experiment operations
- server.zig + server/: Server mode API client
- db.zig: Add pid column to ml_runs table for process tracking
- config.zig: Add force_local, [experiment] section with name/entrypoint
- utils/native_bridge.zig: Native library integration
2026-02-20 21:28:06 -05:00
Jeremie Fraeys
7583932897
feat(cli): add progress UI and rsync assets
...
- Add progress.zig for sync progress display
- Add rsync placeholder and release binaries to assets/rsync/
2026-02-20 15:51:17 -05:00
Jeremie Fraeys
2258f60ade
feat(cli): add utility modules for local mode
...
- Add hash_cache.zig for efficient file hash caching
- Add ignore.zig for .gitignore-style pattern matching
- Add native_hash.zig for C dataset_hash library integration
2026-02-20 15:51:10 -05:00
Jeremie Fraeys
7ce0fd251e
feat(cli): unified commands and local mode support
...
- Update experiment.zig with unified commands (local + server modes)
- Add init.zig for local project initialization
- Update sync.zig for project synchronization
- Update main.zig to route new local mode commands (experiment, run, log)
- Support automatic mode detection from config (sqlite:// vs wss://)
2026-02-20 15:51:04 -05:00
Jeremie Fraeys
2c596038b5
refactor(cli): update build system and config for local mode
...
- Update Makefile with build-sqlite target matching rsync pattern
- Fix build.zig to handle SQLite assets and dataset_hash linking
- Add SQLite asset detection mirroring rsync binary detection
- Update CLI README with local mode documentation
- Restructure rsync assets into rsync/ subdirectory
- Remove obsolete files (fix_arraylist.sh, old rsync_placeholder.bin)
- Add build_rsync.sh script to fetch/build rsync from source
2026-02-20 15:50:52 -05:00
Jeremie Fraeys
ff542b533f
feat(cli): embed SQLite and unify commands for local mode
...
- Add SQLite amalgamation fetch script (make build-sqlite)
- Embed SQLite in release builds, link system lib in dev
- Create sqlite_embedded.zig utility module
- Unify experiment/run/log commands with auto mode detection
- Add Forgejo CI workflow for building with embedded SQLite
- Update READMEs for local mode and build instructions
SQLite follows rsync embedding pattern: assets/sqlite_release_<os>_<arch>/
Zero external dependencies for release builds.
2026-02-20 15:50:04 -05:00
Jeremie Fraeys
6028779239
feat: update CLI, TUI, and security documentation
...
- Add safety checks to Zig build
- Add TUI with job management and narrative views
- Add WebSocket support and export services
- Add smart configuration defaults
- Update API routes with security headers
- Update SECURITY.md with comprehensive policy
- Add Makefile security scanning targets
2026-02-19 15:35:05 -05:00
Jeremie Fraeys
27c8b08a16
test: Reorganize and add unit tests
...
Reorganize tests for better structure and coverage:
- Move container/security_test.go from internal/ to tests/unit/container/
- Move related tests to proper unit test locations
- Delete orphaned test files (startup_blacklist_test.go)
- Add privacy middleware unit tests
- Add worker config unit tests
- Update E2E tests for homelab and websocket scenarios
- Update test fixtures with utility functions
- Add CLI helper script for arraylist fixes
2026-02-18 21:28:13 -05:00
Jeremie Fraeys
cb826b74a3
feat: WebSocket API infrastructure improvements
...
Enhance WebSocket client and server components:
- Add new WebSocket opcodes (CompareRuns, FindRuns, ExportRun, SetRunOutcome)
- Improve WebSocket client with additional response handlers
- Add crypto utilities for secure WebSocket communications
- Add I/O utilities for WebSocket payload handling
- Enhance validation for WebSocket message payloads
- Update routes for new WebSocket endpoints
- Improve monitor and validate command WebSocket integrations
2026-02-18 21:27:48 -05:00
Jeremie Fraeys
b2eba75f09
feat: CLI shell completion for new commands
...
Update bash and zsh completion scripts to include:
- compare, find, export, outcome commands
- privacy command and subcommands
- All new narrative field flags (--hypothesis, --context, etc.)
- Sandboxing options (--network, --read-only, --secret)
2026-02-18 21:27:38 -05:00
Jeremie Fraeys
aaeef69bab
feat: Privacy and PII detection
...
Add privacy protection features to prevent accidental PII leakage:
- PII detection engine supporting emails, phone numbers, SSNs, credit cards
- CLI privacy command for scanning files and text
- Privacy middleware for API request/response filtering
- Suggestion utility for privacy-preserving alternatives
Integrates PII scanning into manifest validation for narrative fields.
2026-02-18 21:27:23 -05:00
Jeremie Fraeys
260e18499e
feat: Research features - narrative fields and outcome tracking
...
Add comprehensive research context tracking to jobs:
- Narrative fields: hypothesis, context, intent, expected_outcome
- Experiment groups and tags for organization
- Run comparison (compare command) for diff analysis
- Run search (find command) with criteria filtering
- Run export (export command) for data portability
- Outcome setting (outcome command) for experiment validation
Update queue and requeue commands to support narrative fields.
Add narrative validation to manifest validator.
Add WebSocket handlers for compare, find, export, and outcome operations.
Includes E2E tests for phase 2 features.
2026-02-18 21:27:05 -05:00
Jeremie Fraeys
a64233d4f6
fix: CLI now properly rejects unknown commands
...
Checkout test / test (push) Successful in 6s
CI with Native Libraries / Check Build Environment (push) Successful in 11s
CI/CD Pipeline / Test (push) Failing after 5m5s
CI/CD Pipeline / Dev Compose Smoke Test (push) Has been skipped
CI/CD Pipeline / Build (push) Has been skipped
CI/CD Pipeline / Test Scripts (push) Has been skipped
CI/CD Pipeline / Security Scan (push) Failing after 4m49s
Documentation / build-and-publish (push) Failing after 24s
CI with Native Libraries / Build and Test Native Libraries (push) Failing after 14m40s
CI with Native Libraries / Build Release Libraries (push) Has been skipped
CI/CD Pipeline / Docker Build (push) Has been skipped
- Add handleUnknownCommand() helper function
- Add else clauses to 'i' and 's' switch cases
- Commands like 'invalid_command', 'infoooo', 'syncccc' now properly rejected
- Prevents silent acceptance of invalid commands
- Test TestCLICommandsE2E/CLIErrorHandling now passes
2026-02-18 16:04:37 -05:00
Jeremie Fraeys
cd2908181c
refactor(cli): reduce code duplication in WebSocket client
...
Add MessageBuilder struct and validation helpers to reduce duplication:
- MessageBuilder: Helper for constructing binary WebSocket messages
- writeOpcode, writeBytes, writeU8/16/32/64
- writeStringU8, writeStringU16 for length-prefixed strings
- Validation helpers: validateApiKeyHash, validateCommitId, validateJobName
- getStream(): Extracted common stream check pattern
Refactored 4 representative send methods to use new helpers:
- sendValidateRequestCommit, sendListJupyterPackages
- sendCancelJob, sendStatusRequest
Consolidated disconnect/close: close() now calls disconnect()
Updated response handlers to use packet.deinit():
- receiveAndHandleResponse
- receiveAndHandleDatasetResponse
Reduces ~100 lines of boilerplate duplication.
Build verified: zig build --release=fast
2026-02-18 13:56:30 -05:00
Jeremie Fraeys
14eba436bf
feat(cli): include command name in error messages and crash reports
...
Implement TODO in handleCommandError:
- Log command name with error for crash report context
- Display 'Error in \'{s}': {s}' instead of generic 'Error: {s}'
- Helps users identify which command failed
Build verified: zig build --release=fast
2026-02-18 13:51:13 -05:00
Jeremie Fraeys
32e65545ae
feat(cli): implement proper JSON parsing for sync progress
...
Use utils/json.zig helpers to parse sync progress messages:
- Add json module import
- Rename json variable to json_mode to avoid shadowing
- Parse status, progress, total fields from JSON response
- Show percentage completion for in-progress syncs
- Handle 'complete' and 'error' status codes
Build verified: zig build --release=fast
2026-02-18 13:49:01 -05:00
Jeremie Fraeys
55b4b23645
feat(cli): integrate embedded rsync for remote sync
...
Replace local file copy with embedded rsync binary for actual remote
synchronization to the server. The embedded rsync is extracted from
assets/ at runtime - no external dependencies required.
Changes:
- Re-add rsync_embedded.zig import
- Replace manual file copying with rsync.sync() call
- Construct remote path as api_key@host:worker_base/commit_id/files/
- Update JSON output to include commit_id
Build verified: zig build --release=fast
2026-02-18 13:45:50 -05:00
Jeremie Fraeys
c31055be30
refactor(cli): remove unused imports from sync.zig
...
Remove unused imports:
- rsync (commented as 'Use embedded rsync' but never used)
- storage (never referenced)
Build verified: zig build --release=fast
2026-02-18 13:39:16 -05:00
Jeremie Fraeys
999af75f39
refactor(cli): use config.getWebSocketUrl() in jupyter.zig
...
Replace 6 instances of inline WebSocket URL construction with the
config.getWebSocketUrl() helper method. This ensures consistent URL
formatting and reduces code duplication.
Functions updated:
- startJupyter
- stopJupyter
- removeJupyter
- listServices
- packageCommands
Build verified: zig build --release=fast
2026-02-18 13:36:03 -05:00
