Commit graph

15 commits

Author SHA1 Message Date
Jeremie Fraeys
2b7319dc2e
refactor(cli): Simplify output system and add terminal utilities
Remove colors dependency from output.zig

Add terminal.zig for TTY detection and terminal width

Update flags.zig with color flag support

Simplify colors.zig to basic ANSI codes

Update main.zig and utils.zig exports
2026-02-23 14:11:59 -05:00
Jeremie Fraeys
7efe8bbfbf
native: security hardening, research trustworthiness, and CVE mitigations
Security Fixes:
- CVE-2024-45339: Add O_EXCL flag to temp file creation in storage_write_entries()
  Prevents symlink attacks on predictable .tmp file paths
- CVE-2025-47290: Use openat_nofollow() in storage_open()
  Closes TOCTOU race condition via path_sanitizer infrastructure
- CVE-2025-0838: Add MAX_BATCH_SIZE=10000 to add_tasks()
  Prevents integer overflow in batch operations

Research Trustworthiness (dataset_hash):
- Deterministic file ordering: std::sort after collect_files()
- Recursive directory traversal: depth-limited with cycle detection
- Documented exclusions: hidden files and special files noted in API

Bug Fixes:
- R1: storage_init path validation for non-existent directories
- R2: safe_strncpy return value check before strcat
- R3: parallel_hash 256-file cap replaced with std::vector
- R4: wire qi_compact_index/qi_rebuild_index stubs
- R5: CompletionLatch race condition fix (hold mutex during decrement)
- R6: ARMv8 SHA256 transform fix (save abcd_pre before vsha256hq_u32)
- R7: fuzz_index_storage header format fix
- R8: enforce null termination in add_tasks/update_tasks
- R9: use 64 bytes (not 65) in combined hash to exclude null terminator
- R10: status field persistence in save()

New Tests:
- test_recursive_dataset.cpp: Verify deterministic recursive hashing
- test_storage_symlink_resistance.cpp: Verify CVE-2024-45339 fix
- test_queue_index_batch_limit.cpp: Verify CVE-2025-0838 fix
- test_sha256_arm_kat.cpp: ARMv8 known-answer tests
- test_storage_init_new_dir.cpp: F1 verification
- test_parallel_hash_large_dir.cpp: F3 verification
- test_queue_index_compact.cpp: F4 verification

All 8 native tests passing. Library ready for research lab deployment.
2026-02-21 13:33:45 -05:00
Jeremie Fraeys
04ac745b01
refactor(cli): Rename note to annotate and re-add experiment command
- Renamed note.zig to annotate.zig (preserves user's preferred naming)
- Updated all references from 'ml note' to 'ml annotate'
- Re-added experiment.zig with create/list/show subcommands
- Updated main.zig dispatch: 'a' for annotate, 'e' for experiment
- Updated printUsage and test block to reflect changes
2026-02-20 21:32:01 -05:00
Jeremie Fraeys
adf4c2a834
refactor(cli): Update main.zig and remove deprecated commands
- main.zig: Update command dispatch and usage text
  - Wire up new commands: note, logs, sync, cancel, watch
  - Remove deprecated command references
  - Updated usage reflects unified command structure
- Delete deprecated command files:
  - annotate.zig (replaced by note.zig)
  - experiment.zig (functionality in run/note/logs)
  - logs.zig (old version, replaced)
  - monitor.zig (unused)
  - narrative.zig (replaced by note --hypothesis/context)
  - outcome.zig (replaced by note --outcome)
  - privacy.zig (replaced by note --privacy)
  - requeue.zig (functionality merged into queue --rerun)
2026-02-20 21:28:42 -05:00
Jeremie Fraeys
7ce0fd251e
feat(cli): unified commands and local mode support
- Update experiment.zig with unified commands (local + server modes)
- Add init.zig for local project initialization
- Update sync.zig for project synchronization
- Update main.zig to route new local mode commands (experiment, run, log)
- Support automatic mode detection from config (sqlite:// vs wss://)
2026-02-20 15:51:04 -05:00
Jeremie Fraeys
260e18499e
feat: Research features - narrative fields and outcome tracking
Add comprehensive research context tracking to jobs:
- Narrative fields: hypothesis, context, intent, expected_outcome
- Experiment groups and tags for organization
- Run comparison (compare command) for diff analysis
- Run search (find command) with criteria filtering
- Run export (export command) for data portability
- Outcome setting (outcome command) for experiment validation

Update queue and requeue commands to support narrative fields.
Add narrative validation to manifest validator.
Add WebSocket handlers for compare, find, export, and outcome operations.

Includes E2E tests for phase 2 features.
2026-02-18 21:27:05 -05:00
Jeremie Fraeys
a64233d4f6
fix: CLI now properly rejects unknown commands
Some checks failed
Checkout test / test (push) Successful in 6s
CI with Native Libraries / Check Build Environment (push) Successful in 11s
CI/CD Pipeline / Test (push) Failing after 5m5s
CI/CD Pipeline / Dev Compose Smoke Test (push) Has been skipped
CI/CD Pipeline / Build (push) Has been skipped
CI/CD Pipeline / Test Scripts (push) Has been skipped
CI/CD Pipeline / Security Scan (push) Failing after 4m49s
Documentation / build-and-publish (push) Failing after 24s
CI with Native Libraries / Build and Test Native Libraries (push) Failing after 14m40s
CI with Native Libraries / Build Release Libraries (push) Has been skipped
CI/CD Pipeline / Docker Build (push) Has been skipped
- Add handleUnknownCommand() helper function
- Add else clauses to 'i' and 's' switch cases
- Commands like 'invalid_command', 'infoooo', 'syncccc' now properly rejected
- Prevents silent acceptance of invalid commands
- Test TestCLICommandsE2E/CLIErrorHandling now passes
2026-02-18 16:04:37 -05:00
Jeremie Fraeys
29d130dcdb
refactor(cli): standardize error handling and use packet.deinit()
- Replace std.process.exit(1) with error.InvalidArgs in sync command
- Replace std.process.exit(1) with error.ValidationFailed in validate command
- Update validate.zig to use protocol.ResponsePacket.deinit() for cleanup
- Build verified: zig build --release=fast
2026-02-18 13:29:53 -05:00
Jeremie Fraeys
9517271bbb
fix(cli): remove unreachable code and unused Command enum
- Fix duplicate 'i' case in Command.fromString that was unreachable
- Remove unused Command enum entirely (dispatch uses inline comparisons)
- Build verified: zig build --release=fast
2026-02-18 13:26:18 -05:00
Jeremie Fraeys
1147958e15
feat: enhance CLI with improved commands and WebSocket handling
- Refactor command structure for better organization
- Improve WebSocket client frame handling
- Add response handler improvements
- Update queue, requeue, and status commands
- Add security module for CLI authentication
2026-02-16 20:38:08 -05:00
Jeremie Fraeys
8e3fa94322
feat(cli): enhance Zig CLI with new commands and improved networking
- Add new commands: annotate, narrative, requeue
- Refactor WebSocket client into modular components (net/ws/)
- Add rsync embedded binary support
- Improve error handling and response packet processing
- Update build.zig and completions
2026-02-12 12:05:10 -05:00
Jeremie Fraeys
5ef24e4c6d feat(cli): add validate/info commands and improve protocol handling 2026-01-05 12:31:20 -05:00
Jeremie Fraeys
cd5640ebd2 Slim and secure: move scripts, clean configs, remove secrets
- Move ci-test.sh and setup.sh to scripts/
- Trim docs/src/zig-cli.md to current structure
- Replace hardcoded secrets with placeholders in configs
- Update .gitignore to block .env*, secrets/, keys, build artifacts
- Slim README.md to reflect current CLI/TUI split
- Add cleanup trap to ci-test.sh
- Ensure no secrets are committed
2025-12-07 13:57:51 -05:00
Jeremie Fraeys
b75bd24bba Add CLI jupyter command for transparent Jupyter management
- Add jupyter.zig command with start/stop/status actions
- Update main.zig to include jupyter command in CLI
- CLI now handles all Jupyter setup transparently
- Data scientists only need: ml jupyter start
- Auto-detects container runtime (Podman/Docker)
- Manages container lifecycle automatically
- Provides clear status and error messages
2025-12-06 16:07:09 -05:00
Jeremie Fraeys
d225ea1f00 feat: implement Zig CLI with comprehensive ML experiment management
- Add modern CLI interface built with Zig for performance
- Include TUI (Terminal User Interface) with bubbletea-like features
- Implement ML experiment commands (run, status, manage)
- Add configuration management and validation
- Include shell completion scripts for bash and zsh
- Add comprehensive CLI testing framework
- Support for multiple ML frameworks and project types

CLI provides fast, efficient interface for ML experiment management
with modern terminal UI and comprehensive feature set.
2025-12-04 16:53:58 -05:00