Commit graph

16 commits

Author SHA1 Message Date
Jeremie Fraeys
ccd1dd7a4d
fix(cli): Zig 0.15 core API changes
- ArrayList: .init(allocator) → .empty, add allocator param to append/deinit/toOwnedSlice
- Atomic: std.atomic.Atomic → std.atomic.Value, lowercase order names (.seq_cst)
- Process: execvp instead of execvpe, inline wait status macros for macOS
- Time: std.time.sleep → std.Thread.sleep
- Error handling: fix isProcessRunning error union comparison
2026-02-21 17:59:05 -05:00
Jeremie Fraeys
7efe8bbfbf
native: security hardening, research trustworthiness, and CVE mitigations
Security Fixes:
- CVE-2024-45339: Add O_EXCL flag to temp file creation in storage_write_entries()
  Prevents symlink attacks on predictable .tmp file paths
- CVE-2025-47290: Use openat_nofollow() in storage_open()
  Closes TOCTOU race condition via path_sanitizer infrastructure
- CVE-2025-0838: Add MAX_BATCH_SIZE=10000 to add_tasks()
  Prevents integer overflow in batch operations

Research Trustworthiness (dataset_hash):
- Deterministic file ordering: std::sort after collect_files()
- Recursive directory traversal: depth-limited with cycle detection
- Documented exclusions: hidden files and special files noted in API

Bug Fixes:
- R1: storage_init path validation for non-existent directories
- R2: safe_strncpy return value check before strcat
- R3: parallel_hash 256-file cap replaced with std::vector
- R4: wire qi_compact_index/qi_rebuild_index stubs
- R5: CompletionLatch race condition fix (hold mutex during decrement)
- R6: ARMv8 SHA256 transform fix (save abcd_pre before vsha256hq_u32)
- R7: fuzz_index_storage header format fix
- R8: enforce null termination in add_tasks/update_tasks
- R9: use 64 bytes (not 65) in combined hash to exclude null terminator
- R10: status field persistence in save()

New Tests:
- test_recursive_dataset.cpp: Verify deterministic recursive hashing
- test_storage_symlink_resistance.cpp: Verify CVE-2024-45339 fix
- test_queue_index_batch_limit.cpp: Verify CVE-2025-0838 fix
- test_sha256_arm_kat.cpp: ARMv8 known-answer tests
- test_storage_init_new_dir.cpp: F1 verification
- test_parallel_hash_large_dir.cpp: F3 verification
- test_queue_index_compact.cpp: F4 verification

All 8 native tests passing. Library ready for research lab deployment.
2026-02-21 13:33:45 -05:00
Jeremie Fraeys
a3b957dcc0
refactor(cli): Update build system and core infrastructure
- Makefile: Update build targets for native library integration
- build.zig: Add SQLite linking and native hash library support
- scripts/build_rsync.sh: Update rsync embedded binary build process
- scripts/build_sqlite.sh: Add SQLite constants generation script
- src/assets/README.md: Document embedded asset structure
- src/utils/rsync_embedded_binary.zig: Update for new build layout
2026-02-20 21:39:51 -05:00
Jeremie Fraeys
551597b5df
feat(cli): Add core infrastructure for local mode support
- mode.zig: Automatic online/offline mode detection with API ping
- manifest.zig: Run manifest read/write/update operations
- core/: Common flags, output formatting, and context management
- local.zig + local/: Local mode experiment operations
- server.zig + server/: Server mode API client
- db.zig: Add pid column to ml_runs table for process tracking
- config.zig: Add force_local, [experiment] section with name/entrypoint
- utils/native_bridge.zig: Native library integration
2026-02-20 21:28:06 -05:00
Jeremie Fraeys
2258f60ade
feat(cli): add utility modules for local mode
- Add hash_cache.zig for efficient file hash caching
- Add ignore.zig for .gitignore-style pattern matching
- Add native_hash.zig for C dataset_hash library integration
2026-02-20 15:51:10 -05:00
Jeremie Fraeys
ff542b533f
feat(cli): embed SQLite and unify commands for local mode
- Add SQLite amalgamation fetch script (make build-sqlite)
- Embed SQLite in release builds, link system lib in dev
- Create sqlite_embedded.zig utility module
- Unify experiment/run/log commands with auto mode detection
- Add Forgejo CI workflow for building with embedded SQLite
- Update READMEs for local mode and build instructions

SQLite follows rsync embedding pattern: assets/sqlite_release_<os>_<arch>/
Zero external dependencies for release builds.
2026-02-20 15:50:04 -05:00
Jeremie Fraeys
6028779239
feat: update CLI, TUI, and security documentation
- Add safety checks to Zig build
- Add TUI with job management and narrative views
- Add WebSocket support and export services
- Add smart configuration defaults
- Update API routes with security headers
- Update SECURITY.md with comprehensive policy
- Add Makefile security scanning targets
2026-02-19 15:35:05 -05:00
Jeremie Fraeys
cb826b74a3
feat: WebSocket API infrastructure improvements
Enhance WebSocket client and server components:
- Add new WebSocket opcodes (CompareRuns, FindRuns, ExportRun, SetRunOutcome)
- Improve WebSocket client with additional response handlers
- Add crypto utilities for secure WebSocket communications
- Add I/O utilities for WebSocket payload handling
- Enhance validation for WebSocket message payloads
- Update routes for new WebSocket endpoints
- Improve monitor and validate command WebSocket integrations
2026-02-18 21:27:48 -05:00
Jeremie Fraeys
aaeef69bab
feat: Privacy and PII detection
Add privacy protection features to prevent accidental PII leakage:
- PII detection engine supporting emails, phone numbers, SSNs, credit cards
- CLI privacy command for scanning files and text
- Privacy middleware for API request/response filtering
- Suggestion utility for privacy-preserving alternatives

Integrates PII scanning into manifest validation for narrative fields.
2026-02-18 21:27:23 -05:00
Jeremie Fraeys
1597c20b73
refactor(cli): consolidate shared utilities and remove code duplication
Extract common helper functions from multiple command files into shared
utility modules:

- Create cli/src/utils/json.zig with json.getString(), getInt(), getFloat(), getBool()
- Create cli/src/utils/manifest.zig with readFileAlloc(), resolvePathWithBase(),
  resolvePathById(), readJobNameFromManifest()
- Add ResponsePacket.deinit() method to net/protocol.zig for consistent cleanup
- Update info.zig, annotate.zig, narrative.zig, requeue.zig to use shared utilities
- Update utils.zig exports for new modules

Eliminates duplicate implementations of:
- jsonGetString() and jsonGetInt() in 4 files
- readFileAlloc() in 4 files
- resolveManifestPath*() functions in 4 files
- ResponsePacket cleanup defer blocks (replaced with .deinit())

Builds cleanly with zig build --release=fast
2026-02-18 13:19:40 -05:00
Jeremie Fraeys
c85575048f
refactor(cli): consolidate shared types and reduce code duplication
Extract common UserContext and authentication logic from cancel.zig and
status.zig into new utils/auth.zig module. Add CommonFlags struct to
utils/flags.zig for shared CLI flags. Add getWebSocketUrl() helper to
Config to eliminate URL construction duplication.

Changes:
- Create cli/src/utils/auth.zig with UserContext and authenticateUser()
- Create cli/src/utils/flags.zig with CommonFlags struct
- Update cancel.zig and status.zig to use shared modules
- Add getWebSocketUrl() helper to config.zig
- Export new modules from utils.zig

Reduces code duplication and improves separation of concerns in the
Zig CLI codebase.
2026-02-18 13:00:48 -05:00
Jeremie Fraeys
8e3fa94322
feat(cli): enhance Zig CLI with new commands and improved networking
- Add new commands: annotate, narrative, requeue
- Refactor WebSocket client into modular components (net/ws/)
- Add rsync embedded binary support
- Improve error handling and response packet processing
- Update build.zig and completions
2026-02-12 12:05:10 -05:00
Jeremie Fraeys
5ef24e4c6d feat(cli): add validate/info commands and improve protocol handling 2026-01-05 12:31:20 -05:00
Jeremie Fraeys
cd5640ebd2 Slim and secure: move scripts, clean configs, remove secrets
- Move ci-test.sh and setup.sh to scripts/
- Trim docs/src/zig-cli.md to current structure
- Replace hardcoded secrets with placeholders in configs
- Update .gitignore to block .env*, secrets/, keys, build artifacts
- Slim README.md to reflect current CLI/TUI split
- Add cleanup trap to ci-test.sh
- Ensure no secrets are committed
2025-12-07 13:57:51 -05:00
Jeremie Fraeys
ea15af1833 Fix multi-user authentication and clean up debug code
- Fix YAML tags in auth config struct (json -> yaml)
- Update CLI configs to use pre-hashed API keys
- Remove double hashing in WebSocket client
- Fix port mapping (9102 -> 9103) in CLI commands
- Update permission keys to use jobs:read, jobs:create, etc.
- Clean up all debug logging from CLI and server
- All user roles now authenticate correctly:
  * Admin: Can queue jobs and see all jobs
  * Researcher: Can queue jobs and see own jobs
  * Analyst: Can see status (read-only access)

Multi-user authentication is now fully functional.
2025-12-06 12:35:32 -05:00
Jeremie Fraeys
d225ea1f00 feat: implement Zig CLI with comprehensive ML experiment management
- Add modern CLI interface built with Zig for performance
- Include TUI (Terminal User Interface) with bubbletea-like features
- Implement ML experiment commands (run, status, manage)
- Add configuration management and validation
- Include shell completion scripts for bash and zsh
- Add comprehensive CLI testing framework
- Support for multiple ML frameworks and project types

CLI provides fast, efficient interface for ML experiment management
with modern terminal UI and comprehensive feature set.
2025-12-04 16:53:58 -05:00