272 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Jeremie Fraeys	305e1b3f2e	ci: update test and benchmark scripts ... **scripts/benchmarks/run-benchmarks-local.sh:** - Add support for native library benchmarks **scripts/ci/test.sh:** - Update CI test commands for new test structure **scripts/dev/smoke-test.sh:** - Improve smoke test reliability and output	2026-02-23 18:04:01 -05:00
Jeremie Fraeys	be67cb77d3	test(benchmarks): update benchmark tests with job cleanup and improvements ... **Payload Performance Test:** - Add job cleanup after each iteration using DeleteJob() - Ensure isolated memory measurements between test runs **All Benchmark Tests:** - General improvements and maintenance updates	2026-02-23 18:03:54 -05:00
Jeremie Fraeys	54ddab887e	build: update Makefile and Zig build for new targets ... **Makefile:** - Add native build targets and test infrastructure - Update benchmark and CI test commands **cli/build.zig:** - Build configuration updates for CLI compilation	2026-02-23 18:03:47 -05:00
Jeremie Fraeys	a70d8aad8e	refactor: remove dead code and fix unused variables ... **Cleanup:** - Delete internal/worker/testutil.go (150 lines of unused test utilities) - Remove unused stateDir() function from internal/jupyter/service_manager.go - Silence unused variable warning in internal/worker/executor/container.go	2026-02-23 18:03:38 -05:00
Jeremie Fraeys	b00439b86e	docs(security): document comprehensive security hardening ... Updates documentation with new security features and hardening guide: **CHANGELOG.md:** - Added detailed security hardening section (2026-02-23) - Documents all phases: file ingestion, sandbox, secrets, audit logging, tests - Lists specific files changed and security controls implemented **docs/src/security.md:** - Added Overview section with defense-in-depth layers - Added Comprehensive Security Hardening section with: - File ingestion security with code examples - Sandbox hardening with complete YAML config - Secrets management with env expansion syntax - HIPAA audit logging with tamper-evident chain hashing	2026-02-23 18:03:25 -05:00
Jeremie Fraeys	fccced6bb3	test(security): add comprehensive security unit tests ... Adds 13 security tests across 4 files for hardening verification: **Path Traversal Tests (path_traversal_test.go):** - TestSecurePathValidator_ValidRelativePath - TestSecurePathValidator_PathTraversalBlocked - TestSecurePathValidator_SymlinkEscape - Tests symlink resolution and path boundary enforcement **File Type Validation Tests (filetype_test.go):** - TestValidateFileType_AllowedTypes - TestValidateFileType_DangerousTypesBlocked - TestValidateModelFile - Tests magic bytes validation and dangerous extension blocking **Secrets Management Tests (secrets_test.go):** - TestExpandSecrets_BasicExpansion - TestExpandSecrets_NestedAndMissingVars - TestValidateNoPlaintextSecrets_HeuristicDetection - Tests env variable expansion and plaintext secret detection with entropy **Audit Logging Tests (audit_test.go):** - TestAuditLogger_ChainIntegrity - TestAuditLogger_VerifyChain - TestAuditLogger_LogFileAccess - TestAuditLogger_Disabled - Tests tamper-evident chain hashing and file access logging	2026-02-23 18:00:45 -05:00
Jeremie Fraeys	92aab06d76	feat(security): implement comprehensive security hardening phases 1-5,7 ... Implements defense-in-depth security for HIPAA and multi-tenant requirements: **Phase 1 - File Ingestion Security:** - SecurePathValidator with symlink resolution and path boundary enforcement in internal/fileutil/secure.go - Magic bytes validation for ML artifacts (safetensors, GGUF, HDF5, numpy) in internal/fileutil/filetype.go - Dangerous extension blocking (.pt, .pkl, .exe, .sh, .zip) - Upload limits (10GB size, 100MB/s rate, 10 uploads/min) **Phase 2 - Sandbox Hardening:** - ApplySecurityDefaults() with secure-by-default principle - network_mode: none, read_only_root: true, no_new_privileges: true - drop_all_caps: true, user_ns: true, run_as_uid/gid: 1000 - PodmanSecurityConfig and BuildSecurityArgs() in internal/container/podman.go - BuildPodmanCommand now accepts full security configuration - Container executor passes SandboxConfig to Podman command builder - configs/seccomp/default-hardened.json blocks dangerous syscalls (ptrace, mount, reboot, kexec_load, open_by_handle_at) **Phase 3 - Secrets Management:** - expandSecrets() for environment variable expansion using ${VAR} syntax - validateNoPlaintextSecrets() with entropy-based detection - Pattern matching for AWS, GitHub, GitLab, OpenAI, Stripe tokens - Shannon entropy calculation (>4 bits/char triggers detection) - Secrets expanded during LoadConfig() before validation **Phase 5 - HIPAA Audit Logging:** - Tamper-evident chain hashing with SHA-256 in internal/audit/audit.go - Event struct extended with PrevHash, EventHash, SequenceNum - File access event types: EventFileRead, EventFileWrite, EventFileDelete - LogFileAccess() helper for HIPAA compliance - VerifyChain() function for tamper detection **Supporting Changes:** - Add DeleteJob() and DeleteJobsByPrefix() to storage package - Integrate SecurePathValidator in artifact scanning	2026-02-23 18:00:33 -05:00
Jeremie Fraeys	aed59967b7	fix(make): Reduce profile-ws-queue test count to prevent timeouts ... Change -count=5 to -count=2 to avoid resource contention 5 sequential runs with 60s timeout each could exceed reasonable time limits	2026-02-23 14:44:34 -05:00
Jeremie Fraeys	ec9e845bb6	fix(test): Fix WebSocketQueue test timeout and race conditions ... Reduce worker polling interval from 5ms to 1ms for faster task pickup Add 100ms buffer after job submission to allow queue to settle Increase timeout from 30s to 60s to prevent flaky failures Fixes intermittent timeout issues in integration tests	2026-02-23 14:38:18 -05:00
Jeremie Fraeys	551e6d4dbc	fix(make): Create tests/bin directory for CPU profiling output ... Add @mkdir -p tests/bin to profile-load, profile-load-norate, and profile-ws-queue targets Fixes 'no such file or directory' error when writing CPU profile files	2026-02-23 14:31:08 -05:00
Jeremie Fraeys	7d1ba75092	chore: Update security scan workflow and SQLite build script	2026-02-23 14:24:00 -05:00
Jeremie Fraeys	6d200b5ac2	fix(docker): Use named volume for Redis to fix permission errors ... Replace bind mount with Docker named volume for Redis data This fixes 'operation not permitted' errors on macOS Docker Desktop where bind mounts fail due to file sharing restrictions	2026-02-23 14:20:23 -05:00
Jeremie Fraeys	0ea2ac00cd	fix(scripts): Create data directories before starting Docker ... Fix Docker mount permission error by creating data/dev/* directories before docker-compose up, preventing 'operation not permitted' error	2026-02-23 14:17:37 -05:00
Jeremie Fraeys	ab20212d07	test: Update duplicate detection tests	2026-02-23 14:14:21 -05:00
Jeremie Fraeys	fa97521488	chore(scripts): Update CI, dev, release, and testing scripts	2026-02-23 14:13:55 -05:00
Jeremie Fraeys	2fdc9b9218	docs: Update privacy/security and research runner docs ... Update native test storage	2026-02-23 14:13:35 -05:00
Jeremie Fraeys	abd27bf0a2	refactor(go): Update Go commands and TUI controller ... Update api-server and gen-keys main files Update TUI controller commands, helpers, and settings	2026-02-23 14:13:14 -05:00
Jeremie Fraeys	6faa13aabf	refactor(cli): Remove progress UI and update native/server code ... Delete cli/src/ui/progress.zig (removing progress bars/spinners) Update native GPU detection modules Update server experiment API	2026-02-23 14:12:48 -05:00
Jeremie Fraeys	fd317c9791	refactor(cli): Update network handlers and info command ... Replace Unicode symbols with ASCII in handshake.zig Add [OK]/[FAIL] status indicators in response_handlers.zig Simplify info.zig output formatting	2026-02-23 14:12:22 -05:00
Jeremie Fraeys	2b7319dc2e	refactor(cli): Simplify output system and add terminal utilities ... Remove colors dependency from output.zig Add terminal.zig for TTY detection and terminal width Update flags.zig with color flag support Simplify colors.zig to basic ANSI codes Update main.zig and utils.zig exports	2026-02-23 14:11:59 -05:00
Jeremie Fraeys	a1988de8b1	style(cli): Standardize printUsage() formatting with tabs and ASCII symbols ... Replace space-padding with consistent tab (\t) alignment in all printUsage() functions. Add ligature-friendly ASCII symbols: - => for results/outcomes (renders as ⇒ with ligatures) - ~> for modifications/changes (renders as ~> with ligatures) - -> for state transitions (renders as → with ligatures) - [OK] / [FAIL] for status indicators All symbols use ASCII 32-126 for xargs-safe, copy-pasteable output.	2026-02-23 14:09:49 -05:00
Jeremie Fraeys	3b194ff2e8	feat: GPU detection transparency and artifact scanner improvements ... - Surface GPUDetectionInfo from parseGPUCountFromConfig for detection metadata - Document FETCH_ML_TOTAL_CPU and FETCH_ML_GPU_SLOTS_PER_GPU env vars - Add debug logging for all env var overrides to stderr - Track config-layer auto-detection in GPUDetectionInfo.ConfigLayerAutoDetected - Add --include-all flag to artifact scanner (includeAll parameter) - Add AMD production mode enforcement (error in non-local mode) - Add GPU detector unit tests for env overrides and AMD aliasing	2026-02-23 12:29:34 -05:00
Jeremie Fraeys	f987ddb86c	chore: remove AI assistant files from repository ... Remove AGENTS.md and .windsurf/ directory from git tracking Files remain in .gitignore to prevent re-adding	2026-02-23 11:23:22 -05:00
Jeremie Fraeys	f2abd94aa6	fix(docs): fix Hugo installation path detection ... - Find Hugo binary dynamically after dpkg install - Create symlink to /usr/bin/hugo for consistency	2026-02-23 11:22:33 -05:00
Jeremie Fraeys	8789fcbe94	chore: add AI assistant files to .gitignore, update AGENTS.md ... - Ignore AGENTS.md and .windsurf/* in .gitignore - Expand AGENTS.md with native lib and Zig CLI build commands	2026-02-23 11:22:22 -05:00
Jeremie Fraeys	47ad62648d	test(e2e): skip gracefully when Redis unavailable, fix cross-device link ... - StartTemporaryRedis now skips tests instead of failing when redis-server unavailable - Fix homelab_e2e_test cross-device link issue using CopyDir instead of Rename	2026-02-21 21:20:47 -05:00
Jeremie Fraeys	bf4a8bcf78	test(auth): skip keychain tests when dbus unavailable	2026-02-21 21:20:03 -05:00
Jeremie Fraeys	1b0781dc68	fix(auth): make DeleteAPIKey resilient to keyring errors ... DeleteAPIKey now ignores primary keyring errors (e.g., dbus unavailable) and always cleans up the fallback store	2026-02-21 21:19:46 -05:00
Jeremie Fraeys	ec568b4c85	fix(build): link libc for Zig tests using C imports	2026-02-21 21:19:09 -05:00
Jeremie Fraeys	03ed1a05bf	fix(build): add iterate permission for test directory ... Zig 0.15 requires .iterate = true flag when opening directories for iteration Fixes panic: reached unreachable code in std.fs.Dir.iterate	2026-02-21 21:10:48 -05:00
Jeremie Fraeys	79faf9556e	ci: fix version mismatches in build-cli.yml ... - Update Zig from 0.15.0 to 0.15.2 (consistent with ci.yml) - Update SQLite from 3450000 to 3480000 (consistent with Makefile)	2026-02-21 21:03:43 -05:00
Jeremie Fraeys	39bf466737	refactor(build): fetch SHA256 from official sources ... - SQLite: fetch from sqlite.org/<YEAR>/<file>.zip.sha256 with embedded fallback - Rsync: fetch from download.samba.org/.../<file>.tar.gz.sha256 with embedded fallback - Remove hardcoded SHA256 requirement when official checksums available	2026-02-21 21:00:23 -05:00
Jeremie Fraeys	2d66a85abc	ci: add SHA256 verification for SQLite and fix CLI build ... - Add SHA256 verification to build_sqlite.sh (SQLite 3.48.0) - Add build-sqlite and CLI build steps to ci.yml build job - Remove cache from benchmark-metrics.yml	2026-02-21 20:58:08 -05:00
Jeremie Fraeys	aaef56e13e	ci: merge ci-native.yml into ci.yml and remove duplicates ... - Merge test-native job from ci-native.yml into ci.yml - Remove duplicate security-scan job (kept in security-scan.yml) - Remove cache steps from merged native job (infra not available) - Update Go version to 1.25.0 for consistency - Update docker-build needs to include test-native - Delete ci-native.yml	2026-02-21 20:54:18 -05:00
Jeremie Fraeys	f6c3e650b5	fix(build): disable openssl in rsync build to fix missing headers ... rsync configure requires --disable-openssl when OpenSSL dev headers unavailable Also removes dependency on openssl/md4.h and openssl/md5.h	2026-02-21 20:48:13 -05:00
Jeremie Fraeys	2e2bba47a2	fix(build): disable pipefail around tar head pipe to prevent SIGPIPE ... The tar | head -n 1 | cut pipeline causes SIGPIPE (error 141) when head closes the pipe before tar finishes writing	2026-02-21 20:46:15 -05:00
Jeremie Fraeys	7bc1c8a982	fix(build): avoid SIGPIPE in rsync SHA256 verification ... Replace echo | sha256sum -c - pipeline with direct comparison Fixes error 141 (SIGPIPE) caused by pipefail with early pipe close	2026-02-21 20:44:27 -05:00
Jeremie Fraeys	97c066af4f	fix(build): add rsync SHA256 hash to skip GPG verification ... Add SHA256 for rsync 3.3.0: 7399e9a6708c32d678a72a63219e96f23be0be2336e50fd1348498d07041df90 This allows the build to proceed without requiring GPG keyring setup in CI	2026-02-21 20:42:35 -05:00
Jeremie Fraeys	5f2fc66200	fix(ci): correct Zig download URLs to ziglang.org format ... Change from incorrect GitHub releases URL to official ziglang.org URL Format: ziglang.org/download/{VERSION}/zig-{ARCH}-{OS}-{VERSION}.tar.xz	2026-02-21 20:39:18 -05:00
Jeremie Fraeys	f9e80f8bdf	ci: remove all caching from CI workflow ... Cache infrastructure is not implemented in the Forgejo instance, causing timeouts Removed: - actions/cache steps for Go modules, Zig build, native libs, apt packages - Docker buildx layer caching (cache-from/cache-to)	2026-02-21 20:37:51 -05:00
Jeremie Fraeys	169dd98642	fix(ci): use GitHub releases for Zig download with better error handling ... - Switch from ziglang.org/download to github.com/ziglang/zig/releases - Add curl -fsSL --retry 3 for reliable downloads - Use file-based extraction instead of pipe-to-tar for verification	2026-02-21 18:18:54 -05:00
Jeremie Fraeys	ed7b5032a9	build: update Makefile and TUI controller integration	2026-02-21 18:00:09 -05:00
Jeremie Fraeys	be39b37aec	feat: native GPU detection and NVML bridge for macOS and Linux ... - Add dynamic NVML loading for Linux GPU detection - Add macOS GPU detection via IOKit framework - Add Zig NVML wrapper for cross-platform GPU queries - Update native bridge to support platform-specific GPU libs - Add CMake support for NVML dynamic library	2026-02-21 17:59:59 -05:00
Jeremie Fraeys	1a1844e9e9	fix(cli): remaining ArrayList API fixes in dataset and queue commands	2026-02-21 17:59:51 -05:00
Jeremie Fraeys	b1c9bc97fc	fix(cli): CLI structure, manifest, and asset fixes ... - Fix commands.zig imports (logs.zig → log.zig, remove missing modules) - Fix manifest.writeManifest to accept allocator param - Add db.Stmt type alias for sqlite3_stmt - Fix rsync placeholder to be valid shell script (#!/bin/sh)	2026-02-21 17:59:20 -05:00
Jeremie Fraeys	382c67edfc	fix(cli): WebSocket protocol and sync command fixes ... - Add sendSyncRun method for run synchronization - Add sendRerunRequest method for queue rerun - Add sync_run (0x26) and rerun_request (0x27) opcodes - Fix protocol import path to relative path - Fix db.Stmt type alias usage in sync.zig	2026-02-21 17:59:14 -05:00
Jeremie Fraeys	ccd1dd7a4d	fix(cli): Zig 0.15 core API changes ... - ArrayList: .init(allocator) → .empty, add allocator param to append/deinit/toOwnedSlice - Atomic: std.atomic.Atomic → std.atomic.Value, lowercase order names (.seq_cst) - Process: execvp instead of execvpe, inline wait status macros for macOS - Time: std.time.sleep → std.Thread.sleep - Error handling: fix isProcessRunning error union comparison	2026-02-21 17:59:05 -05:00
Jeremie Fraeys	20fde4f79d	feat: integrate NVML GPU monitoring into TUI ... - Update TUI controller loadGPU() to use NVML when available - Prioritize NVML over nvidia-smi command for better performance - Show additional metrics: power draw, SM clock when available - Maintain fallback to nvidia-smi and system_profiler	2026-02-21 15:17:22 -05:00
Jeremie Fraeys	c56e53cb52	fix: NVML stub support for systems without NVIDIA drivers ... - Add stub implementation in nvml_gpu.cpp when NVML not available - CMakeLists.txt checks for both NVML library and headers - Build succeeds on macOS/non-NVIDIA systems with stub - Runtime detection via gpu_is_available() prevents runtime errors	2026-02-21 15:16:54 -05:00
Jeremie Fraeys	05b7af6991	feat: implement NVML-based GPU monitoring ... - Add native/nvml_gpu/ C++ library wrapping NVIDIA Management Library - Add Go bindings in internal/worker/gpu_nvml_native.go and gpu_nvml_stub.go - Update gpu_detector.go to use NVML for accurate GPU count detection - Update native/CMakeLists.txt to build nvml_gpu library - Provides real-time GPU utilization, memory, temperature, clocks, power - Falls back to environment variable when NVML unavailable	2026-02-21 15:16:09 -05:00