fetch_ml

Author	SHA1	Message	Date
Jeremie Fraeys	3b194ff2e8	feat: GPU detection transparency and artifact scanner improvements Some checks failed Build CLI with Embedded SQLite / build (arm64, aarch64-linux) (push) Waiting to run Details Build CLI with Embedded SQLite / build (x86_64, x86_64-linux) (push) Waiting to run Details Build CLI with Embedded SQLite / build-macos (arm64) (push) Waiting to run Details Build CLI with Embedded SQLite / build-macos (x86_64) (push) Waiting to run Details Security Scan / Security Analysis (push) Waiting to run Details Security Scan / Native Library Security (push) Waiting to run Details Checkout test / test (push) Successful in 6s Details CI/CD Pipeline / Test (push) Failing after 1s Details CI/CD Pipeline / Dev Compose Smoke Test (push) Has been skipped Details CI/CD Pipeline / Build (push) Has been skipped Details CI/CD Pipeline / Test Scripts (push) Has been skipped Details CI/CD Pipeline / Test Native Libraries (push) Has been skipped Details CI/CD Pipeline / GPU Golden Test Matrix (push) Has been skipped Details Documentation / build-and-publish (push) Failing after 39s Details CI/CD Pipeline / Docker Build (push) Has been skipped Details - Surface GPUDetectionInfo from parseGPUCountFromConfig for detection metadata - Document FETCH_ML_TOTAL_CPU and FETCH_ML_GPU_SLOTS_PER_GPU env vars - Add debug logging for all env var overrides to stderr - Track config-layer auto-detection in GPUDetectionInfo.ConfigLayerAutoDetected - Add --include-all flag to artifact scanner (includeAll parameter) - Add AMD production mode enforcement (error in non-local mode) - Add GPU detector unit tests for env overrides and AMD aliasing	2026-02-23 12:29:34 -05:00
Jeremie Fraeys	23e5f3d1dc	refactor(api): internal refactoring for TUI and worker modules - Refactor internal/worker and internal/queue packages - Update cmd/tui for monitoring interface - Update test configurations	2026-02-20 15:51:23 -05:00
Jeremie Fraeys	a1ce267b86	feat: Implement all worker stub methods with real functionality - VerifySnapshot: SHA256 verification using integrity package - EnforceTaskProvenance: Strict and best-effort provenance validation - RunJupyterTask: Full Jupyter service lifecycle (start/stop/remove/restore/list_packages) - RunJob: Job execution using executor.JobRunner - PrewarmNextOnce: Prewarming with queue integration All methods now use new architecture components instead of placeholders	2026-02-17 17:37:56 -05:00
Jeremie Fraeys	72b4b29ecd	perf: add profiling benchmarks and parallel Go baseline for C++ optimization Add comprehensive benchmarking suite for C++ optimization targets: - tests/benchmarks/dataset_hash_bench_test.go - dirOverallSHA256Hex profiling - tests/benchmarks/queue_bench_test.go - filesystem queue profiling - tests/benchmarks/artifact_and_snapshot_bench_test.go - scanArtifacts/extractTarGz profiling - tests/unit/worker/artifacts_test.go - moved from internal/ for clean separation Add parallel Go implementation as baseline for C++ comparison: - internal/worker/data_integrity.go: dirOverallSHA256HexParallel() with worker pool - Benchmarks show 2.1x speedup (3.97ms -> 1.90ms) vs sequential Exported wrappers for testing: - ScanArtifacts() - artifact scanning - ExtractTarGz() - tar.gz extraction - DirOverallSHA256HexParallel() - parallel hashing Profiling results (Apple M2 Ultra): - dirOverallSHA256Hex: 78% syscall overhead (target for mmap C++) - rebuildIndex: 96% syscall overhead (target for binary index C++) - scanArtifacts: 87% syscall overhead (target for fast traversal C++) - extractTarGz: 95% syscall overhead (target for parallel gzip C++) Related: C++ optimization strategy in memory 5d5f0bb6	2026-02-12 12:04:02 -05:00

4 commits