jfraeysd/fetch_ml
0
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
150 commits 1 branch 0 tags 38 MiB
Commit graph

11 commits

Author SHA1 Message Date
Jeremie Fraeys
38c09c92bb
test(benchmarks): fix native lib benchmarks when disabled 
- Add skip checks to native queue benchmarks when FETCHML_NATIVE_LIBS=0
- Skip TestGoNativeArtifactScanLeak cleanly instead of 100 warnings
- Add build tags (!native_libs/native_libs) for Go vs Native comparison
- Add benchmark-native and benchmark-compare Makefile targets
 2026-02-18 12:45:30 -05:00
Jeremie Fraeys
3187ff26ea
refactor: complete maintainability phases 1-9 and fix all tests 
Test fixes (all 41 test packages now pass):
- Fix ComputeTaskProvenance - add dataset_specs JSON output
- Fix EnforceTaskProvenance - populate all metadata fields in best-effort mode
- Fix PrewarmNextOnce - preserve prewarm state when queue empty
- Fix RunManifest directory creation in SetupJobDirectories
- Add ManifestWriter to test worker (simpleManifestWriter)
- Fix worker ID mismatch (use cfg.WorkerID)
- Fix WebSocket binary protocol responses
- Implement all WebSocket handlers: QueueJob, QueueJobWithSnapshot, StatusRequest,
  CancelJob, Prune, ValidateRequest (with run manifest validation), LogMetric,
  GetExperiment, DatasetList/Register/Info/Search

Maintainability phases completed:
- Phases 1-6: Domain types, error system, config boundaries, worker/API/queue splits
- Phase 7: TUI cleanup - reorganize model package (jobs.go, messages.go, styles.go, keys.go)
- Phase 8: MLServer unification - consolidate worker + TUI into internal/network/mlserver.go
- Phase 9: CI enforcement - add scripts/ci-checks.sh with 5 checks:
  * No internal/ -> cmd/ imports
  * domain/ has zero internal imports
  * File size limit (500 lines, rigid)
  * No circular imports
  * Package naming conventions

Documentation:
- Add docs/src/file-naming-conventions.md
- Add make ci-checks target

Lines changed: +756/-36 (WebSocket fixes), +518/-320 (TUI), +263/-20 (Phase 8-9)
 2026-02-17 20:32:14 -05:00
Jeremie Fraeys
43d241c28d
feat: implement C++ native libraries for performance-critical operations 
- Add arena allocator for zero-allocation hot paths
- Add thread pool for parallel operations
- Add mmap utilities for memory-mapped I/O
- Implement queue_index with heap-based priority queue
- Implement dataset_hash with SIMD support (SHA-NI, ARMv8)
- Add runtime SIMD detection for cross-platform correctness
- Add comprehensive tests and benchmarks
 2026-02-16 20:38:04 -05:00
Jeremie Fraeys
d408a60eb1
ci: push all workflow updates
Some checks failed
Documentation / build-and-publish (push) Waiting to run
Details
Test / test (push) Waiting to run
Details
Checkout test / test (push) Successful in 5s
Details
CI with Native Libraries / test-native (push) Has been cancelled
Details
CI with Native Libraries / build-release (push) Has been cancelled
Details
2026-02-12 13:28:15 -05:00
Jeremie Fraeys
1dcc1e11d5
chore(build): update build system, scripts, and additional tests 
- Update Makefile with native build targets (preparing for C++)
- Add profiler and performance regression detector commands
- Update CI/testing scripts
- Add additional unit tests for API, jupyter, queue, manifest
 2026-02-12 12:05:55 -05:00
Jeremie Fraeys
94112f0af5 ci: align workflows, build scripts, and docs with current architecture 2026-01-05 12:34:23 -05:00
Jeremie Fraeys
03cead6319 Organize docker-compose files and fix test output paths 
- Move docker-compose.prod.yml and docker-compose.homelab-secure.yml to deployments/
- Create deployments/README.md with usage instructions
- Update test scripts to use new deployment paths
- Fix performance regression detection to output to tests/bin/
- All test outputs now properly organized in tests/bin/
 2025-12-06 13:45:05 -05:00
Jeremie Fraeys
5a19358d00 Organize configs and scripts, create testing protocol 
- Reorganize configs into environments/, workers/, deprecated/ folders
- Reorganize scripts into testing/, deployment/, maintenance/, benchmarks/ folders
- Add comprehensive testing guide documentation
- Add new Makefile targets: test-full, test-auth, test-status
- Update script paths in Makefile to match new organization
- Create testing protocol documentation
- Add cleanup status checking functionality

Testing framework now includes:
- Quick authentication tests (make test-auth)
- Full test suite runner (make test-full)
- Cleanup status monitoring (make test-status)
- Comprehensive documentation and troubleshooting guides
 2025-12-06 13:08:15 -05:00
Jeremie Fraeys
c80e01b752 Add comprehensive self-cleaning system 
- Add cleanup.sh script with dry-run, force, and all options
- Add auto-cleanup service setup for macOS (launchd) and Linux (systemd)
- Add cleanup-status.sh for monitoring Docker resources
- Add Makefile targets: self-cleanup, auto-cleanup
- Features colored output, confirmation prompts, and detailed logging
- Auto-cleanup runs daily to keep system clean
- Status monitoring shows resources and service state
 2025-12-06 12:40:35 -05:00
Jeremie Fraeys
ea15af1833 Fix multi-user authentication and clean up debug code 
- Fix YAML tags in auth config struct (json -> yaml)
- Update CLI configs to use pre-hashed API keys
- Remove double hashing in WebSocket client
- Fix port mapping (9102 -> 9103) in CLI commands
- Update permission keys to use jobs:read, jobs:create, etc.
- Clean up all debug logging from CLI and server
- All user roles now authenticate correctly:
  * Admin: Can queue jobs and see all jobs
  * Researcher: Can queue jobs and see own jobs
  * Analyst: Can see status (read-only access)

Multi-user authentication is now fully functional.
 2025-12-06 12:35:32 -05:00
Jeremie Fraeys
c5049a2fdf feat: initialize FetchML ML platform with core project structure 
- Add comprehensive README with architecture overview and quick start guide
- Set up Go module with production-ready dependencies
- Configure build system with Makefile for development and production builds
- Add Docker Compose for local development environment
- Include project configuration files (linting, Python, etc.)

This establishes the foundation for a production-ready ML experiment platform
with task queuing, monitoring, and modern CLI/API interface.
 2025-12-04 16:52:09 -05:00