Commit graph

56 commits

Author SHA1 Message Date
Jeremie Fraeys
72b4b29ecd
perf: add profiling benchmarks and parallel Go baseline for C++ optimization
Add comprehensive benchmarking suite for C++ optimization targets:
- tests/benchmarks/dataset_hash_bench_test.go - dirOverallSHA256Hex profiling
- tests/benchmarks/queue_bench_test.go - filesystem queue profiling
- tests/benchmarks/artifact_and_snapshot_bench_test.go - scanArtifacts/extractTarGz profiling
- tests/unit/worker/artifacts_test.go - moved from internal/ for clean separation

Add parallel Go implementation as baseline for C++ comparison:
- internal/worker/data_integrity.go: dirOverallSHA256HexParallel() with worker pool
- Benchmarks show 2.1x speedup (3.97ms -> 1.90ms) vs sequential

Exported wrappers for testing:
- ScanArtifacts() - artifact scanning
- ExtractTarGz() - tar.gz extraction
- DirOverallSHA256HexParallel() - parallel hashing

Profiling results (Apple M2 Ultra):
- dirOverallSHA256Hex: 78% syscall overhead (target for mmap C++)
- rebuildIndex: 96% syscall overhead (target for binary index C++)
- scanArtifacts: 87% syscall overhead (target for fast traversal C++)
- extractTarGz: 95% syscall overhead (target for parallel gzip C++)

Related: C++ optimization strategy in memory 5d5f0bb6
2026-02-12 12:04:02 -05:00
Jeremie Fraeys
eba4b4f766
testing checkout on docker
All checks were successful
Checkout test / test (push) Successful in 36s
2026-01-19 14:51:38 -05:00
Jeremie Fraeys
9832fc6f1d
testing checkout on docker
Some checks failed
Checkout test / test (push) Failing after 17s
2026-01-19 14:48:15 -05:00
Jeremie Fraeys
28c1c14d4a
testing checkout on docker
All checks were successful
Checkout test / test (push) Successful in 7s
2026-01-19 14:31:52 -05:00
Jeremie Fraeys
890c632fa7
testing checkout on docker
All checks were successful
Checkout test / test (push) Successful in 46s
2026-01-19 14:09:10 -05:00
Jeremie Fraeys
fc5bee5076
retry actions
Some checks are pending
Checkout test / test (push) Waiting to run
2026-01-19 14:04:19 -05:00
Jeremie Fraeys
9a990d2805
testing checkout
Some checks are pending
Checkout test / test (push) Waiting to run
2026-01-19 13:12:44 -05:00
Jeremie Fraeys
d6c8ca66d0 first commit
Some checks failed
Documentation / build (push) Has been cancelled
License Check / license-check (push) Has been cancelled
Documentation / deploy (push) Has been cancelled
2026-01-18 22:16:16 -05:00
Jeremie Fraeys
3fa4f6ae51 docs: update architecture/queue pages and remove legacy development setup 2026-01-05 13:26:39 -05:00
Jeremie Fraeys
f6e506a632 chore(docs): remove legacy Jekyll docs/_pages after Hugo migration 2026-01-05 12:41:09 -05:00
Jeremie Fraeys
3d58387207 chore(compose): remove deprecated root docker-compose files (moved to deployments/) 2026-01-05 12:41:04 -05:00
Jeremie Fraeys
1aed78839b docs(dev): document validate workflow, CLI/TUI UX contract, and consolidate dev/testing docs 2026-01-05 12:37:46 -05:00
Jeremie Fraeys
8157f73a70 docs(ops): consolidate deployment and performance monitoring docs for Caddy-based setup 2026-01-05 12:37:40 -05:00
Jeremie Fraeys
c0eeeda940 feat(experiment): improve experiment lifecycle and update first-experiment guide 2026-01-05 12:37:34 -05:00
Jeremie Fraeys
6b771e4a50 feat(jupyter): improve runtime management and update security/workflow docs 2026-01-05 12:37:27 -05:00
Jeremie Fraeys
94112f0af5 ci: align workflows, build scripts, and docs with current architecture 2026-01-05 12:34:23 -05:00
Jeremie Fraeys
dab680a60d feat(tracking): add pluggable tracking backends and audit support 2026-01-05 12:33:57 -05:00
Jeremie Fraeys
a8287f3087 test: expand unit/integration/e2e coverage for new worker/api behavior 2026-01-05 12:31:36 -05:00
Jeremie Fraeys
f726806770 chore(ops): reorganize deployments/monitoring and remove legacy scripts 2026-01-05 12:31:26 -05:00
Jeremie Fraeys
5ef24e4c6d feat(cli): add validate/info commands and improve protocol handling 2026-01-05 12:31:20 -05:00
Jeremie Fraeys
82034c68f3 feat(worker): add integrity checks, snapshot staging, and prewarm support 2026-01-05 12:31:13 -05:00
Jeremie Fraeys
add4a90e62 feat(api): refactor websocket handlers; add health and prometheus middleware 2026-01-05 12:31:07 -05:00
Jeremie Fraeys
6ff5324e74 refactor(storage,queue): split storage layer and add sqlite queue backend 2026-01-05 12:31:02 -05:00
Jeremie Fraeys
e901ddd810 chore(repo): add dockerignore, changelog, and ignore local artifacts 2026-01-05 12:30:57 -05:00
Jeremie Fraeys
17e5b9b71a docs: migrate to Hugo Book theme and publish via GitHub Pages 2026-01-05 12:22:24 -05:00
Jeremie Fraeys
cd5640ebd2 Slim and secure: move scripts, clean configs, remove secrets
- Move ci-test.sh and setup.sh to scripts/
- Trim docs/src/zig-cli.md to current structure
- Replace hardcoded secrets with placeholders in configs
- Update .gitignore to block .env*, secrets/, keys, build artifacts
- Slim README.md to reflect current CLI/TUI split
- Add cleanup trap to ci-test.sh
- Ensure no secrets are committed
2025-12-07 13:57:51 -05:00
Jeremie Fraeys
b75bd24bba Add CLI jupyter command for transparent Jupyter management
- Add jupyter.zig command with start/stop/status actions
- Update main.zig to include jupyter command in CLI
- CLI now handles all Jupyter setup transparently
- Data scientists only need: ml jupyter start
- Auto-detects container runtime (Podman/Docker)
- Manages container lifecycle automatically
- Provides clear status and error messages
2025-12-06 16:07:09 -05:00
Jeremie Fraeys
9dd4261873 Add Jupyter workflow documentation to docs/
- Create comprehensive Jupyter Workflow Integration guide
- Include ML tools usage examples (MLflow, Streamlit, Dash, Panel, Bokeh)
- Add CLI integration instructions and troubleshooting
- Update docs index to reference new workflow guide
- Provide complete setup and usage instructions
2025-12-06 16:02:49 -05:00
Jeremie Fraeys
7312451cfe Test and verify CLI-Jupyter workflow integration
- Successfully tested Jupyter notebook server with ML tools
- Verified all 6 ML tools working: MLflow 3.7.0, Streamlit 1.52.1, Dash 3.3.0, Panel 1.8.4, Bokeh 3.8.1
- MLflow experiment tracking working (created run ID: 25e6b467101845f1ab577c9cfe553c9c)
- CLI integration helper copied to workspace
- Fixed Jupyter permission issues with proper directory setup
2025-12-06 16:01:03 -05:00
Jeremie Fraeys
34c632dcde Create CLI-Jupyter integration workflow
- Add jupyter_launcher.sh script to start Jupyter with ML tools
- Create cli_integration.py helper for CLI operations
- Add sample notebook structure for experiments
- Create workflow documentation for seamless data science integration
2025-12-06 15:59:08 -05:00
Jeremie Fraeys
5cf16ac27d Clean up podman directory and verify Jupyter integration
- Remove redundant requirements file
- Test and verify Jupyter notebook 7.5.0 works
- ML tools container successfully built with all tools
- All 6 ML tools (MLflow, WandB, Streamlit, Dash, Panel, Bokeh) working
2025-12-06 15:56:01 -05:00
Jeremie Fraeys
3178cdf575 Enable ML tools integration for data scientists
- Add MLflow, WandB, Streamlit, Dash, Panel, Bokeh to environment.yml
- Update security policy to allow network access for ML tools
- Modify secure_runner.py to check tool permissions
- Add test script and usage guide
- Enable localhost network access for dashboard tools
2025-12-06 15:49:21 -05:00
Jeremie Fraeys
69dc9e6af4 Clean up README files and enhance testing documentation
- Remove 14 duplicate README files from test fixtures
- Clean up and restructure docs/src/testing.md with comprehensive testing guide
- Update main README.md to highlight testing and reference docs/ structure
- Remove empty .new files
- Keep only valuable, directory-specific README files (reduced from 34 to 20)
2025-12-06 15:43:51 -05:00
Jeremie Fraeys
605829dfc3 Update documentation for new features and fix outdated references
- Update deployment.md to reference new deployments/ directory structure
- Update CLI reference with new multi-user authentication system
- Add roles and permissions examples to configuration
- Fix docker-compose paths in testing documentation
- Remove references to non-existent docker-compose.test.yml
- Update troubleshooting with correct test commands
- Remove misplaced README files from test directories
2025-12-06 13:46:40 -05:00
Jeremie Fraeys
03cead6319 Organize docker-compose files and fix test output paths
- Move docker-compose.prod.yml and docker-compose.homelab-secure.yml to deployments/
- Create deployments/README.md with usage instructions
- Update test scripts to use new deployment paths
- Fix performance regression detection to output to tests/bin/
- All test outputs now properly organized in tests/bin/
2025-12-06 13:45:05 -05:00
Jeremie Fraeys
b90cb610c1 Clean up test artifacts from project root
- Move compiled test binaries to tests/binaries/
- Move profiling output files to tests/binaries/
- Add README.md for test binaries documentation
- Project root now clean of test artifacts
2025-12-06 13:41:12 -05:00
Jeremie Fraeys
83ba2f3415 Fix multi-user authentication and WebSocket issues
- Fix CLI WebSocket port (9101 vs 9103) in both status and authenticateUser
- Add researcher_user and analyst_user to server config with proper permissions
- Fix API key hashes for all users (complete 64-char SHA256)
- Enable IP whitelist with localhost and private network ranges
- Fix memory leaks in WebSocket handshake (proper key cleanup)
- Fix binary character display in server responses
- All authentication tests now pass: admin, researcher, analyst

Status: Multi-user authentication fully functional
2025-12-06 13:38:08 -05:00
Jeremie Fraeys
96dcfe6458 Add WebSocket handshake delay to fix timing issue
- Add 10ms delay after successful WebSocket handshake
- Use std.posix.nanosleep for proper timing
- This should prevent 'client sent data before handshake complete' error
- CLI WebSocket implementation improved but needs server testing

Status: WebSocket handshake timing improved, ready for testing
2025-12-06 13:24:20 -05:00
Jeremie Fraeys
4ca747bae3 Improve WebSocket handshake reading
- Fix WebSocket handshake response reading to wait for complete headers
- Increase buffer size for handshake response
- Add proper HTTP header completion detection
- Server still reports handshake issues but CLI connection improved

Status: Partial WebSocket implementation, needs further debugging
2025-12-06 13:19:53 -05:00
Jeremie Fraeys
7125dc3ab8 Partially fix API server and CLI connection
- Add Redis configuration to local config
- Fix API key format (api_keys vs apikeys)
- Update CLI to use port 9101
- Disable IP whitelist for testing
- Server now connects to Redis and authenticates
- WebSocket connection reaches server but handshake fails
- CLI needs WebSocket protocol implementation fix

Status: Server running, auth working, WebSocket handshake needs debugging
2025-12-06 13:19:07 -05:00
Jeremie Fraeys
1d55fefd52 Add comprehensive documentation and testing framework
- Add configuration reference with all options and examples
- Add quick start testing guide for 5-minute setup
- Document all environment and worker configurations
- Add CLI configuration examples for all user roles
- Include permission system documentation
- Add troubleshooting guides and validation commands
- Complete testing framework with multiple documentation levels

Documentation now includes:
- Quick start testing guide (5-minute setup)
- Comprehensive configuration reference
- Testing protocol and guide
- Troubleshooting and validation
- Permission system reference
- Environment variable documentation
2025-12-06 13:09:56 -05:00
Jeremie Fraeys
5a19358d00 Organize configs and scripts, create testing protocol
- Reorganize configs into environments/, workers/, deprecated/ folders
- Reorganize scripts into testing/, deployment/, maintenance/, benchmarks/ folders
- Add comprehensive testing guide documentation
- Add new Makefile targets: test-full, test-auth, test-status
- Update script paths in Makefile to match new organization
- Create testing protocol documentation
- Add cleanup status checking functionality

Testing framework now includes:
- Quick authentication tests (make test-auth)
- Full test suite runner (make test-full)
- Cleanup status monitoring (make test-status)
- Comprehensive documentation and troubleshooting guides
2025-12-06 13:08:15 -05:00
Jeremie Fraeys
c80e01b752 Add comprehensive self-cleaning system
- Add cleanup.sh script with dry-run, force, and all options
- Add auto-cleanup service setup for macOS (launchd) and Linux (systemd)
- Add cleanup-status.sh for monitoring Docker resources
- Add Makefile targets: self-cleanup, auto-cleanup
- Features colored output, confirmation prompts, and detailed logging
- Auto-cleanup runs daily to keep system clean
- Status monitoring shows resources and service state
2025-12-06 12:40:35 -05:00
Jeremie Fraeys
ea15af1833 Fix multi-user authentication and clean up debug code
- Fix YAML tags in auth config struct (json -> yaml)
- Update CLI configs to use pre-hashed API keys
- Remove double hashing in WebSocket client
- Fix port mapping (9102 -> 9103) in CLI commands
- Update permission keys to use jobs:read, jobs:create, etc.
- Clean up all debug logging from CLI and server
- All user roles now authenticate correctly:
  * Admin: Can queue jobs and see all jobs
  * Researcher: Can queue jobs and see own jobs
  * Analyst: Can see status (read-only access)

Multi-user authentication is now fully functional.
2025-12-06 12:35:32 -05:00
Jeremie Fraeys
10a3afaafb fix: update production environment variable check
- Change FETCH_ML_ENV check from 'production' to 'prod'
- Aligns with common environment naming conventions
- Fixes authentication validation for production deployment
2025-12-04 17:06:32 -05:00
Jeremie Fraeys
c5df0f40ff docs: add comprehensive development guide with flexible tooling
- Add detailed development setup instructions
- Include debugging, testing, and performance monitoring guides
- Provide IDE recommendations without enforcing specific configurations
- Make tooling optional and developer-friendly
- Add database management and configuration guidance

Creates welcoming development environment that respects developer preferences
while providing comprehensive guidance for contributing to FetchML platform.
2025-12-04 17:03:23 -05:00
Jeremie Fraeys
d5aae02ba0 chore: ignore SQLite temporary files and database files
- Add SQLite WAL and SHM file patterns to gitignore
- Include main database files to prevent accidental commits
- Ensures database state doesn't get committed to version control
2025-12-04 16:56:45 -05:00
Jeremie Fraeys
e5dcb347d8 feat: add GitHub workflows and development tooling
- Add comprehensive CI/CD workflows for testing and releases
- Include issue and pull request templates
- Add GitHub labeler configuration for automated triage
- Include license check and stale issue management
- Add Windsurf rules for development workflow
- Include database directory structure with gitkeep

Provides complete GitHub automation and development tooling
for streamlined contribution and project management.
2025-12-04 16:56:25 -05:00
Jeremie Fraeys
c980167041 test: implement comprehensive test suite with multiple test types
- Add end-to-end tests for complete workflow validation
- Include integration tests for API and database interactions
- Add unit tests for all major components and utilities
- Include performance tests for payload handling
- Add CLI API integration tests
- Include Podman container integration tests
- Add WebSocket and queue execution tests
- Include shell script tests for setup validation

Provides comprehensive test coverage ensuring platform reliability
and functionality across all components and interactions.
2025-12-04 16:55:13 -05:00
Jeremie Fraeys
bb25743b0f feat: add comprehensive setup scripts and management tools
- Add production setup scripts for automated deployment
- Include monitoring setup and configuration validation
- Add legacy setup scripts for various Linux distributions
- Implement Bitwarden integration for secure credential management
- Add development and production environment setup
- Include comprehensive management tools and utilities
- Add shell script library with common functions

Provides complete automation for setup, deployment, and management
of FetchML platform in development and production environments.
2025-12-04 16:55:04 -05:00