jfraeysd/fetch_ml
0
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
91 commits 1 branch 0 tags 38 MiB
Commit graph

12 commits

Author SHA1 Message Date
Jeremie Fraeys
7305e2bc21
test: add comprehensive test coverage and command improvements 
- Add logs and debug end-to-end tests
- Add test helper utilities
- Improve test fixtures and templates
- Update API server and config lint commands
- Add multi-user database initialization
 2026-02-16 20:38:15 -05:00
Jeremie Fraeys
d408a60eb1
ci: push all workflow updates
Some checks failed
Documentation / build-and-publish (push) Waiting to run
Details
Test / test (push) Waiting to run
Details
Checkout test / test (push) Successful in 5s
Details
CI with Native Libraries / test-native (push) Has been cancelled
Details
CI with Native Libraries / build-release (push) Has been cancelled
Details
2026-02-12 13:28:15 -05:00
Jeremie Fraeys
2854d3df95
chore(cleanup): remove legacy artifacts and add tooling configs
Some checks failed
Documentation / build-and-publish (push) Has been cancelled
Details
Checkout test / test (push) Has been cancelled
Details 
- Remove .github/ directory (migrated to .forgejo/)
- Remove .local-artifacts/ benchmark results
- Add AGENTS.md for coding assistants
- Add .windsurf/rules/ for development guidelines
- Update .gitignore
 2026-02-12 12:06:09 -05:00
Jeremie Fraeys
1dcc1e11d5
chore(build): update build system, scripts, and additional tests 
- Update Makefile with native build targets (preparing for C++)
- Add profiler and performance regression detector commands
- Update CI/testing scripts
- Add additional unit tests for API, jupyter, queue, manifest
 2026-02-12 12:05:55 -05:00
Jeremie Fraeys
72b4b29ecd
perf: add profiling benchmarks and parallel Go baseline for C++ optimization 
Add comprehensive benchmarking suite for C++ optimization targets:
- tests/benchmarks/dataset_hash_bench_test.go - dirOverallSHA256Hex profiling
- tests/benchmarks/queue_bench_test.go - filesystem queue profiling
- tests/benchmarks/artifact_and_snapshot_bench_test.go - scanArtifacts/extractTarGz profiling
- tests/unit/worker/artifacts_test.go - moved from internal/ for clean separation

Add parallel Go implementation as baseline for C++ comparison:
- internal/worker/data_integrity.go: dirOverallSHA256HexParallel() with worker pool
- Benchmarks show 2.1x speedup (3.97ms -> 1.90ms) vs sequential

Exported wrappers for testing:
- ScanArtifacts() - artifact scanning
- ExtractTarGz() - tar.gz extraction
- DirOverallSHA256HexParallel() - parallel hashing

Profiling results (Apple M2 Ultra):
- dirOverallSHA256Hex: 78% syscall overhead (target for mmap C++)
- rebuildIndex: 96% syscall overhead (target for binary index C++)
- scanArtifacts: 87% syscall overhead (target for fast traversal C++)
- extractTarGz: 95% syscall overhead (target for parallel gzip C++)

Related: C++ optimization strategy in memory 5d5f0bb6
 2026-02-12 12:04:02 -05:00
Jeremie Fraeys
a8287f3087 test: expand unit/integration/e2e coverage for new worker/api behavior 2026-01-05 12:31:36 -05:00
Jeremie Fraeys
cd5640ebd2 Slim and secure: move scripts, clean configs, remove secrets 
- Move ci-test.sh and setup.sh to scripts/
- Trim docs/src/zig-cli.md to current structure
- Replace hardcoded secrets with placeholders in configs
- Update .gitignore to block .env*, secrets/, keys, build artifacts
- Slim README.md to reflect current CLI/TUI split
- Add cleanup trap to ci-test.sh
- Ensure no secrets are committed
 2025-12-07 13:57:51 -05:00
Jeremie Fraeys
69dc9e6af4 Clean up README files and enhance testing documentation 
- Remove 14 duplicate README files from test fixtures
- Clean up and restructure docs/src/testing.md with comprehensive testing guide
- Update main README.md to highlight testing and reference docs/ structure
- Remove empty .new files
- Keep only valuable, directory-specific README files (reduced from 34 to 20)
 2025-12-06 15:43:51 -05:00
Jeremie Fraeys
03cead6319 Organize docker-compose files and fix test output paths 
- Move docker-compose.prod.yml and docker-compose.homelab-secure.yml to deployments/
- Create deployments/README.md with usage instructions
- Update test scripts to use new deployment paths
- Fix performance regression detection to output to tests/bin/
- All test outputs now properly organized in tests/bin/
 2025-12-06 13:45:05 -05:00
Jeremie Fraeys
b90cb610c1 Clean up test artifacts from project root 
- Move compiled test binaries to tests/binaries/
- Move profiling output files to tests/binaries/
- Add README.md for test binaries documentation
- Project root now clean of test artifacts
 2025-12-06 13:41:12 -05:00
Jeremie Fraeys
ea15af1833 Fix multi-user authentication and clean up debug code 
- Fix YAML tags in auth config struct (json -> yaml)
- Update CLI configs to use pre-hashed API keys
- Remove double hashing in WebSocket client
- Fix port mapping (9102 -> 9103) in CLI commands
- Update permission keys to use jobs:read, jobs:create, etc.
- Clean up all debug logging from CLI and server
- All user roles now authenticate correctly:
  * Admin: Can queue jobs and see all jobs
  * Researcher: Can queue jobs and see own jobs
  * Analyst: Can see status (read-only access)

Multi-user authentication is now fully functional.
 2025-12-06 12:35:32 -05:00
Jeremie Fraeys
c980167041 test: implement comprehensive test suite with multiple test types 
- Add end-to-end tests for complete workflow validation
- Include integration tests for API and database interactions
- Add unit tests for all major components and utilities
- Include performance tests for payload handling
- Add CLI API integration tests
- Include Podman container integration tests
- Add WebSocket and queue execution tests
- Include shell script tests for setup validation

Provides comprehensive test coverage ensuring platform reliability
and functionality across all components and interactions.
 2025-12-04 16:55:13 -05:00