jfraeysd/fetch_ml
0
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
429 commits 1 branch 0 tags 38 MiB
Commit graph

6 commits

Author SHA1 Message Date
Jeremie Fraeys
2b1ef10514
test(chaos): add worker disconnect chaos test and queue improvements 
Chaos testing:
- Add worker_disconnect_chaos_test.go for network partition resilience
- Test scheduler hub recovery and job reassignment scenarios

Queue layer updates:
- event_store.go: add event sourcing for queue operations
- native_queue.go: extend native queue with batch operations and indexing
 2026-03-12 12:08:21 -04:00
Jeremie Fraeys
cb142213fa
chore(build): update build system, Dockerfiles, and dependencies 
Build and deployment improvements:

Makefile:
- Native library build targets with ASan support
- Cross-platform compilation helpers
- Performance benchmark targets
- Security scan integration

Docker:
- secure-prod.Dockerfile: Hardened production image (non-root, minimal surface)
- simple.Dockerfile: Lightweight development image

Scripts:
- build/: Go and native library build scripts, cross-platform builds
- ci/: checks.sh, test.sh, verify-paths.sh for validation
- benchmarks/: Local performance testing and regression tracking
- dev/: Monitoring setup

Dependencies: Update to latest stable with security patches

Commands:
- api-server/main.go: Server initialization updates
- data_manager/data_sync.go: Data sync with visibility
- errors/main.go: Error handling improvements
- tui/: TUI improvements for group management
 2026-03-08 13:03:48 -04:00
Jeremie Fraeys
dbf96020af
refactor(dependency-hygiene): Fix Redis leak, simplify TUI wrapper, clean go.mod 
Phase 1: Fix Redis Schema Leak
- Create internal/storage/dataset.go with DatasetStore abstraction
- Remove all direct Redis calls from cmd/data_manager/data_sync.go
- data_manager now uses DatasetStore for transfer tracking and metadata

Phase 2: Simplify TUI Services
- Embed *queue.TaskQueue directly in services.TaskQueue
- Eliminate 60% of wrapper boilerplate (203 -> ~100 lines)
- Keep only TUI-specific methods (EnqueueTask, GetJobStatus, experiment methods)

Phase 5: Clean go.mod Dependencies
- Remove duplicate go-redis/redis/v8 dependency
- Migrate internal/storage/migrate.go to redis/go-redis/v9
- Separate test-only deps (miniredis, testify) into own block

Results:
- Zero direct Redis calls in cmd/
- 60% fewer lines in TUI services
- Cleaner dependency structure
 2026-02-17 21:13:49 -05:00
Jeremie Fraeys
94112f0af5 ci: align workflows, build scripts, and docs with current architecture 2026-01-05 12:34:23 -05:00
Jeremie Fraeys
ea15af1833 Fix multi-user authentication and clean up debug code 
- Fix YAML tags in auth config struct (json -> yaml)
- Update CLI configs to use pre-hashed API keys
- Remove double hashing in WebSocket client
- Fix port mapping (9102 -> 9103) in CLI commands
- Update permission keys to use jobs:read, jobs:create, etc.
- Clean up all debug logging from CLI and server
- All user roles now authenticate correctly:
  * Admin: Can queue jobs and see all jobs
  * Researcher: Can queue jobs and see own jobs
  * Analyst: Can see status (read-only access)

Multi-user authentication is now fully functional.
 2025-12-06 12:35:32 -05:00
Jeremie Fraeys
803677be57 feat: implement Go backend with comprehensive API and internal packages 
- Add API server with WebSocket support and REST endpoints
- Implement authentication system with API keys and permissions
- Add task queue system with Redis backend and error handling
- Include storage layer with database migrations and schemas
- Add comprehensive logging, metrics, and telemetry
- Implement security middleware and network utilities
- Add experiment management and container orchestration
- Include configuration management with smart defaults
 2025-12-04 16:53:53 -05:00