jfraeysd/fetch_ml
0
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
100 commits 1 branch 0 tags 38 MiB
Commit graph

8 commits

Author SHA1 Message Date
Jeremie Fraeys
db7fbbd8d5
refactor: Phase 5 - split API package into focused files 
Reorganized internal/api/ package to follow single-concern principle:

- api/factory.go (new file, 257 lines)
  - Extracted component initialization from server.go
  - initializeComponents(), setupLogger(), initExperimentManager()
  - initTaskQueue(), initDatabase(), initDatabaseSchema()
  - initSecurity(), initJupyterServiceManager(), initAuditLogger()

- api/middleware.go (new file, 31 lines)
  - Extracted wrapWithMiddleware() - security middleware chain
  - Centralized auth, rate limiting, CORS, security headers

- api/server.go (reduced from 446 to 212 lines)
  - Now focused on Server lifecycle: NewServer, Start, WaitForShutdown, Close
  - Removed initialization logic (moved to factory.go)
  - Removed middleware wrapper (moved to middleware.go)

- api/metrics_middleware.go (existing, 64 lines)
  - Already had wrapWithMetrics(), left in place

Lines redistributed: ~180 lines from monolithic server.go
Build status: Compiles successfully
 2026-02-17 13:11:02 -05:00
Jeremie Fraeys
d1bef0a450
refactor: Phase 3 - fix config/storage boundaries 
Move schema ownership to infrastructure layer:

- Redis keys: config/constants.go -> queue/keys.go (TaskQueueKey, TaskPrefix, etc.)

- Filesystem paths: config/paths.go -> storage/paths.go (JobPaths)

- Create config/shared.go with RedisConfig, SSHConfig

- Update all imports: worker/, api/helpers, api/ws_jobs, api/ws_validate

- Clean up: remove duplicates from queue/task.go, queue/queue.go, config/paths.go

Build status: Compiles successfully
 2026-02-17 12:49:53 -05:00
Jeremie Fraeys
b05470b30a
refactor: improve API structure and WebSocket protocol 
- Extract WebSocket protocol handling to dedicated module
- Add helper functions for DB operations, validation, and responses
- Improve WebSocket frame handling and opcodes
- Refactor dataset, job, and Jupyter handlers
- Add duplicate detection processing
 2026-02-16 20:38:12 -05:00
Jeremie Fraeys
2e701340e5
feat(core): API, worker, queue, and manifest improvements 
- Add protocol buffer optimizations (internal/api/protocol.go)
- Add filesystem queue backend (internal/queue/filesystem_queue.go)
- Add run manifest support (internal/manifest/run_manifest.go)
- Worker and jupyter task refinements
- Exported test wrappers for benchmarking
 2026-02-12 12:05:17 -05:00
Jeremie Fraeys
add4a90e62 feat(api): refactor websocket handlers; add health and prometheus middleware 2026-01-05 12:31:07 -05:00
Jeremie Fraeys
cd5640ebd2 Slim and secure: move scripts, clean configs, remove secrets 
- Move ci-test.sh and setup.sh to scripts/
- Trim docs/src/zig-cli.md to current structure
- Replace hardcoded secrets with placeholders in configs
- Update .gitignore to block .env*, secrets/, keys, build artifacts
- Slim README.md to reflect current CLI/TUI split
- Add cleanup trap to ci-test.sh
- Ensure no secrets are committed
 2025-12-07 13:57:51 -05:00
Jeremie Fraeys
ea15af1833 Fix multi-user authentication and clean up debug code 
- Fix YAML tags in auth config struct (json -> yaml)
- Update CLI configs to use pre-hashed API keys
- Remove double hashing in WebSocket client
- Fix port mapping (9102 -> 9103) in CLI commands
- Update permission keys to use jobs:read, jobs:create, etc.
- Clean up all debug logging from CLI and server
- All user roles now authenticate correctly:
  * Admin: Can queue jobs and see all jobs
  * Researcher: Can queue jobs and see own jobs
  * Analyst: Can see status (read-only access)

Multi-user authentication is now fully functional.
 2025-12-06 12:35:32 -05:00
Jeremie Fraeys
803677be57 feat: implement Go backend with comprehensive API and internal packages 
- Add API server with WebSocket support and REST endpoints
- Implement authentication system with API keys and permissions
- Add task queue system with Redis backend and error handling
- Include storage layer with database migrations and schemas
- Add comprehensive logging, metrics, and telemetry
- Implement security middleware and network utilities
- Add experiment management and container orchestration
- Include configuration management with smart defaults
 2025-12-04 16:53:53 -05:00