jfraeysd/fetch_ml
0
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
166 commits 1 branch 0 tags 38 MiB
Commit graph

6 commits

Author SHA1 Message Date
Jeremie Fraeys
3775bc3ee0
refactor: replace panic with error returns and update maintenance 
- Replace 9 panic() calls in smart_defaults.go with error returns
- Add ErrUnknownProfile error type for better error handling
- Update all callers (worker/config.go, tui/config.go, tui/cli_config.go, tui/main.go)
- Update CHANGELOG.md with recent WebSocket handler improvements
- Add metrics persistence, dataset handlers, and test organization notes
- Config validation passes (make configlint)
- All tests pass (go test ./tests/unit/api/ws)
 2026-02-18 14:44:21 -05:00
Jeremie Fraeys
320e6fd409
refactor(dependency-hygiene): Move path functions from config to storage 
Move ExpandPath function and path-related utilities from internal/config to internal/storage where they belong.

Files updated:
- internal/worker/config.go: use storage.ExpandPath
- internal/network/ssh.go: use storage.ExpandPath
- cmd/data_manager/data_manager_config.go: use storage.ExpandPath
- internal/api/server_config.go: use storage.ExpandPath

internal/storage/paths.go already contained the canonical implementation.

Result: Path utilities now live in storage layer, config package focuses on configuration structs.
 2026-02-17 21:15:23 -05:00
Jeremie Fraeys
a5c1a9fc0b
refactor: Phase 4 - split worker package into focused files 
Split 551-line worker/core.go into single-concern files:

- worker/config.go (+44 lines)
  - Added config parsing: envInt(), parseCPUFromConfig(), parseGPUCountFromConfig()
  - parseGPUSlotsPerGPUFromConfig()
  - Now has all config logic in one place (440 lines total)

- worker/metrics.go (new file, 172 lines)
  - Extracted setupMetricsExporter() with ~30 Prometheus metric registrations
  - Isolated metrics logic for easy modification

- worker/factory.go (new file, 183 lines)
  - Extracted NewWorker() factory function
  - Moved prePullImages(), pullImage() from core.go
  - Centralized worker instantiation

- worker/worker.go (renamed from core.go, ~100 lines)
  - Now just defines Worker struct, MLServer, JupyterManager
  - Clean, focused file without mixed concerns

Lines redistributed: ~350 lines moved from monolithic core.go
Build status: Compiles successfully
 2026-02-17 12:57:02 -05:00
Jeremie Fraeys
a93b6715fd
feat: add native library bridge and queue integration 
- Add native_queue.go with CGO bindings for queue operations
- Add native_queue_stub.go for non-CGO builds
- Add hash_selector to choose between Go and native implementations
- Add native_bridge_libs.go for CGO builds with native_libs tag
- Add native_bridge_nocgo.go stub for non-CGO builds
- Update queue errors and task handling for native integration
- Update worker config and runloop for native library support
 2026-02-16 20:38:30 -05:00
Jeremie Fraeys
2e701340e5
feat(core): API, worker, queue, and manifest improvements 
- Add protocol buffer optimizations (internal/api/protocol.go)
- Add filesystem queue backend (internal/queue/filesystem_queue.go)
- Add run manifest support (internal/manifest/run_manifest.go)
- Worker and jupyter task refinements
- Exported test wrappers for benchmarking
 2026-02-12 12:05:17 -05:00
Jeremie Fraeys
82034c68f3 feat(worker): add integrity checks, snapshot staging, and prewarm support 2026-01-05 12:31:13 -05:00