jfraeysd/fetch_ml
0
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
fetch_ml/internal/worker
History
Jeremie Fraeys 22f3d66f1d
refactor: Phase 2 - Extract executor implementations 
Created executor package with extracted job execution logic:

1. internal/worker/executor/local.go (104 lines)
   - LocalExecutor implements JobExecutor interface
   - Execute() method for local bash script execution
   - generateScript() helper for creating experiment scripts

2. internal/worker/executor/container.go (229 lines)
   - ContainerExecutor implements JobExecutor interface
   - Execute() method for podman container execution
   - EnvironmentPool interface for image caching
   - Tracking tool provisioning (MLflow, TensorBoard, Wandb)
   - Volume and cache setup
   - selectDependencyManifest() helper

3. internal/worker/executor/runner.go (131 lines)
   - JobRunner orchestrates execution
   - ExecutionMode enum (Auto, Local, Container)
   - Run() method with directory setup and executor selection
   - finalize() for success/failure handling

Key design decisions:
- Executors depend on interfaces (ManifestWriter, not Worker)
- JobRunner composes both executors
- No direct Worker dependencies in executor package
- SetupJobDirectories reused from execution package

Build status: Compiles successfully
2026-02-17 14:14:04 -05:00
..
execution refactor: Phase 4 deferred - Extract GPU utilities and execution helpers 2026-02-17 14:03:11 -05:00
executor refactor: Phase 2 - Extract executor implementations 2026-02-17 14:14:04 -05:00
interfaces refactor: Phase 1 - Extract worker interfaces 2026-02-17 14:10:03 -05:00
artifacts.go perf: add profiling benchmarks and parallel Go baseline for C++ optimization 2026-02-12 12:04:02 -05:00
config.go refactor: Phase 4 - split worker package into focused files 2026-02-17 12:57:02 -05:00
data_integrity.go ci: push all workflow updates 2026-02-12 13:28:15 -05:00
execution.go refactor: Phase 4 deferred - Extract GPU utilities and execution helpers 2026-02-17 14:03:11 -05:00
factory.go refactor: Phase 4 - split worker package into focused files 2026-02-17 12:57:02 -05:00
gpu.go refactor: Phase 4 deferred - Extract GPU utilities and execution helpers 2026-02-17 14:03:11 -05:00
gpu_detector.go feat(worker): add integrity checks, snapshot staging, and prewarm support 2026-01-05 12:31:13 -05:00
hash_selector.go feat: add native library bridge and queue integration 2026-02-16 20:38:30 -05:00
jupyter_task.go feat(core): API, worker, queue, and manifest improvements 2026-02-12 12:05:17 -05:00
metrics.go refactor: Phase 4 - split worker package into focused files 2026-02-17 12:57:02 -05:00
native_bridge.go feat: implement C++ native libraries for performance-critical operations 2026-02-16 20:38:04 -05:00
native_bridge_libs.go feat: add native library bridge and queue integration 2026-02-16 20:38:30 -05:00
native_bridge_nocgo.go feat: add native library bridge and queue integration 2026-02-16 20:38:30 -05:00
runloop.go feat: add native library bridge and queue integration 2026-02-16 20:38:30 -05:00
snapshot_store.go perf: add profiling benchmarks and parallel Go baseline for C++ optimization 2026-02-12 12:04:02 -05:00
worker.go refactor: Phase 4 - split worker package into focused files 2026-02-17 12:57:02 -05:00