fetch_ml

jfraeysd/fetch_ml

Fork 0

Commit graph

Author	SHA1	Message	Date
Jeremie Fraeys	a1ce267b86	feat: Implement all worker stub methods with real functionality - VerifySnapshot: SHA256 verification using integrity package - EnforceTaskProvenance: Strict and best-effort provenance validation - RunJupyterTask: Full Jupyter service lifecycle (start/stop/remove/restore/list_packages) - RunJob: Job execution using executor.JobRunner - PrewarmNextOnce: Prewarming with queue integration All methods now use new architecture components instead of placeholders	2026-02-17 17:37:56 -05:00
Jeremie Fraeys	38fa017b8e	refactor: Phase 6 - Complete migration, remove legacy files BREAKING CHANGE: Legacy worker files removed, Worker struct simplified Changes: 1. worker.go - Simplified to 8 fields using composed dependencies: - runLoop, runner, metrics, health (from new packages) - Removed: server, queue, running, datasetCache, ctx, cancel, etc. 2. factory.go - Updated NewWorker to use new structure - Uses lifecycle.NewRunLoop - Integrates jupyter.Manager properly 3. Removed legacy files: - execution.go (1,016 lines) - data_integrity.go (929 lines) - runloop.go (555 lines) - jupyter_task.go (144 lines) - simplified.go (demonstration no longer needed) 4. Fixed references to use new packages: - hash_selector.go -> integrity.DirOverallSHA256Hex - snapshot_store.go -> integrity.NormalizeSHA256ChecksumHex - metrics.go - Removed resource-dependent metrics temporarily 5. Added RecordQueueLatency to metrics.Metrics for lifecycle.MetricsRecorder Worker struct: 27 fields -> 8 fields (70% reduction) Build status: Compiles successfully	2026-02-17 14:39:48 -05:00
Jeremie Fraeys	a93b6715fd	feat: add native library bridge and queue integration - Add native_queue.go with CGO bindings for queue operations - Add native_queue_stub.go for non-CGO builds - Add hash_selector to choose between Go and native implementations - Add native_bridge_libs.go for CGO builds with native_libs tag - Add native_bridge_nocgo.go stub for non-CGO builds - Update queue errors and task handling for native integration - Update worker config and runloop for native library support	2026-02-16 20:38:30 -05:00

Author

SHA1

Message

Date

Jeremie Fraeys

a1ce267b86

feat: Implement all worker stub methods with real functionality

- VerifySnapshot: SHA256 verification using integrity package
- EnforceTaskProvenance: Strict and best-effort provenance validation
- RunJupyterTask: Full Jupyter service lifecycle (start/stop/remove/restore/list_packages)
- RunJob: Job execution using executor.JobRunner
- PrewarmNextOnce: Prewarming with queue integration

All methods now use new architecture components instead of placeholders

2026-02-17 17:37:56 -05:00

Jeremie Fraeys

38fa017b8e

refactor: Phase 6 - Complete migration, remove legacy files

BREAKING CHANGE: Legacy worker files removed, Worker struct simplified

Changes:
1. worker.go - Simplified to 8 fields using composed dependencies:
   - runLoop, runner, metrics, health (from new packages)
   - Removed: server, queue, running, datasetCache, ctx, cancel, etc.

2. factory.go - Updated NewWorker to use new structure
   - Uses lifecycle.NewRunLoop
   - Integrates jupyter.Manager properly

3. Removed legacy files:
   - execution.go (1,016 lines)
   - data_integrity.go (929 lines)
   - runloop.go (555 lines)
   - jupyter_task.go (144 lines)
   - simplified.go (demonstration no longer needed)

4. Fixed references to use new packages:
   - hash_selector.go -> integrity.DirOverallSHA256Hex
   - snapshot_store.go -> integrity.NormalizeSHA256ChecksumHex
   - metrics.go - Removed resource-dependent metrics temporarily

5. Added RecordQueueLatency to metrics.Metrics for lifecycle.MetricsRecorder

Worker struct: 27 fields -> 8 fields (70% reduction)

Build status: Compiles successfully

2026-02-17 14:39:48 -05:00

Jeremie Fraeys

a93b6715fd

feat: add native library bridge and queue integration

- Add native_queue.go with CGO bindings for queue operations
- Add native_queue_stub.go for non-CGO builds
- Add hash_selector to choose between Go and native implementations
- Add native_bridge_libs.go for CGO builds with native_libs tag
- Add native_bridge_nocgo.go stub for non-CGO builds
- Update queue errors and task handling for native integration
- Update worker config and runloop for native library support

2026-02-16 20:38:30 -05:00

3 commits