Phase 2: Deterministic Manifests
- Add manifest.Validator with required field checking
- Support Validate() and ValidateStrict() modes
- Integrate validation into worker executor before execution
- Block execution if manifest missing commit_id or deps_manifest_sha256
Phase 5: Pinned Dependencies
- Add hermetic.dockerfile template with pinned system deps
- Frozen package versions: libblas3, libcudnn8, etc.
- Support for deps_manifest.json and requirements.txt with hashes
- Image tagging strategy: deps-<first-8-of-sha256>
Phase 8: Tests as Specifications
- Add queue_spec_test.go with executable scheduler specs
- Document priority ordering (higher first)
- Document FIFO tiebreaker for same priority
- Test cases for negative/zero priorities
Phase 10: Local Dev Parity
- Create root-level docker-compose.dev.yml
- Simplified from deployments/ for quick local dev
- Redis + API server + Worker with hot reload volumes
- Debug ports: 9101 (API), 6379 (Redis)
Implement TODO in handleCommandError:
- Log command name with error for crash report context
- Display 'Error in \'{s}': {s}' instead of generic 'Error: {s}'
- Helps users identify which command failed
Build verified: zig build --release=fast
Use utils/json.zig helpers to parse sync progress messages:
- Add json module import
- Rename json variable to json_mode to avoid shadowing
- Parse status, progress, total fields from JSON response
- Show percentage completion for in-progress syncs
- Handle 'complete' and 'error' status codes
Build verified: zig build --release=fast
Replace local file copy with embedded rsync binary for actual remote
synchronization to the server. The embedded rsync is extracted from
assets/ at runtime - no external dependencies required.
Changes:
- Re-add rsync_embedded.zig import
- Replace manual file copying with rsync.sync() call
- Construct remote path as api_key@host:worker_base/commit_id/files/
- Update JSON output to include commit_id
Build verified: zig build --release=fast
- Replace std.process.exit(1) with error.InvalidArgs in sync command
- Replace std.process.exit(1) with error.ValidationFailed in validate command
- Update validate.zig to use protocol.ResponsePacket.deinit() for cleanup
- Build verified: zig build --release=fast
Replace inline WebSocket URL construction with Config.getWebSocketUrl()
helper method in all command files. This eliminates code duplication
and ensures consistent URL formatting across the CLI.
Files updated:
- annotate.zig, dataset.zig, experiment.zig, logs.zig
- narrative.zig, prune.zig, queue.zig, requeue.zig
- sync.zig, validate.zig, watch.zig
The helper properly handles ws:// vs wss:// based on port (443).
Extract common UserContext and authentication logic from cancel.zig and
status.zig into new utils/auth.zig module. Add CommonFlags struct to
utils/flags.zig for shared CLI flags. Add getWebSocketUrl() helper to
Config to eliminate URL construction duplication.
Changes:
- Create cli/src/utils/auth.zig with UserContext and authenticateUser()
- Create cli/src/utils/flags.zig with CommonFlags struct
- Update cancel.zig and status.zig to use shared modules
- Add getWebSocketUrl() helper to config.zig
- Export new modules from utils.zig
Reduces code duplication and improves separation of concerns in the
Zig CLI codebase.
- Add skip checks to native queue benchmarks when FETCHML_NATIVE_LIBS=0
- Skip TestGoNativeArtifactScanLeak cleanly instead of 100 warnings
- Add build tags (!native_libs/native_libs) for Go vs Native comparison
- Add benchmark-native and benchmark-compare Makefile targets
Move ExpandPath function and path-related utilities from internal/config to internal/storage where they belong.
Files updated:
- internal/worker/config.go: use storage.ExpandPath
- internal/network/ssh.go: use storage.ExpandPath
- cmd/data_manager/data_manager_config.go: use storage.ExpandPath
- internal/api/server_config.go: use storage.ExpandPath
internal/storage/paths.go already contained the canonical implementation.
Result: Path utilities now live in storage layer, config package focuses on configuration structs.
- VerifySnapshot: SHA256 verification using integrity package
- EnforceTaskProvenance: Strict and best-effort provenance validation
- RunJupyterTask: Full Jupyter service lifecycle (start/stop/remove/restore/list_packages)
- RunJob: Job execution using executor.JobRunner
- PrewarmNextOnce: Prewarming with queue integration
All methods now use new architecture components instead of placeholders
- Changed package from worker to worker_test to match other test files
- Updated all type references to use worker.* prefix
- Fixed Worker field access to use exported fields (ID, Config, etc.)
Build status: Compiles successfully
- Renamed selectDependencyManifest to SelectDependencyManifest (exported)
- Added re-export in worker package for backward compatibility
- Updated internal call in container.go to use exported function
- API helpers can now access via worker.SelectDependencyManifest
Build status: Compiles successfully
- Create jobRunner using NewJobRunner with local and container executors
- Assign jobRunner to Worker.runner field
- JobRunner available for future task execution orchestration
Build status: Compiles successfully
- Re-enabled all resource metrics (CPU, GPU, acquisition stats)
- Metrics are conditionally registered only when w.resources != nil
- Added nil check to prevent panics if resource manager not initialized
Build status: Compiles successfully
Created simplified.go demonstrating target architecture:
internal/worker/simplified.go (109 lines)
- SimplifiedWorker struct with 6 fields vs original 27 fields
- Uses composed dependencies from previous phases:
- lifecycle.RunLoop for task lifecycle management
- executor.JobRunner for job execution
- lifecycle.HealthMonitor for health tracking
- lifecycle.MetricsRecorder for metrics
Key improvements demonstrated:
- Dependency injection via SimplifiedWorkerConfig
- Clear separation of concerns
- No direct resource access (queue, metrics, etc.)
- Each component implements a defined interface
- Easy to test with mock implementations
Note: This is a demonstration of the target architecture.
The original Worker struct remains for backward compatibility.
Migration would happen incrementally in future PRs.
Build status: Compiles successfully
Created lifecycle package with foundational types for future extraction:
1. internal/worker/lifecycle/runloop.go (117 lines)
- TaskExecutor interface for task execution contract
- RunLoopConfig for run loop configuration
- RunLoop type with core orchestration logic
- MetricsRecorder and Logger interfaces for dependencies
- Start(), Stop() methods for loop control
- executeTask() method for task lifecycle management
2. internal/worker/lifecycle/health.go (52 lines)
- HealthMonitor type for health tracking
- RecordHeartbeat(), IsHealthy(), MarkUnhealthy() methods
- Heartbeater interface for heartbeat operations
- HeartbeatLoop() function for background heartbeats
Note: These are interface/type foundations for Phase 5.
The actual Worker struct methods remain in runloop.go until
Phase 5 when they'll migrate to use these abstractions.
Build status: Compiles successfully
Created interfaces package to break tight coupling:
1. internal/worker/interfaces/executor.go (30 lines)
- JobExecutor interface for job execution
- ExecutionEnv struct for execution context
- ExecutionResult struct for results
2. internal/worker/interfaces/tracker.go (20 lines)
- ProgressTracker interface for execution stages
- StageStart, StageComplete, StageFailed methods
- JobComplete for final status
3. internal/worker/interfaces/manifest.go (18 lines)
- ManifestWriter interface for manifest operations
- Upsert method for update/create
- BuildInitial method for creating new manifests
These interfaces will enable:
- Dependency injection in future phases
- Mocking for unit tests
- Clean separation between orchestration and execution
Build status: Compiles successfully
