3187ff26ea
Test fixes (all 41 test packages now pass): - Fix ComputeTaskProvenance - add dataset_specs JSON output - Fix EnforceTaskProvenance - populate all metadata fields in best-effort mode - Fix PrewarmNextOnce - preserve prewarm state when queue empty - Fix RunManifest directory creation in SetupJobDirectories - Add ManifestWriter to test worker (simpleManifestWriter) - Fix worker ID mismatch (use cfg.WorkerID) - Fix WebSocket binary protocol responses - Implement all WebSocket handlers: QueueJob, QueueJobWithSnapshot, StatusRequest, CancelJob, Prune, ValidateRequest (with run manifest validation), LogMetric, GetExperiment, DatasetList/Register/Info/Search Maintainability phases completed: - Phases 1-6: Domain types, error system, config boundaries, worker/API/queue splits - Phase 7: TUI cleanup - reorganize model package (jobs.go, messages.go, styles.go, keys.go) - Phase 8: MLServer unification - consolidate worker + TUI into internal/network/mlserver.go - Phase 9: CI enforcement - add scripts/ci-checks.sh with 5 checks: * No internal/ -> cmd/ imports * domain/ has zero internal imports * File size limit (500 lines, rigid) * No circular imports * Package naming conventions Documentation: - Add docs/src/file-naming-conventions.md - Add make ci-checks target Lines changed: +756/-36 (WebSocket fixes), +518/-320 (TUI), +263/-20 (Phase 8-9) |
||
|---|---|---|
| .. | ||
| benchmarks | ||
| lib | ||
| maintenance | ||
| testing | ||
| ci-checks.sh | ||
| ci-test.sh | ||
| manage-artifacts.sh | ||
| README.md | ||
| setup_monitoring.py | ||
| smoke-test-native.sh | ||
| smoke-test.sh | ||
| track_performance.sh | ||
| verify_release.sh | ||
Scripts Directory
This directory contains setup and utility scripts for FetchML.
Production Scripts
setup-prod.sh
Purpose: Automated production setup for Rocky Linux bare metal deployment
Usage: sudo ./scripts/setup-prod.sh [base_path] [user] [group]
What it does:
- Creates system user and groups
- Sets up directory structure (
/data/ml-experiments/*) - Installs dependencies (Go, Podman, Redis)
- Configures GPU support for Podman
- Creates systemd service files
- Sets up log rotation
Example:
sudo ./scripts/setup-prod.sh /data/ml-experiments ml-user ml-group
Configuration validation
Validate configs using the built-in config lint targets:
make configlint
make worker-configlint
Cleanup Recommendation
These legacy scripts can be removed or archived. The current production setup only needs:
setup-prod.sh
Usage Workflow
First-Time Production Setup
# 1. Run production setup
sudo ./scripts/setup-prod.sh
# 2. Copy and configure
sudo cp configs/api/prod.yaml /etc/fetch_ml/config.yaml
sudo cp configs/workers/worker-prod.toml /etc/fetch_ml/worker.toml
sudo vim /etc/fetch_ml/config.yaml # Update API keys, etc.
# 3. Build and install
make prod
sudo make install
# 4. Validate
./bin/configlint --schema configs/schema/api_server_config.yaml /etc/fetch_ml/config.yaml
./bin/configlint --schema configs/schema/worker_config_schema.yaml /etc/fetch_ml/worker.toml
# 5. Start services
sudo systemctl start fetchml-api fetchml-worker
sudo systemctl enable fetchml-api fetchml-worker
Development Setup (macOS)
# Use docker-compose for local development
docker-compose up -d
# Or run components directly
make dev
./bin/api-server -config configs/api/dev.yaml
Script Maintenance
When adding new scripts:
- Add executable permission:
chmod +x scripts/new-script.sh - Add header comment with purpose and usage
- Update this README
- Use consistent error handling and logging