jfraeysd/fetch_ml
0
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
fetch_ml/scripts
History
Jeremie Fraeys e127f97442
chore: implement centralized path registry and file organization conventions 
Add PathRegistry for centralized path management:
- Create internal/config/paths.go with PathRegistry type
- Binary paths: BinDir(), APIServerBinary(), WorkerBinary(), etc.
- Data paths: DataDir(), JupyterStateDir(), ExperimentsDir()
- Config paths: ConfigDir(), APIServerConfig()
- Helper methods: EnsureDir(), EnsureDirSecure(), FileExists()
- Auto-detect repo root by looking for go.mod

Update .gitignore for root protection:
- Add explicit /api-server, /worker, /tui, /data_manager rules
- Add /coverage.out and .DS_Store to root protection
- Prevents accidental commits of binaries to root

Add path verification script:
- Create scripts/verify-paths.sh
- Checks for binaries in root directory
- Checks for .DS_Store files
- Checks for coverage.out in root
- Verifies data/ is gitignored
- Returns exit code 1 on violations

Cleaned .DS_Store files from repository
2026-02-18 16:48:50 -05:00
..
benchmarks chore(build): update build system, scripts, and additional tests 2026-02-12 12:05:55 -05:00
lib chore(build): update build system, scripts, and additional tests 2026-02-12 12:05:55 -05:00
maintenance chore(build): update build system, scripts, and additional tests 2026-02-12 12:05:55 -05:00
release fix: remove accidentally committed api-server binary 2026-02-18 16:31:40 -05:00
testing chore(build): update build system, scripts, and additional tests 2026-02-12 12:05:55 -05:00
ci-checks.sh chore: make file size check a warning instead of failure 2026-02-17 20:32:50 -05:00
ci-test.sh chore(build): update build system, scripts, and additional tests 2026-02-12 12:05:55 -05:00
detect_native.go test(integration): add websocket queue and hash benchmarks 2026-02-18 12:46:06 -05:00
manage-artifacts.sh docs: update README and CHANGELOG 2026-02-16 20:38:57 -05:00
README.md chore(build): update build system, scripts, and additional tests 2026-02-12 12:05:55 -05:00
setup_monitoring.py chore(ops): reorganize deployments/monitoring and remove legacy scripts 2026-01-05 12:31:26 -05:00
smoke-test-native.sh docs: add native libraries documentation and smoke tests 2026-02-16 20:38:46 -05:00
smoke-test.sh chore(build): update build system, scripts, and additional tests 2026-02-12 12:05:55 -05:00
track_performance.sh chore(build): update build system, scripts, and additional tests 2026-02-12 12:05:55 -05:00
verify-paths.sh chore: implement centralized path registry and file organization conventions 2026-02-18 16:48:50 -05:00
verify_release.sh chore(build): update build system, scripts, and additional tests 2026-02-12 12:05:55 -05:00

README.md

Scripts Directory

This directory contains setup and utility scripts for FetchML.

Production Scripts

setup-prod.sh

Purpose: Automated production setup for Rocky Linux bare metal deployment
Usage: sudo ./scripts/setup-prod.sh [base_path] [user] [group]
What it does:

  • Creates system user and groups
  • Sets up directory structure (/data/ml-experiments/*)
  • Installs dependencies (Go, Podman, Redis)
  • Configures GPU support for Podman
  • Creates systemd service files
  • Sets up log rotation

Example:

sudo ./scripts/setup-prod.sh /data/ml-experiments ml-user ml-group

Configuration validation

Validate configs using the built-in config lint targets:

make configlint
make worker-configlint

Cleanup Recommendation

These legacy scripts can be removed or archived. The current production setup only needs:

  • setup-prod.sh

Usage Workflow

First-Time Production Setup

# 1. Run production setup
sudo ./scripts/setup-prod.sh

# 2. Copy and configure
sudo cp configs/api/prod.yaml /etc/fetch_ml/config.yaml
sudo cp configs/workers/worker-prod.toml /etc/fetch_ml/worker.toml
sudo vim /etc/fetch_ml/config.yaml  # Update API keys, etc.

# 3. Build and install
make prod
sudo make install

# 4. Validate
./bin/configlint --schema configs/schema/api_server_config.yaml /etc/fetch_ml/config.yaml
./bin/configlint --schema configs/schema/worker_config_schema.yaml /etc/fetch_ml/worker.toml

# 5. Start services
sudo systemctl start fetchml-api fetchml-worker
sudo systemctl enable fetchml-api fetchml-worker

Development Setup (macOS)

# Use docker-compose for local development
docker-compose up -d

# Or run components directly
make dev
./bin/api-server -config configs/api/dev.yaml

Script Maintenance

When adding new scripts:

  1. Add executable permission: chmod +x scripts/new-script.sh
  2. Add header comment with purpose and usage
  3. Update this README
  4. Use consistent error handling and logging