chore(build): update build system, scripts, and additional tests

- Update Makefile with native build targets (preparing for C++)
- Add profiler and performance regression detector commands
- Update CI/testing scripts
- Add additional unit tests for API, jupyter, queue, manifest

2026-02-12 12:05:55 -05:00

2 KiB

Raw Blame History

Scripts Directory

This directory contains setup and utility scripts for FetchML.

Production Scripts

`setup-prod.sh`

Purpose: Automated production setup for Rocky Linux bare metal deployment
Usage: sudo ./scripts/setup-prod.sh [base_path] [user] [group]
What it does:

Creates system user and groups
Sets up directory structure (/data/ml-experiments/*)
Installs dependencies (Go, Podman, Redis)
Configures GPU support for Podman
Creates systemd service files
Sets up log rotation

Example:

sudo ./scripts/setup-prod.sh /data/ml-experiments ml-user ml-group

Configuration validation

Validate configs using the built-in config lint targets:

make configlint
make worker-configlint

Cleanup Recommendation

These legacy scripts can be removed or archived. The current production setup only needs:

setup-prod.sh

Usage Workflow

First-Time Production Setup

# 1. Run production setup
sudo ./scripts/setup-prod.sh

# 2. Copy and configure
sudo cp configs/api/prod.yaml /etc/fetch_ml/config.yaml
sudo cp configs/workers/worker-prod.toml /etc/fetch_ml/worker.toml
sudo vim /etc/fetch_ml/config.yaml  # Update API keys, etc.

# 3. Build and install
make prod
sudo make install

# 4. Validate
./bin/configlint --schema configs/schema/api_server_config.yaml /etc/fetch_ml/config.yaml
./bin/configlint --schema configs/schema/worker_config_schema.yaml /etc/fetch_ml/worker.toml

# 5. Start services
sudo systemctl start fetchml-api fetchml-worker
sudo systemctl enable fetchml-api fetchml-worker

Development Setup (macOS)

# Use docker-compose for local development
docker-compose up -d

# Or run components directly
make dev
./bin/api-server -config configs/api/dev.yaml

Script Maintenance

When adding new scripts:

Add executable permission: chmod +x scripts/new-script.sh
Add header comment with purpose and usage
Update this README
Use consistent error handling and logging

2 KiB Raw Blame History