jfraeysd/fetch_ml
0
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
fetch_ml/scripts
History
Jeremie Fraeys ea15af1833 Fix multi-user authentication and clean up debug code 
- Fix YAML tags in auth config struct (json -> yaml)
- Update CLI configs to use pre-hashed API keys
- Remove double hashing in WebSocket client
- Fix port mapping (9102 -> 9103) in CLI commands
- Update permission keys to use jobs:read, jobs:create, etc.
- Clean up all debug logging from CLI and server
- All user roles now authenticate correctly:
  * Admin: Can queue jobs and see all jobs
  * Researcher: Can queue jobs and see own jobs
  * Analyst: Can see status (read-only access)

Multi-user authentication is now fully functional.
2025-12-06 12:35:32 -05:00
..
legacy feat: add comprehensive setup scripts and management tools 2025-12-04 16:55:04 -05:00
lib feat: add comprehensive setup scripts and management tools 2025-12-04 16:55:04 -05:00
cleanup-benchmarks.sh Fix multi-user authentication and clean up debug code 2025-12-06 12:35:32 -05:00
create_bitwarden_fetchml_item.sh feat: add comprehensive setup scripts and management tools 2025-12-04 16:55:04 -05:00
manage-artifacts.sh Fix multi-user authentication and clean up debug code 2025-12-06 12:35:32 -05:00
README.md feat: add comprehensive setup scripts and management tools 2025-12-04 16:55:04 -05:00
run-benchmarks-local.sh Fix multi-user authentication and clean up debug code 2025-12-06 12:35:32 -05:00
setup-monitoring-prod.sh feat: add comprehensive setup scripts and management tools 2025-12-04 16:55:04 -05:00
setup-prod.sh feat: add comprehensive setup scripts and management tools 2025-12-04 16:55:04 -05:00
setup-production.sh feat: add comprehensive setup scripts and management tools 2025-12-04 16:55:04 -05:00
test-homelab-secure.sh Fix multi-user authentication and clean up debug code 2025-12-06 12:35:32 -05:00
test-prod.sh Fix multi-user authentication and clean up debug code 2025-12-06 12:35:32 -05:00
validate-prod-config.sh feat: add comprehensive setup scripts and management tools 2025-12-04 16:55:04 -05:00

README.md

Scripts Directory

This directory contains setup and utility scripts for FetchML.

Production Scripts

setup-prod.sh

Purpose: Automated production setup for Rocky Linux bare metal deployment
Usage: sudo ./scripts/setup-prod.sh [base_path] [user] [group]
What it does:

  • Creates system user and groups
  • Sets up directory structure (/data/ml-experiments/*)
  • Installs dependencies (Go, Podman, Redis)
  • Configures GPU support for Podman
  • Creates systemd service files
  • Sets up log rotation

Example:

sudo ./scripts/setup-prod.sh /data/ml-experiments ml-user ml-group

validate-prod-config.sh

Purpose: Validates production configuration files
Usage: ./scripts/validate-prod-config.sh [api-config] [worker-config]
What it does:

  • Checks config file syntax
  • Verifies base_path consistency
  • Tests Redis connectivity
  • Validates Podman setup
  • Checks directory permissions

Example:

./scripts/validate-prod-config.sh configs/config-prod.yaml configs/worker-prod.toml

Legacy Setup Scripts (Deprecated)

The following scripts are from earlier iterations and are deprecated in favor of setup-prod.sh:

  • setup_rocky.sh - Use setup-prod.sh instead
  • setup_ubuntu.sh - Ubuntu support (not primary target)
  • auto_setup.sh - Old automated setup (superseded)
  • setup_common.sh - Common functions (integrated into setup-prod.sh)
  • quick_start.sh - Quick dev setup (use docker-compose on macOS instead)
  • test_tools.sh - Tool testing (integrated into validate-prod-config.sh)

Cleanup Recommendation

These legacy scripts can be removed or archived. The current production setup only needs:

  • setup-prod.sh
  • validate-prod-config.sh

Usage Workflow

First-Time Production Setup

# 1. Run production setup
sudo ./scripts/setup-prod.sh

# 2. Copy and configure
sudo cp configs/config-prod.yaml /etc/fetch_ml/config.yaml
sudo cp configs/worker-prod.toml /etc/fetch_ml/worker.toml
sudo vim /etc/fetch_ml/config.yaml  # Update API keys, etc.

# 3. Build and install
make prod
sudo make install

# 4. Validate
./scripts/validate-prod-config.sh /etc/fetch_ml/config.yaml /etc/fetch_ml/worker.toml

# 5. Start services
sudo systemctl start fetchml-api fetchml-worker
sudo systemctl enable fetchml-api fetchml-worker

Development Setup (macOS)

# Use docker-compose for local development
docker-compose up -d

# Or run components directly
make dev
./bin/api-server -config configs/config-local.yaml

Script Maintenance

When adding new scripts:

  1. Add executable permission: chmod +x scripts/new-script.sh
  2. Add header comment with purpose and usage
  3. Update this README
  4. Use consistent error handling and logging