CI/CD Pipeline¶

Automated testing, building, and releasing for fetch_ml.

Workflows¶

CI Workflow (`.github/workflows/ci.yml`)¶

Runs on every push to main/develop and all pull requests.

Jobs: 1. test - Go backend tests with Redis 2. build - Build all binaries (Go + Zig CLI) 3. test-scripts - Validate deployment scripts 4. security-scan - Trivy and Gosec security scans 5. docker-build - Build and push Docker images (main branch only)

Test Coverage: - Go unit tests with race detection - internal/queue package tests - Zig CLI tests
- Integration tests - Security audits

Release Workflow (`.github/workflows/release.yml`)¶

Runs on version tags (e.g., v1.0.0).

Jobs:

build-cli (matrix build)
Linux x86_64 (static musl)
macOS x86_64
macOS ARM64
Downloads platform-specific static rsync
Embeds rsync for zero-dependency releases
build-go-backends
Cross-platform Go builds
api-server, worker, tui, data_manager, user_manager
create-release
Collects all artifacts
Generates SHA256 checksums
Creates GitHub release with notes

Release Process¶

Creating a Release¶

# 1. Update version
git tag v1.0.0

# 2. Push tag
git push origin v1.0.0

# 3. CI automatically builds and releases

Release Artifacts¶

CLI Binaries (with embedded rsync): - ml-linux-x86_64.tar.gz (~450-650KB) - ml-macos-x86_64.tar.gz (~450-650KB) - ml-macos-arm64.tar.gz (~450-650KB)

Go Backends: - fetch_ml_api-server.tar.gz - fetch_ml_worker.tar.gz - fetch_ml_tui.tar.gz - fetch_ml_data_manager.tar.gz - fetch_ml_user_manager.tar.gz

Checksums: - checksums.txt - Combined SHA256 sums - Individual .sha256 files per binary

Development Workflow¶

Local Testing¶

# Run all tests
make test

# Run specific package tests
go test ./internal/queue/...

# Build CLI
cd cli && zig build dev

# Run formatters and linters
make lint

# Security scans are handled automatically in CI by the `security-scan` job

Optional heavy end-to-end tests¶

Some e2e tests exercise full Docker deployments and performance scenarios and are skipped by default to keep local/CI runs fast. You can enable them explicitly with environment variables:

# Run Docker deployment e2e tests
FETCH_ML_E2E_DOCKER=1 go test ./tests/e2e/...

# Run performance-oriented e2e tests
FETCH_ML_E2E_PERF=1 go test ./tests/e2e/...

Without these variables, TestDockerDeploymentE2E and TestPerformanceE2E will t.Skip, while all lighter e2e tests still run.

Pull Request Checks¶

All PRs must pass: - ✅ Go tests (with Redis) - ✅ CLI tests - ✅ Security scans - ✅ Code linting - ✅ Build verification

Configuration¶

Environment Variables¶

GO_VERSION: '1.25.0'
ZIG_VERSION: '0.15.2'

Secrets¶

Required for releases: - GITHUB_TOKEN - Automatic, provided by GitHub Actions

Monitoring¶

Build Status¶

Check workflow runs at:

https://github.com/jfraeys/fetch_ml/actions

Artifacts¶

Download build artifacts from: - Successful workflow runs (30-day retention) - GitHub Releases (permanent)

For implementation details: - .github/workflows/ci.yml - .github/workflows/release.yml