fetch_ml/docs/src/cicd.md

title	url	weight
CI/CD Pipeline	/cicd/	5

CI/CD Pipeline

Automated testing, building, and releasing for fetch_ml.

Workflows

CI Workflow (.forgejo/workflows/ci.yml)

Runs on every push to main/develop and all pull requests.

Jobs:

1. test - Go backend tests with Redis
2. build - Build all binaries (Go + Zig CLI)
3. test-scripts - Validate deployment scripts
4. security-scan - Trivy and Gosec security scans
5. docker-build - Build and push Docker images (main branch only)

Test Coverage:

• Go unit tests with race detection
• internal/queue package tests
• Zig CLI tests
• Integration tests
• Security audits

Release Workflow (.forgejo/workflows/release-mirror.yml)

Runs on version tags (e.g., v1.0.0).

Jobs:

1. build-cli (matrix build)

   • Linux x86_64 (static musl)
   • macOS x86_64
   • macOS ARM64
   • Downloads platform-specific static rsync
   • Embeds rsync for zero-dependency releases

2. build-go-backends

   • Cross-platform Go builds
   • api-server, worker, tui, data_manager, user_manager

3. create-release

   • Collects all artifacts
   • Generates SHA256 checksums
   • Mirrors release artifacts to GitHub Releases

Release Process

Creating a Release

# 1. Update version
git tag v1.0.0

# 2. Push tag
git push origin v1.0.0

# 3. CI automatically builds and releases

Release Artifacts

CLI Binaries (with embedded rsync):

• ml-linux-x86_64.tar.gz (~450-650KB)
• ml-macos-x86_64.tar.gz (~450-650KB)
• ml-macos-arm64.tar.gz (~450-650KB)

Go Backends:

• fetch_ml_api-server.tar.gz
• fetch_ml_worker.tar.gz
• fetch_ml_tui.tar.gz
• fetch_ml_data_manager.tar.gz
• fetch_ml_user_manager.tar.gz

Checksums:

• checksums.txt - Combined SHA256 sums
• Individual .sha256 files per binary

Development Workflow

Local Testing

# Run all tests
make test

# Run specific package tests
go test ./internal/queue/...

# Build CLI
cd cli && zig build --release=fast

# Run formatters and linters
make lint

# Security scans are handled automatically in CI by the `security-scan` job

Optional heavy end-to-end tests

Some e2e tests exercise full Docker deployments and performance scenarios and are skipped by default to keep local/CI runs fast. You can enable them explicitly with environment variables:

# Run Docker deployment e2e tests
FETCH_ML_E2E_DOCKER=1 go test ./tests/e2e/...

# Run performance-oriented e2e tests
FETCH_ML_E2E_PERF=1 go test ./tests/e2e/...

Without these variables, TestDockerDeploymentE2E and TestPerformanceE2E will t.Skip, while all lighter e2e tests still run.

Pull Request Checks

All PRs must pass:

• ✅ Go tests (with Redis)
• ✅ CLI tests
• ✅ Security scans
• ✅ Code linting
• ✅ Build verification

Configuration

Environment Variables

GO_VERSION: '1.25.0'
ZIG_VERSION: '0.15.2'

Secrets

Required for releases:

• GH_MIRROR_TOKEN - GitHub token for publishing mirrored releases
• GH_MIRROR_REPO (variable) - GitHub repo slug, e.g. jfraeysd/fetch_ml

Monitoring

Build Status

Check workflow runs at:

https://git.jfraeys.com/jfraeysd/fetch_ml/actions

Artifacts

Download build artifacts from:

• Successful workflow runs (30-day retention)
• GitHub Releases (permanent)

---

For implementation details:

• .forgejo/workflows/ci.yml
• .forgejo/workflows/release-mirror.yml