fetch_ml/docs/src/cicd.md

title	url	weight
CI/CD Pipeline	/cicd/	5

CI/CD Pipeline

Automated testing, building, and releasing for fetch_ml.

Workflows

CI Workflow (.forgejo/workflows/ci.yml)

Runs on every push to main/develop and all pull requests.

Jobs:

1. test - Go backend tests with Redis
2. build - Build all binaries (Go + Zig CLI)
3. test-scripts - Validate deployment scripts
4. security-scan - Trivy and Gosec security scans
5. docker-build - Build and push Docker images (main branch only)

Test Coverage:

• Go unit tests with race detection
• internal/queue package tests
• Zig CLI tests
• Integration tests
• Security audits

Test Matrix Workflow (.forgejo/workflows/test-matrix.yml)

Tests all build configurations (pure, cgo, native) on every push to main and PRs.

Build Types:

• pure - CGO_ENABLED=0, fully static, no dependencies
• cgo - CGO_ENABLED=1, OS-specific features
• native - CGO_ENABLED=1 with native C++ libraries for performance

Jobs:

1. Build with each build type
2. Run unit tests
3. Run integration tests
4. Compare benchmarks (native vs pure)

Release Workflow (.forgejo/workflows/release-mirror.yml)

Runs on version tags (e.g., v1.0.0). Builds for Linux x86_64 only (ARM64 deferred, macOS removed per deployment plan).

Jobs:

1. build-cli

   • Linux x86_64 (static musl)
   • Downloads platform-specific static rsync
   • Embeds rsync for zero-dependency releases

2. build-go-backends

   • Linux x86_64 Go builds only
   • api-server, worker, tui, data_manager, user_manager

3. create-release

   • Collects all artifacts
   • Generates SHA256 checksums
   • Mirrors release artifacts to GitHub Releases

Release Process

Creating a Release

# 1. Update version
git tag v1.0.0

# 2. Push tag
git push origin v1.0.0

# 3. CI automatically builds and releases

Release Artifacts

CLI Binaries (with embedded rsync):

• ml-linux-x86_64.tar.gz (~450-650KB)

Go Backends (Linux x86_64 only):

• fetch_ml_api-server_linux_amd64.tar.gz
• fetch_ml_worker_linux_amd64.tar.gz
• fetch_ml_tui_linux_amd64.tar.gz
• fetch_ml_data_manager_linux_amd64.tar.gz
• fetch_ml_user_manager_linux_amd64.tar.gz

Build Types:

• _pure - CGO_ENABLED=0, fully static
• _cgo - CGO_ENABLED=1, OS-specific features
• _native - CGO_ENABLED=1 with native C++ libraries

Checksums:

• checksums.txt - Combined SHA256 sums
• Individual .sha256 files per binary

Note: ARM64 and macOS builds deferred per deployment plan. Can be added later if needed.

Development Workflow

Build Scripts

The project includes modular build scripts in scripts/build/ for different build configurations:

# Build native C++ libraries
./scripts/build/build-native.sh

# Build Go binary (usage: build-go.sh <pure|cgo|native> <os> <arch> <source>)
./scripts/build/build-go.sh pure linux amd64 cmd/api-server/main.go
./scripts/build/build-go.sh native linux amd64 cmd/worker/worker_server.go

# Build CLI with Zig
./scripts/build/build-cli.sh

# Full build orchestration (all build types)
./scripts/build/cross-platform.sh

# Or use Makefile targets
make build-pure    # Pure Go binaries
make build-cgo     # CGO binaries  
make build-native  # Native library binaries
make build-cli     # CLI only
make build-all     # Everything

Local Testing

# Run all tests
make test

# Run specific package tests
go test ./internal/queue/...

# Build CLI
cd cli && zig build --release=fast

# Run formatters and linters
make lint

# Security scans are handled automatically in CI by the `security-scan` job

Optional heavy end-to-end tests

Some e2e tests exercise full Docker deployments and performance scenarios and are skipped by default to keep local/CI runs fast. You can enable them explicitly with environment variables:

# Run Docker deployment e2e tests
FETCH_ML_E2E_DOCKER=1 go test ./tests/e2e/...

# Run performance-oriented e2e tests
FETCH_ML_E2E_PERF=1 go test ./tests/e2e/...

Without these variables, TestDockerDeploymentE2E and TestPerformanceE2E will t.Skip, while all lighter e2e tests still run.

Pull Request Checks

All PRs must pass:

• ✅ Go tests (with Redis)
• ✅ CLI tests
• ✅ Security scans
• ✅ Code linting
• ✅ Build verification

Configuration

Environment Variables

GO_VERSION: '1.25.0'
ZIG_VERSION: '0.15.2'

Secrets

Required for releases:

• GH_MIRROR_TOKEN - GitHub token for publishing mirrored releases
• GH_MIRROR_REPO (variable) - GitHub repo slug, e.g. jfraeysd/fetch_ml

Monitoring

Build Status

Check workflow runs at:

https://git.jfraeys.com/jfraeysd/fetch_ml/actions

Artifacts

Download build artifacts from:

• Successful workflow runs (30-day retention)
• GitHub Releases (permanent)

---

For implementation details:

• .forgejo/workflows/ci.yml
• .forgejo/workflows/release-mirror.yml