---
title: "CI/CD Pipeline"
url: "/cicd/"
weight: 5
---

# CI/CD Pipeline

Automated testing, building, and releasing for fetch_ml.

## Workflows

### CI Workflow (`.forgejo/workflows/ci.yml`)

Runs on every push to `main`/`develop` and all pull requests.

**Jobs:**
1. **test** - Go backend tests with Redis
2. **build** - Build all binaries (Go + Zig CLI)
3. **test-scripts** - Validate deployment scripts
4. **security-scan** - Trivy and Gosec security scans
5. **docker-build** - Build and push Docker images (main branch only)

**Test Coverage:**
- Go unit tests with race detection
- `internal/queue` package tests
- Zig CLI tests  
- Integration tests
- Security audits

### Test Matrix Workflow (`.forgejo/workflows/test-matrix.yml`)

Tests all build configurations (pure, cgo, native) on every push to main and PRs.

**Build Types:**
- **pure** - CGO_ENABLED=0, fully static, no dependencies
- **cgo** - CGO_ENABLED=1, OS-specific features
- **native** - CGO_ENABLED=1 with native C++ libraries for performance

**Jobs:**
1. Build with each build type
2. Run unit tests
3. Run integration tests
4. Compare benchmarks (native vs pure)

### Release Workflow (`.forgejo/workflows/release-mirror.yml`)

Runs on version tags (e.g., `v1.0.0`). Builds for **Linux x86_64 only** (ARM64 deferred, macOS removed per deployment plan).

**Jobs:**

1. **build-cli**
   - Linux x86_64 (static musl)
   - Downloads platform-specific static rsync
   - Embeds rsync for zero-dependency releases

2. **build-go-backends**
   - Linux x86_64 Go builds only
   - api-server, worker, tui, data_manager, user_manager

3. **create-release**
   - Collects all artifacts
   - Generates SHA256 checksums
   - Mirrors release artifacts to GitHub Releases

## Release Process

### Creating a Release

```bash
# 1. Update version
git tag v1.0.0

# 2. Push tag
git push origin v1.0.0

# 3. CI automatically builds and releases
```

### Release Artifacts

**CLI Binaries (with embedded rsync):**
- `ml-linux-x86_64.tar.gz` (~450-650KB)

**Go Backends (Linux x86_64 only):**
- `fetch_ml_api-server_linux_amd64.tar.gz`
- `fetch_ml_worker_linux_amd64.tar.gz`
- `fetch_ml_tui_linux_amd64.tar.gz`
- `fetch_ml_data_manager_linux_amd64.tar.gz`
- `fetch_ml_user_manager_linux_amd64.tar.gz`

**Build Types:**
- `_pure` - CGO_ENABLED=0, fully static
- `_cgo` - CGO_ENABLED=1, OS-specific features  
- `_native` - CGO_ENABLED=1 with native C++ libraries

**Checksums:**
- `checksums.txt` - Combined SHA256 sums
- Individual `.sha256` files per binary

**Note:** ARM64 and macOS builds deferred per deployment plan. Can be added later if needed.

## Development Workflow

### Build Scripts

The project includes modular build scripts in `scripts/build/` for different build configurations:

```bash
# Build native C++ libraries
./scripts/build/build-native.sh

# Build Go binary (usage: build-go.sh <pure|cgo|native> <os> <arch> <source>)
./scripts/build/build-go.sh pure linux amd64 cmd/api-server/main.go
./scripts/build/build-go.sh native linux amd64 cmd/worker/worker_server.go

# Build CLI with Zig
./scripts/build/build-cli.sh

# Full build orchestration (all build types)
./scripts/build/cross-platform.sh

# Or use Makefile targets
make build-pure    # Pure Go binaries
make build-cgo     # CGO binaries  
make build-native  # Native library binaries
make build-cli     # CLI only
make build-all     # Everything
```

### Local Testing

```bash
# Run all tests
make test

# Run specific package tests
go test ./internal/queue/...

# Build CLI
cd cli && zig build --release=fast

# Run formatters and linters
make lint

# Security scans are handled automatically in CI by the `security-scan` job
```

#### Optional heavy end-to-end tests

Some e2e tests exercise full Docker deployments and performance scenarios and are
**skipped by default** to keep local/CI runs fast. You can enable them explicitly
with environment variables:

```bash
# Run Docker deployment e2e tests
FETCH_ML_E2E_DOCKER=1 go test ./tests/e2e/...

# Run performance-oriented e2e tests
FETCH_ML_E2E_PERF=1 go test ./tests/e2e/...
```

Without these variables, `TestDockerDeploymentE2E` and `TestPerformanceE2E` will
`t.Skip`, while all lighter e2e tests still run.

### Pull Request Checks

All PRs must pass:
- ✅ Go tests (with Redis)
- ✅ CLI tests
- ✅ Security scans
- ✅ Code linting
- ✅ Build verification

## Configuration

### Environment Variables

```yaml
GO_VERSION: '1.25.0'
ZIG_VERSION: '0.15.2'
```

### Secrets

Required for releases:
- `GH_MIRROR_TOKEN` - GitHub token for publishing mirrored releases
- `GH_MIRROR_REPO` (variable) - GitHub repo slug, e.g. `jfraeysd/fetch_ml`

## Monitoring

### Build Status

Check workflow runs at:
```
https://git.jfraeys.com/jfraeysd/fetch_ml/actions
```

### Artifacts

Download build artifacts from:
- Successful workflow runs (30-day retention)
- GitHub Releases (permanent)

---

For implementation details:
- `.forgejo/workflows/ci.yml`
- `.forgejo/workflows/release-mirror.yml`