---
layout: page
title: "CI/CD Pipeline"
permalink: /cicd/
nav_order: 5
---

# CI/CD Pipeline

Automated testing, building, and releasing for fetch_ml.

## Workflows

### CI Workflow (`.github/workflows/ci.yml`)

Runs on every push to `main`/`develop` and all pull requests.

**Jobs:**
1. **test** - Go backend tests with Redis
2. **build** - Build all binaries (Go + Zig CLI)
3. **test-scripts** - Validate deployment scripts
4. **security-scan** - Trivy and Gosec security scans
5. **docker-build** - Build and push Docker images (main branch only)

**Test Coverage:**
- Go unit tests with race detection
- `internal/queue` package tests
- Zig CLI tests  
- Integration tests
- Security audits

### Release Workflow (`.github/workflows/release.yml`)

Runs on version tags (e.g., `v1.0.0`).

**Jobs:**

1. **build-cli** (matrix build)
   - Linux x86_64 (static musl)
   - macOS x86_64
   - macOS ARM64
   - Downloads platform-specific static rsync
   - Embeds rsync for zero-dependency releases

2. **build-go-backends**
   - Cross-platform Go builds
   - api-server, worker, tui, data_manager, user_manager

3. **create-release**
   - Collects all artifacts
   - Generates SHA256 checksums
   - Creates GitHub release with notes

## Release Process

### Creating a Release

```bash
# 1. Update version
git tag v1.0.0

# 2. Push tag
git push origin v1.0.0

# 3. CI automatically builds and releases
```

### Release Artifacts

**CLI Binaries (with embedded rsync):**
- `ml-linux-x86_64.tar.gz` (~450-650KB)
- `ml-macos-x86_64.tar.gz` (~450-650KB)
- `ml-macos-arm64.tar.gz` (~450-650KB)

**Go Backends:**
- `fetch_ml_api-server.tar.gz`
- `fetch_ml_worker.tar.gz`
- `fetch_ml_tui.tar.gz`
- `fetch_ml_data_manager.tar.gz`
- `fetch_ml_user_manager.tar.gz`

**Checksums:**
- `checksums.txt` - Combined SHA256 sums
- Individual `.sha256` files per binary

## Development Workflow

### Local Testing

```bash
# Run all tests
make test

# Run specific package tests
go test ./internal/queue/...

# Build CLI
cd cli && zig build dev

# Run formatters and linters
make lint

# Security scans are handled automatically in CI by the `security-scan` job
```

#### Optional heavy end-to-end tests

Some e2e tests exercise full Docker deployments and performance scenarios and are
**skipped by default** to keep local/CI runs fast. You can enable them explicitly
with environment variables:

```bash
# Run Docker deployment e2e tests
FETCH_ML_E2E_DOCKER=1 go test ./tests/e2e/...

# Run performance-oriented e2e tests
FETCH_ML_E2E_PERF=1 go test ./tests/e2e/...
```

Without these variables, `TestDockerDeploymentE2E` and `TestPerformanceE2E` will
`t.Skip`, while all lighter e2e tests still run.

### Pull Request Checks

All PRs must pass:
- ✅ Go tests (with Redis)
- ✅ CLI tests
- ✅ Security scans
- ✅ Code linting
- ✅ Build verification

## Configuration

### Environment Variables

```yaml
GO_VERSION: '1.25.0'
ZIG_VERSION: '0.15.2'
```

### Secrets

Required for releases:
- `GITHUB_TOKEN` - Automatic, provided by GitHub Actions

## Monitoring

### Build Status

Check workflow runs at:
```
https://github.com/jfraeys/fetch_ml/actions
```

### Artifacts

Download build artifacts from:
- Successful workflow runs (30-day retention)
- GitHub Releases (permanent)

---

For implementation details:
- [.github/workflows/ci.yml](https://github.com/jfraeys/fetch_ml/blob/main/.github/workflows/ci.yml)
- [.github/workflows/release.yml](https://github.com/jfraeys/fetch_ml/blob/main/.github/workflows/release.yml)