5d75f3576b
- Update TEST_COVERAGE_MAP with current requirements - Refresh ADR-004 with C++ implementation details - Update architecture, deployment, and security docs - Improve CLI/TUI UX contract documentation
209 lines
4.9 KiB
Markdown
209 lines
4.9 KiB
Markdown
---
|
|
title: "CI/CD Pipeline"
|
|
url: "/cicd/"
|
|
weight: 5
|
|
---
|
|
|
|
# CI/CD Pipeline
|
|
|
|
Automated testing, building, and releasing for fetch_ml.
|
|
|
|
## Workflows
|
|
|
|
### CI Workflow (`.forgejo/workflows/ci.yml`)
|
|
|
|
Runs on every push to `main`/`develop` and all pull requests.
|
|
|
|
**Jobs:**
|
|
1. **test** - Go backend tests with Redis
|
|
2. **build** - Build all binaries (Go + Zig CLI)
|
|
3. **test-scripts** - Validate deployment scripts
|
|
4. **security-scan** - Trivy and Gosec security scans
|
|
5. **docker-build** - Build and push Docker images (main branch only)
|
|
|
|
**Test Coverage:**
|
|
- Go unit tests with race detection
|
|
- `internal/queue` package tests
|
|
- Zig CLI tests
|
|
- Integration tests
|
|
- Security audits
|
|
|
|
### Test Matrix Workflow (`.forgejo/workflows/test-matrix.yml`)
|
|
|
|
Tests all build configurations (pure, cgo, native) on every push to main and PRs.
|
|
|
|
**Build Types:**
|
|
- **pure** - CGO_ENABLED=0, fully static, no dependencies
|
|
- **cgo** - CGO_ENABLED=1, OS-specific features
|
|
- **native** - CGO_ENABLED=1 with native C++ libraries for performance
|
|
|
|
**Jobs:**
|
|
1. Build with each build type
|
|
2. Run unit tests
|
|
3. Run integration tests
|
|
4. Compare benchmarks (native vs pure)
|
|
|
|
### Release Workflow (`.forgejo/workflows/release-mirror.yml`)
|
|
|
|
Runs on version tags (e.g., `v1.0.0`). Builds for **Linux x86_64 only** (ARM64 deferred, macOS removed per deployment plan).
|
|
|
|
**Jobs:**
|
|
|
|
1. **build-cli**
|
|
- Linux x86_64 (static musl)
|
|
- Downloads platform-specific static rsync
|
|
- Embeds rsync for zero-dependency releases
|
|
|
|
2. **build-go-backends**
|
|
- Linux x86_64 Go builds only
|
|
- api-server, worker, tui, data_manager, user_manager
|
|
|
|
3. **create-release**
|
|
- Collects all artifacts
|
|
- Generates SHA256 checksums
|
|
- Mirrors release artifacts to GitHub Releases
|
|
|
|
## Release Process
|
|
|
|
### Creating a Release
|
|
|
|
```bash
|
|
# 1. Update version
|
|
git tag v1.0.0
|
|
|
|
# 2. Push tag
|
|
git push origin v1.0.0
|
|
|
|
# 3. CI automatically builds and releases
|
|
```
|
|
|
|
### Release Artifacts
|
|
|
|
**CLI Binaries (with embedded rsync):**
|
|
- `ml-linux-x86_64.tar.gz` (~450-650KB)
|
|
|
|
**Go Backends (Linux x86_64 only):**
|
|
- `fetch_ml_api-server_linux_amd64.tar.gz`
|
|
- `fetch_ml_worker_linux_amd64.tar.gz`
|
|
- `fetch_ml_tui_linux_amd64.tar.gz`
|
|
- `fetch_ml_data_manager_linux_amd64.tar.gz`
|
|
- `fetch_ml_user_manager_linux_amd64.tar.gz`
|
|
|
|
**Build Types:**
|
|
- `_pure` - CGO_ENABLED=0, fully static
|
|
- `_cgo` - CGO_ENABLED=1, OS-specific features
|
|
- `_native` - CGO_ENABLED=1 with native C++ libraries
|
|
|
|
**Checksums:**
|
|
- `checksums.txt` - Combined SHA256 sums
|
|
- Individual `.sha256` files per binary
|
|
|
|
**Note:** ARM64 and macOS builds deferred per deployment plan. Can be added later if needed.
|
|
|
|
## Development Workflow
|
|
|
|
### Build Scripts
|
|
|
|
The project includes modular build scripts in `scripts/build/` for different build configurations:
|
|
|
|
```bash
|
|
# Build native C++ libraries
|
|
./scripts/build/build-native.sh
|
|
|
|
# Build Go binary (usage: build-go.sh <pure|cgo|native> <os> <arch> <source>)
|
|
./scripts/build/build-go.sh pure linux amd64 cmd/api-server/main.go
|
|
./scripts/build/build-go.sh native linux amd64 cmd/worker/worker_server.go
|
|
|
|
# Build CLI with Zig
|
|
./scripts/build/build-cli.sh
|
|
|
|
# Full build orchestration (all build types)
|
|
./scripts/build/cross-platform.sh
|
|
|
|
# Or use Makefile targets
|
|
make build-pure # Pure Go binaries
|
|
make build-cgo # CGO binaries
|
|
make build-native # Native library binaries
|
|
make build-cli # CLI only
|
|
make build-all # Everything
|
|
```
|
|
|
|
### Local Testing
|
|
|
|
```bash
|
|
# Run all tests
|
|
make test
|
|
|
|
# Run specific package tests
|
|
go test ./internal/queue/...
|
|
|
|
# Build CLI
|
|
cd cli && zig build --release=fast
|
|
|
|
# Run formatters and linters
|
|
make lint
|
|
|
|
# Security scans are handled automatically in CI by the `security-scan` job
|
|
```
|
|
|
|
#### Optional heavy end-to-end tests
|
|
|
|
Some e2e tests exercise full Docker deployments and performance scenarios and are
|
|
**skipped by default** to keep local/CI runs fast. You can enable them explicitly
|
|
with environment variables:
|
|
|
|
```bash
|
|
# Run Docker deployment e2e tests
|
|
FETCH_ML_E2E_DOCKER=1 go test ./tests/e2e/...
|
|
|
|
# Run performance-oriented e2e tests
|
|
FETCH_ML_E2E_PERF=1 go test ./tests/e2e/...
|
|
```
|
|
|
|
Without these variables, `TestDockerDeploymentE2E` and `TestPerformanceE2E` will
|
|
`t.Skip`, while all lighter e2e tests still run.
|
|
|
|
### Pull Request Checks
|
|
|
|
All PRs must pass:
|
|
- ✅ Go tests (with Redis)
|
|
- ✅ CLI tests
|
|
- ✅ Security scans
|
|
- ✅ Code linting
|
|
- ✅ Build verification
|
|
|
|
## Configuration
|
|
|
|
### Environment Variables
|
|
|
|
```yaml
|
|
GO_VERSION: '1.25.0'
|
|
ZIG_VERSION: '0.15.2'
|
|
```
|
|
|
|
### Secrets
|
|
|
|
Required for releases:
|
|
- `GH_MIRROR_TOKEN` - GitHub token for publishing mirrored releases
|
|
- `GH_MIRROR_REPO` (variable) - GitHub repo slug, e.g. `jfraeysd/fetch_ml`
|
|
|
|
## Monitoring
|
|
|
|
### Build Status
|
|
|
|
Check workflow runs at:
|
|
```
|
|
https://git.jfraeys.com/jfraeysd/fetch_ml/actions
|
|
```
|
|
|
|
### Artifacts
|
|
|
|
Download build artifacts from:
|
|
- Successful workflow runs (30-day retention)
|
|
- GitHub Releases (permanent)
|
|
|
|
---
|
|
|
|
For implementation details:
|
|
- `.forgejo/workflows/ci.yml`
|
|
- `.forgejo/workflows/release-mirror.yml`
|