385d2cf386
- Add complete API documentation and architecture guides - Include quick start, installation, and deployment guides - Add troubleshooting and security documentation - Include CLI reference and configuration schema docs - Add production monitoring and operations guides - Implement MkDocs configuration with search functionality - Include comprehensive user and developer documentation Provides complete documentation for users and developers covering all aspects of the FetchML platform.
165 lines
3.4 KiB
Markdown
165 lines
3.4 KiB
Markdown
---
|
|
layout: page
|
|
title: "CI/CD Pipeline"
|
|
permalink: /cicd/
|
|
nav_order: 5
|
|
---
|
|
|
|
# CI/CD Pipeline
|
|
|
|
Automated testing, building, and releasing for fetch_ml.
|
|
|
|
## Workflows
|
|
|
|
### CI Workflow (`.github/workflows/ci.yml`)
|
|
|
|
Runs on every push to `main`/`develop` and all pull requests.
|
|
|
|
**Jobs:**
|
|
1. **test** - Go backend tests with Redis
|
|
2. **build** - Build all binaries (Go + Zig CLI)
|
|
3. **test-scripts** - Validate deployment scripts
|
|
4. **security-scan** - Trivy and Gosec security scans
|
|
5. **docker-build** - Build and push Docker images (main branch only)
|
|
|
|
**Test Coverage:**
|
|
- Go unit tests with race detection
|
|
- `internal/queue` package tests
|
|
- Zig CLI tests
|
|
- Integration tests
|
|
- Security audits
|
|
|
|
### Release Workflow (`.github/workflows/release.yml`)
|
|
|
|
Runs on version tags (e.g., `v1.0.0`).
|
|
|
|
**Jobs:**
|
|
|
|
1. **build-cli** (matrix build)
|
|
- Linux x86_64 (static musl)
|
|
- macOS x86_64
|
|
- macOS ARM64
|
|
- Downloads platform-specific static rsync
|
|
- Embeds rsync for zero-dependency releases
|
|
|
|
2. **build-go-backends**
|
|
- Cross-platform Go builds
|
|
- api-server, worker, tui, data_manager, user_manager
|
|
|
|
3. **create-release**
|
|
- Collects all artifacts
|
|
- Generates SHA256 checksums
|
|
- Creates GitHub release with notes
|
|
|
|
## Release Process
|
|
|
|
### Creating a Release
|
|
|
|
```bash
|
|
# 1. Update version
|
|
git tag v1.0.0
|
|
|
|
# 2. Push tag
|
|
git push origin v1.0.0
|
|
|
|
# 3. CI automatically builds and releases
|
|
```
|
|
|
|
### Release Artifacts
|
|
|
|
**CLI Binaries (with embedded rsync):**
|
|
- `ml-linux-x86_64.tar.gz` (~450-650KB)
|
|
- `ml-macos-x86_64.tar.gz` (~450-650KB)
|
|
- `ml-macos-arm64.tar.gz` (~450-650KB)
|
|
|
|
**Go Backends:**
|
|
- `fetch_ml_api-server.tar.gz`
|
|
- `fetch_ml_worker.tar.gz`
|
|
- `fetch_ml_tui.tar.gz`
|
|
- `fetch_ml_data_manager.tar.gz`
|
|
- `fetch_ml_user_manager.tar.gz`
|
|
|
|
**Checksums:**
|
|
- `checksums.txt` - Combined SHA256 sums
|
|
- Individual `.sha256` files per binary
|
|
|
|
## Development Workflow
|
|
|
|
### Local Testing
|
|
|
|
```bash
|
|
# Run all tests
|
|
make test
|
|
|
|
# Run specific package tests
|
|
go test ./internal/queue/...
|
|
|
|
# Build CLI
|
|
cd cli && zig build dev
|
|
|
|
# Run formatters and linters
|
|
make lint
|
|
|
|
# Security scans are handled automatically in CI by the `security-scan` job
|
|
```
|
|
|
|
#### Optional heavy end-to-end tests
|
|
|
|
Some e2e tests exercise full Docker deployments and performance scenarios and are
|
|
**skipped by default** to keep local/CI runs fast. You can enable them explicitly
|
|
with environment variables:
|
|
|
|
```bash
|
|
# Run Docker deployment e2e tests
|
|
FETCH_ML_E2E_DOCKER=1 go test ./tests/e2e/...
|
|
|
|
# Run performance-oriented e2e tests
|
|
FETCH_ML_E2E_PERF=1 go test ./tests/e2e/...
|
|
```
|
|
|
|
Without these variables, `TestDockerDeploymentE2E` and `TestPerformanceE2E` will
|
|
`t.Skip`, while all lighter e2e tests still run.
|
|
|
|
### Pull Request Checks
|
|
|
|
All PRs must pass:
|
|
- ✅ Go tests (with Redis)
|
|
- ✅ CLI tests
|
|
- ✅ Security scans
|
|
- ✅ Code linting
|
|
- ✅ Build verification
|
|
|
|
## Configuration
|
|
|
|
### Environment Variables
|
|
|
|
```yaml
|
|
GO_VERSION: '1.25.0'
|
|
ZIG_VERSION: '0.15.2'
|
|
```
|
|
|
|
### Secrets
|
|
|
|
Required for releases:
|
|
- `GITHUB_TOKEN` - Automatic, provided by GitHub Actions
|
|
|
|
## Monitoring
|
|
|
|
### Build Status
|
|
|
|
Check workflow runs at:
|
|
```
|
|
https://github.com/jfraeys/fetch_ml/actions
|
|
```
|
|
|
|
### Artifacts
|
|
|
|
Download build artifacts from:
|
|
- Successful workflow runs (30-day retention)
|
|
- GitHub Releases (permanent)
|
|
|
|
---
|
|
|
|
For implementation details:
|
|
- [.github/workflows/ci.yml](https://github.com/jfraeys/fetch_ml/blob/main/.github/workflows/ci.yml)
|
|
- [.github/workflows/release.yml](https://github.com/jfraeys/fetch_ml/blob/main/.github/workflows/release.yml)
|