# Verification & Maintenance

Continuous enforcement, drift detection, and compliance maintenance for the FetchML platform.

## Overview

The verification layer provides structural enforcement at compile time, behavioral invariants across random inputs, drift detection from security baseline, supply chain integrity, and audit log verification. Together these form the difference between "we tested this once" and "we can prove it holds continuously."

## Components

### Schema Validation

**Purpose:** Ensures that `manifest.Artifacts`, `Config`, and `SandboxConfig` structs match a versioned schema at compile time. If a field is added, removed, or retyped without updating the schema, the build fails.

**Files:**
- `internal/manifest/schema.json` - Canonical JSON Schema for manifest validation
- `internal/manifest/schema_version.go` - Schema versioning and compatibility
- `internal/manifest/schema_test.go` - Drift detection tests

**Key Invariants:**
- `Environment` field is required and non-null in every `Artifacts` record
- `Environment.ConfigHash` is a non-empty string
- `Environment.DetectionMethod` is one of enumerated values
- `Exclusions` is present (may be empty array, never null)
- `compliance_mode` if present is one of `"hipaa"`, `"standard"`

**Commands:**
```bash
make verify-schema           # Check schema hasn't drifted
make test-schema-validation  # Test validation works correctly
```

**CI Integration:** Runs on every commit via `verification.yml` workflow.

### Custom Linting Rules

**Purpose:** Enforces structural invariants that can't be expressed as tests—such as `CreateDetector` never being called without capturing `DetectionInfo`, or any function returning `manifest.Artifacts` populating `Environment`.

**Tool:** Custom `go vet` analyzers using `golang.org/x/tools/go/analysis`.

**Analyzers:**

| Analyzer | Rule | Rationale |
|----------|------|-----------|
| `nobaredetector` | Flag any call to `GPUDetectorFactory.CreateDetector()` not assigned to a variable also receiving `CreateDetectorWithInfo()` | `CreateDetector` silently discards `GPUDetectionInfo` needed for manifest and audit log |
| `manifestenv` | Flag any function with return type `manifest.Artifacts` where `Environment` field is not explicitly set before return | Enforces schema validation at the call site, not just in tests |
| `noinlinecreds` | Flag any struct literal of type `Config` where `RedisPassword`, `SecretKey`, or `AccessKey` fields are set to non-empty string literals | Credentials must not appear in source or config files |
| `hippacomplete` | Flag any switch or if-else on `compliance_mode == "hipaa"` that does not check all six hard-required fields | Prevents partial HIPAA enforcement from silently passing |

**Files:**
- `tools/fetchml-vet/cmd/fetchml-vet/main.go` - CLI entry point
- `tools/fetchml-vet/analyzers/nobaredetector.go`
- `tools/fetchml-vet/analyzers/manifestenv.go`
- `tools/fetchml-vet/analyzers/noinlinecredentials.go`
- `tools/fetchml-vet/analyzers/hipaacomplete.go`

**Commands:**
```bash
make lint-custom             # Build and run custom analyzers
```

**CI Integration:** Runs on every commit via `verification.yml` workflow. Lint failures are build failures, not warnings.

### Audit Chain Integrity Verification

**Purpose:** Proves audit logs have not been tampered with by verifying the integrity chain. Each entry includes a hash of the previous entry, forming a Merkle-chain. Any insertion, deletion, or modification breaks the chain.

**Implementation:**

```go
type Event struct {
    Timestamp    time.Time              `json:"timestamp"`
    EventType    EventType              `json:"event_type"`
    UserID       string                 `json:"user_id,omitempty"`
    // ... other fields
    PrevHash     string                 `json:"prev_hash"`     // hash of previous entry
    EntryHash    string                 `json:"entry_hash"`    // hash of this entry's fields + prev_hash
    SequenceNum  int64                  `json:"sequence_num"`
}
```

**Components:**
- `internal/audit/verifier.go` - Chain verification logic
- `cmd/audit-verifier/main.go` - Standalone CLI tool
- `tests/unit/audit/verifier_test.go` - Unit tests

**Features:**
- **Continuous verification:** Background job runs every 15 minutes (HIPAA) or hourly (other)
- **Tamper detection:** Identifies first sequence number where chain breaks
- **External verification:** Chain root hash can be published to append-only store (S3 Object Lock, Azure Immutable Blob)

**Commands:**
```bash
make verify-audit                                    # Run unit tests
make verify-audit-chain AUDIT_LOG_PATH=/path/to.log  # Verify specific log file
```

**CLI Usage:**
```bash
# Single verification
./bin/audit-verifier -log-path=/var/log/fetchml/audit.log

# Continuous monitoring
./bin/audit-verifier -log-path=/var/log/fetchml/audit.log -continuous -interval=15m
```

**CI Integration:** Runs on every commit via `verification.yml` workflow.

## Maintenance Cadence

| Activity | Frequency | Blocking | Location |
|----------|-----------|----------|----------|
| Schema drift check | Every commit | Yes | `verification.yml` |
| Property-based tests | Every commit | Yes | `verification.yml` (planned) |
| Custom lint rules | Every commit | Yes | `verification.yml` |
| gosec + nancy | Every commit | Yes | `security-scan.yml` |
| trivy image scan | Every commit | Yes (CRITICAL) | `security-scan.yml` |
| Audit chain verification | 15 min (HIPAA), hourly | Alerts | Deployment config |
| Mutation testing | Pre-release | Yes (< 80%) | Release workflow (planned) |
| Fault injection | Nightly + pre-release | Yes (pre-release) | Nightly workflow (planned) |
| OpenSSF Scorecard | Weekly | Alerts (>1pt drop) | Weekly workflow (planned) |
| Reproducibility | Toolchain changes | Yes | Verify workflow (planned) |

## Usage

### Quick Verification (Development)

```bash
make verify-quick    # Fast checks: schema only
```

### Full Verification (CI)

```bash
make verify-all      # All verification checks
```

### Install Verification Tools

```bash
make install-verify-deps    # Install all verification tooling
```

## CI/CD Integration

The `verification.yml` workflow runs automatically on:
- Every push to `main` or `develop`
- Every pull request to `main` or `develop`
- Nightly (for scorecard and extended checks)

Jobs:
1. **schema-drift-check** - Schema validation
2. **custom-lint** - Custom analyzers
3. **audit-verification** - Audit chain integrity
4. **security-scan-extended** - Extended security scanning
5. **scorecard** - OpenSSF Scorecard (weekly)

## Planned Components

| Component | Status | Description |
|-----------|--------|-------------|
| Property-Based Testing | Planned | `gopter` for behavioral invariants across all valid inputs |
| Mutation Testing | Planned | `go-mutesting` to verify tests catch security invariants |
| SLSA Conformance | Planned | Supply chain provenance at Level 2/3 |
| Continuous Scanning | Partial | trivy, grype, checkov, nancy integration |
| Reproducible Builds | Planned | Binary and container image reproducibility |
| Fault Injection | Planned | toxiproxy, libfiu for resilience testing |
| OpenSSF Scorecard | Partial | Scorecard evaluation and badge |

## Relationship to Security Plan

This verification layer builds on the Security Plan by adding continuous enforcement:

```
Security Plan (implement controls)
    ↓
Verification Plan (enforce and maintain controls)
    ↓
Ongoing: scanning, scoring, fault injection, audit verification
```

The Compliance Dashboard and Compliance Reporting features from the Security Plan consume outputs from this verification layer—scan results, mutation scores, SLSA provenance, Scorecard ratings, and audit chain verification status feed directly into compliance reporting.