# ADR-005: Use Batch-First APIs for CGo Overhead Amortization

## Status
Proposed

## Context
CGo has a boundary crossing overhead of approximately 100ns per call. If native functions are called individually for many small files, the CGo overhead negates syscall gains. This is particularly problematic for `dataset_hash`, which historically operated on a per-file basis.

The naive approach:
```go
// SLOW: CGo overhead dominates
for _, file := range files {
    hash := C.fh_hash_file(file)  // 100ns + syscall
}
```

The optimized approach must batch operations to amortize the 100ns overhead across many files.

## Decision
All C++ native libraries MUST expose **batch-first APIs**. Single-function interfaces are explicitly prohibited for high-call-frequency operations.

**Required API Pattern:**
```go
// Batch hash directory contents
// Single CGo call for entire directory
func dirOverallSHA256HexNative(root string) (string, error) {
    croot := C.CString(root)
    defer C.free(unsafe.Pointer(croot))
    
    result := C.fh_hash_directory_combined(croot, 0)  // 0 = auto threads
    if result == nil {
        return "", errors.New("hash failed")
    }
    defer C.fh_free_string(result)
    
    return C.GoString(result), nil
}
```

**C++ Interface Design:**
```c
// dataset_hash.h - REVISED (batch only)
// ALWAYS use batch interface. Single-file interface removed.

// Batch hash directory contents
// out_hashes: pre-allocated array of 65-char buffers
int fh_hash_directory_batch(
    const char* dir_path,
    char** out_hashes,
    char** out_paths,
    uint32_t max_results,
    uint32_t* out_count,
    uint32_t num_threads
);

// Simple combined hash (single CGo call)
char* fh_hash_directory_combined(const char* dir_path, uint32_t num_threads);
```

## Consequences

### Positive
- CGo overhead amortized across N files (100ns total vs 100ns × N)
- Single CGo call per operation simplifies error handling
- Forces coarse-grained operations that match Go concurrency model

### Negative
- API less granular than pure Go equivalent
- Caller must batch requests or use Go fallback
- Memory management for pre-allocated buffers

## Options Considered

### Option 1: Single-File Native with Caching
**Pros:** Simple API, matches Go stdlib  
**Cons:** CGo overhead > syscall win for small files  
**Verdict:** Rejected - overhead dominates

### Option 2: Automatic Batching Layer in Go
**Pros:** Transparent to caller  
**Cons:** Complexity, unpredictable latency, buffering issues  
**Verdict:** Rejected - explicit is better than implicit

### Option 3: Batch-First APIs (Chosen)
**Pros:** Predictable performance, caller controls batching  
**Cons:** Requires caller awareness  
**Verdict:** Accepted - performance-critical paths use batch explicitly

## Enforcement

### Build-Time
- Single-file C functions removed from headers
- `native_bridge.go` only exposes batch interfaces

### Run-Time
- Go fallback for small batches (<10 files or <1MB total)
- Native only used when batch size justifies overhead

```go
func hashFilesBatch(paths []string) ([]string, error) {
    if !UseNativeLibs || len(paths) == 0 {
        return hashFilesBatchGo(paths)
    }
    
    // Only use native for batches >10 files or total size >1MB
    totalSize := estimateSize(paths)
    if len(paths) < 10 && totalSize < 1024*1024 {
        return hashFilesBatchGo(paths)
    }
    
    return hashFilesBatchNative(paths)
}
```

### CI Enforcement
- Benchmarks fail if CGo > 1% of total runtime
- Profile-guided check: `go test -cpuprofile` analysis

## Rationale

The 100ns CGo overhead is fixed and unavoidable. The only solution is to make each crossing do more work. Batch-first APIs ensure that syscall wins (78-96% reduction) exceed CGo costs by operating on coarse-grained units (directories, not files; task arrays, not individual tasks).

This is a trade-off: less granular APIs for predictable performance. The batch constraint is documented and enforced at API boundary, not hidden in implementation.
