# Native C++ Libraries

FetchML includes optional C++ native libraries for performance-critical operations. These libraries are loaded dynamically via cgo and provide significant syscall reduction compared to pure Go implementations.

## Overview

| Library | Purpose | Syscall Reduction |
|---------|---------|-------------------|
| **dataset_hash** | mmap + SIMD SHA256 hashing | 78% |
| **queue_index** | Binary index format | 96% |
| **artifact_scanner** | Fast directory traversal | 87% |
| **streaming_io** | Parallel gzip extraction | 95% |

## Requirements

- CMake 3.15+
- C++17 compiler
- zlib

## Building

### Development Build

```bash
make native-build
```

### Production Optimized

```bash
make native-release  # -O3 optimized
```

### Debug with ASan

```bash
make native-debug    # AddressSanitizer enabled
```

### Smoke Test

```bash
make native-smoke    # C++ tests + Go integration
```

## Enabling at Runtime

```bash
export FETCHML_NATIVE_LIBS=1
```

## Deployment

Ship the native libraries alongside your Go binaries:

- **Linux**: `lib*.so` files
- **macOS**: `lib*.dylib` files

The libraries are loaded dynamically via cgo. If not found, FetchML automatically falls back to pure Go implementations.

## Building with Native Support

```bash
make prod-with-native    # Copies .so/.dylib files to bin/
```

## Architecture

### Library Structure

```
native/
├── common/          # Shared utilities (mmap, thread pool, arena)
├── queue_index/     # Storage, heap, priority queue
└── dataset_hash/    # Crypto, I/O, threading
```

### Security Boundaries

All native libraries implement input validation at C API boundaries:

- **Path validation**: Rejects traversal sequences (`..`) and null bytes
- **Buffer safety**: `strncpy` with explicit null termination
- **Mmap limits**: 100MB cap prevents unbounded memory exposure
- **Atomic writes**: Temp file + rename ensures data integrity

## Testing

### C++ Unit Tests

```bash
cd native/build && ctest --output-on-failure
```

### Go Integration Tests

```bash
FETCHML_NATIVE_LIBS=1 go test ./tests/benchmarks/...
FETCHML_NATIVE_LIBS=1 go test ./tests/e2e/...
```

### ASan Build

```bash
cmake .. -DENABLE_ASAN=ON
make
ASAN_OPTIONS=detect_leaks=1 ./test_executable
```

## Performance Validation

Run benchmarks to verify native libraries outperform pure Go:

```bash
# Go implementation
FETCHML_NATIVE_LIBS=0 go test -bench=. ./tests/benchmarks/

# Native implementation
FETCHML_NATIVE_LIBS=1 go test -bench=. ./tests/benchmarks/
```

## Troubleshooting

### Library not found

Ensure the native libraries are in the library search path:

```bash
# Linux
export LD_LIBRARY_PATH=/path/to/native/build:$LD_LIBRARY_PATH

# macOS
export DYLD_LIBRARY_PATH=/path/to/native/build:$DYLD_LIBRARY_PATH
```

### Build errors

Common issues:

1. **Missing cmake**: Install with `apt-get install cmake` or `brew install cmake`
2. **Missing C++ compiler**: Install `build-essential` (Linux) or Xcode (macOS)
3. **Missing zlib**: Install `zlib1g-dev` (Linux) or it's built-in (macOS)