fetch_ml/SECURITY.md

Security Policy

Reporting a Vulnerability

Please report security vulnerabilities to security@fetchml.io. Do NOT open public issues for security bugs.

Response timeline:

• Acknowledgment: within 48 hours
• Initial assessment: within 5 days
• Fix released: within 30 days (critical), 90 days (high)

Security Features

FetchML implements defense-in-depth security for ML research systems:

Authentication & Authorization

• Argon2id API Key Hashing: Memory-hard hashing resists GPU cracking
• RBAC with Role Inheritance: Granular permissions (admin, data_scientist, data_engineer, viewer, operator)
• Constant-time Comparison: Prevents timing attacks on key validation

Cryptographic Practices

• Ed25519 Manifest Signing: Tamper detection for run manifests
• SHA-256 with Salt: Legacy key support with migration path
• Secure Key Generation: 256-bit entropy for all API keys

Container Security

• Rootless Podman: No privileged containers
• Capability Dropping: --cap-drop ALL by default
• No New Privileges: no-new-privileges security opt
• Read-only Root Filesystem: Immutable base image

Input Validation

• Path Traversal Prevention: Canonical path validation
• Command Injection Protection: Shell metacharacter filtering
• Length Limits: Prevents DoS via oversized inputs

Audit & Monitoring

• Structured Audit Logging: JSON-formatted security events
• Hash-chained Logs: Tamper-evident audit trail
• Anomaly Detection: Brute force, privilege escalation alerts
• Security Metrics: Prometheus integration

Supply Chain

• Dependency Scanning: gosec + govulncheck in CI
• No unsafe Package: Prohibited in production code
• Manifest Signing: Ed25519 signatures for integrity

Supported Versions

Version	Supported
0.2.x	✅
0.1.x	❌

Security Checklist (Pre-Release)

Code Review

• No hardcoded secrets
• No unsafe usage without justification
• All user inputs validated
• All file paths canonicalized
• No secrets in error messages

Dependency Audit

• go mod verify passes
• govulncheck shows no vulnerabilities
• All dependencies pinned
• No unmaintained dependencies

Container Security

• No privileged containers
• Rootless execution
• Seccomp/AppArmor applied
• Network isolation

Cryptography

• Argon2id for key hashing
• Ed25519 for signing
• TLS 1.3 only
• No weak ciphers

Testing

• Security tests pass
• Fuzz tests for parsers
• Authentication bypass tested
• Container escape tested

Security Commands

# Run security scan
make security-scan

# Check for vulnerabilities
govulncheck ./...

# Static analysis
gosec ./...

# Check for unsafe usage
grep -r "unsafe\." --include="*.go" ./internal ./cmd

# Build with sanitizers
cd native && cmake -DENABLE_ASAN=ON .. && make

Threat Model

Attack Surfaces

1. External API: Researchers submitting malicious jobs
2. Container Runtime: Escape to host system
3. Data Exfiltration: Stealing datasets/models
4. Privilege Escalation: Researcher → admin
5. Supply Chain: Compromised dependencies
6. Secrets Leakage: API keys in logs/errors

Mitigations

Threat	Mitigation
Malicious Jobs	Input validation, container sandboxing, resource limits
Container Escape	Rootless, no-new-privileges, seccomp, read-only root
Data Exfiltration	Network policies, audit logging, rate limiting
Privilege Escalation	RBAC, least privilege, anomaly detection
Supply Chain	Dependency scanning, manifest signing, pinned versions
Secrets Leakage	Log sanitization, secrets manager, memory clearing

Responsible Disclosure

We follow responsible disclosure practices:

1. Report privately: Email security@fetchml.io with details
2. Provide details: Steps to reproduce, impact assessment
3. Allow time: We need 30-90 days to fix before public disclosure
4. Acknowledgment: We credit researchers who report valid issues

Security Team

• security@fetchml.io - Security issues and questions
• security-response@fetchml.io - Active incident response

---

Last updated: 2026-02-19

---

Security Guide for Fetch ML Homelab

The following section covers security best practices for deploying Fetch ML in a homelab environment.

Quick Setup

Secure setup requires manual configuration:

1. Generate API keys: Use the instructions in API Security below
2. Create TLS certificates: Use OpenSSL commands in Troubleshooting
3. Configure security: Copy and edit configs/api/homelab-secure.yaml
4. Set permissions: Ensure .api-keys and .env.secure have 600 permissions

Security Features

Authentication

• API Key Authentication: SHA256 hashed API keys
• Role-based Access Control: Admin, researcher, analyst roles
• Permission System: Granular permissions per resource

Network Security

• TLS/SSL: HTTPS encrypted communication
• IP Whitelisting: Restrict access to trusted networks
• Rate Limiting: Prevent abuse and DoS attacks
• Reverse Proxy: Caddy with security headers

Data Protection

• Path Traversal Protection: Prevents directory escape attacks
• Package Validation: Blocks dangerous Python packages
• Input Validation: Comprehensive input sanitization

Configuration Files

Secure Config Location

• configs/api/homelab-secure.yaml - Main secure configuration

API Keys

• .api-keys - Generated API keys (600 permissions)
• Never commit to version control
• Store in password manager

TLS Certificates

• ssl/cert.pem - TLS certificate
• ssl/key.pem - Private key (600 permissions)

Environment Variables

• .env.secure - JWT secret and other secrets (600 permissions)

Deployment Options

Option 1: Docker Compose (Recommended)

# Configure secure setup manually (see Quick Setup above)
# Copy and edit the secure configuration
cp configs/api/homelab-secure.yaml configs/api/my-secure.yaml
# Edit with your API keys, TLS settings, and IP whitelist

# Deploy with security overlay
docker-compose -f docker-compose.yml -f docker-compose.homelab-secure.yml up -d

Option 2: Direct Deployment

# Configure secure setup manually (see Quick Setup above)
# Copy and edit the secure configuration
cp configs/api/homelab-secure.yaml configs/api/my-secure.yaml
# Edit with your API keys, TLS settings, and IP whitelist

# Load environment variables
source .env.secure

# Start server
./api-server -config configs/api/my-secure.yaml

Security Checklist

Before Deployment

• Generate unique API keys (don't use defaults)
• Set strong JWT secret
• Enable TLS/SSL
• Configure IP whitelist for your network
• Set up rate limiting
• Enable Redis authentication

Network Security

• Use HTTPS only (disable HTTP)
• Restrict API access to trusted IPs
• Use reverse proxy (caddy)
• Enable security headers
• Monitor access logs

Data Protection

• Regular backups of configuration
• Secure storage of API keys
• Encrypt sensitive data at rest
• Regular security updates

Monitoring

• Enable security logging
• Monitor failed authentication attempts
• Set up alerts for suspicious activity
• Regular security audits

API Security

Authentication Headers

# Use API key in header
curl -H "X-API-Key: your-api-key" https://localhost:9101/health

# Or Bearer token
curl -H "Authorization: Bearer your-api-key" https://localhost:9101/health

Rate Limits

• Default: 60 requests per minute
• Burst: 10 requests
• Per IP address

IP Whitelisting

Configure in config-homelab-secure.yaml:

security:
  ip_whitelist:
    - "127.0.0.1"
    - "192.168.1.0/24"  # Your local network

Container Security

Docker Security

• Use non-root users
• Minimal container images
• Resource limits
• Network segmentation

Podman Security

• Rootless containers
• SELinux confinement
• Seccomp profiles
• Read-only filesystems where possible

Troubleshooting

Common Issues

TLS Certificate Errors

# Regenerate certificates
openssl req -x509 -newkey rsa:4096 -keyout ssl/key.pem -out ssl/cert.pem -days 365 -nodes \
  -subj "/C=US/ST=Homelab/L=Local/O=FetchML/CN=localhost"

API Key Authentication Failed

# Check your API key
grep "ADMIN_API_KEY" .api-keys

# Verify hash matches config
echo -n "your-api-key" | sha256sum | cut -d' ' -f1

IP Whitelist Blocking

# Check your IP
curl -s https://api.ipify.org

# Add to whitelist in config

Security Logs

Monitor these files:

• logs/fetch_ml.log - Application logs
• Caddy access logs (configure if enabled)
• Docker logs: docker logs ml-experiments-api

Best Practices

1. Regular Updates: Keep dependencies updated
2. Principle of Least Privilege: Minimal required permissions
3. Defense in Depth: Multiple security layers
4. Monitor and Alert: Security monitoring
5. Backup and Recovery: Regular secure backups

Emergency Procedures

Compromised API Keys

1. Immediately revoke compromised keys
2. Generate new API keys
3. Update all clients
4. Review access logs

Security Incident

1. Isolate affected systems
2. Preserve evidence
3. Review access logs
4. Update security measures
5. Document incident

Support

For security issues:

• Check logs for error messages
• Review configuration files
• Test with minimal setup
• Report security vulnerabilities responsibly