diff --git a/docs/src/configuration-reference.md b/docs/src/configuration-reference.md new file mode 100644 index 0000000..111b4e3 --- /dev/null +++ b/docs/src/configuration-reference.md @@ -0,0 +1,331 @@ +# Configuration Reference + +## Overview + +This document provides a comprehensive reference for all configuration options in the FetchML project. + +## Environment Configurations + +### Local Development +**File:** `configs/environments/config-local.yaml` + +```yaml +auth: + enabled: true + apikeys: + dev_user: + hash: "2baf1f40105d9501fe319a8ec463fdf4325a2a5df445adf3f572f626253678c9" + admin: true + roles: ["admin"] + permissions: + "*": true + +server: + address: ":9101" + tls: + enabled: false + +security: + rate_limit: + enabled: false + ip_whitelist: + - "127.0.0.1" + - "::1" + - "localhost" +``` + +### Multi-User Setup +**File:** `configs/environments/config-multi-user.yaml` + +```yaml +auth: + enabled: true + apikeys: + admin_user: + hash: "5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8" + admin: true + roles: ["user", "admin"] + permissions: + read: true + write: true + delete: true + + researcher1: + hash: "ef92b778ba7a6c8f2150019a5678047b6a9a2b95cef8189518f9b35c54d2e3ae" + admin: false + roles: ["user", "researcher"] + permissions: + jobs:read: true + jobs:create: true + jobs:update: true + jobs:delete: false + + analyst1: + hash: "a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3" + admin: false + roles: ["user", "analyst"] + permissions: + jobs:read: true + jobs:create: false + jobs:update: false + jobs:delete: false +``` + +### Production +**File:** `configs/environments/config-prod.yaml` + +```yaml +auth: + enabled: true + apikeys: + # Production users configured here + +server: + address: ":9101" + tls: + enabled: true + cert_file: "/app/ssl/cert.pem" + key_file: "/app/ssl/key.pem" + +security: + rate_limit: + enabled: true + requests_per_minute: 30 + ip_whitelist: + - "127.0.0.1" + - "::1" + - "192.168.0.0/16" + - "10.0.0.0/8" + +redis: + url: "redis://redis:6379" + max_connections: 10 + +logging: + level: "info" + file: "/app/logs/app.log" + audit_file: "/app/logs/audit.log" +``` + +## Worker Configurations + +### Production Worker +**File:** `configs/workers/worker-prod.toml` + +```toml +[worker] +name = "production-worker" +id = "worker-prod-1" + +[server] +host = "api-server" +port = 9101 +api_key = "your-api-key-here" + +[execution] +max_concurrent_jobs = 2 +timeout_minutes = 60 +retry_attempts = 3 + +[resources] +memory_limit = "4Gi" +cpu_limit = "2" +gpu_enabled = false + +[storage] +work_dir = "/tmp/fetchml-jobs" +cleanup_interval_minutes = 30 +``` + +### Docker Worker +**File:** `configs/workers/worker-docker.yaml` + +```yaml +worker: + name: "docker-worker" + id: "worker-docker-1" + +server: + host: "api-server" + port: 9101 + api_key: "your-api-key-here" + +execution: + max_concurrent_jobs: 1 + timeout_minutes: 30 + retry_attempts: 3 + +resources: + memory_limit: "2Gi" + cpu_limit: "1" + gpu_enabled: false + +docker: + enabled: true + image: "fetchml/worker:latest" + volume_mounts: + - "/tmp:/tmp" + - "/var/run/docker.sock:/var/run/docker.sock" +``` + +## CLI Configuration + +### User Config File +**Location:** `~/.ml/config.toml` + +```toml +[server] +worker_host = "localhost" +worker_user = "appuser" +worker_base = "/app" +worker_port = 22 + +[auth] +api_key = "your-hashed-api-key" + +[cli] +default_timeout = 30 +verbose = false +``` + +### Multi-User CLI Configs + +**Admin Config:** `~/.ml/config-admin.toml` +```toml +[server] +worker_host = "localhost" +worker_user = "appuser" +worker_base = "/app" +worker_port = 22 + +[auth] +api_key = "5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8" +``` + +**Researcher Config:** `~/.ml/config-researcher.toml` +```toml +[server] +worker_host = "localhost" +worker_user = "appuser" +worker_base = "/app" +worker_port = 22 + +[auth] +api_key = "ef92b778ba7a6c8f2150019a5678047b6a9a2b95cef8189518f9b35c54d2e3ae" +``` + +**Analyst Config:** `~/.ml/config-analyst.toml` +```toml +[server] +worker_host = "localhost" +worker_user = "appuser" +worker_base = "/app" +worker_port = 22 + +[auth] +api_key = "a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3" +``` + +## Configuration Options + +### Authentication + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `auth.enabled` | bool | false | Enable authentication | +| `auth.apikeys` | map | {} | API key configurations | +| `auth.apikeys.[user].hash` | string | - | SHA256 hash of API key | +| `auth.apikeys.[user].admin` | bool | false | Admin privileges | +| `auth.apikeys.[user].roles` | array | [] | User roles | +| `auth.apikeys.[user].permissions` | map | {} | User permissions | + +### Server + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `server.address` | string | ":9101" | Server bind address | +| `server.tls.enabled` | bool | false | Enable TLS | +| `server.tls.cert_file` | string | - | TLS certificate file | +| `server.tls.key_file` | string | - | TLS private key file | + +### Security + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `security.rate_limit.enabled` | bool | true | Enable rate limiting | +| `security.rate_limit.requests_per_minute` | int | 60 | Rate limit | +| `security.ip_whitelist` | array | [] | Allowed IP addresses | + +### Redis + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `redis.url` | string | "redis://localhost:6379" | Redis connection URL | +| `redis.max_connections` | int | 10 | Max Redis connections | + +### Logging + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `logging.level` | string | "info" | Log level | +| `logging.file` | string | - | Log file path | +| `logging.audit_file` | string | - | Audit log path | + +## Permission System + +### Permission Keys + +| Permission | Description | +|------------|-------------| +| `jobs:read` | Read job information | +| `jobs:create` | Create new jobs | +| `jobs:update` | Update existing jobs | +| `jobs:delete` | Delete jobs | +| `*` | All permissions (admin only) | + +### Role-Based Permissions + +| Role | Default Permissions | +|------|-------------------| +| admin | All permissions | +| researcher | jobs:read, jobs:create, jobs:update | +| analyst | jobs:read | +| user | No default permissions | + +## Environment Variables + +| Variable | Default | Description | +|----------|---------|-------------| +| `FETCHML_CONFIG` | - | Path to config file | +| `FETCHML_LOG_LEVEL` | "info" | Override log level | +| `FETCHML_REDIS_URL` | - | Override Redis URL | +| `CLI_CONFIG` | - | Path to CLI config file | + +## Troubleshooting + +### Common Configuration Issues + +1. **Authentication Failures** + - Check API key hashes are correct SHA256 + - Verify YAML syntax + - Ensure auth.enabled: true + +2. **Connection Issues** + - Verify server address and ports + - Check firewall settings + - Validate network connectivity + +3. **Permission Issues** + - Check user roles and permissions + - Verify permission key format + - Ensure admin users have "*": true + +### Configuration Validation + +```bash +# Validate server configuration +go run cmd/api-server/main.go --config configs/environments/config-local.yaml --validate + +# Test CLI configuration +./cli/zig-out/bin/ml status --debug +``` \ No newline at end of file diff --git a/docs/src/quick-start-testing.md b/docs/src/quick-start-testing.md new file mode 100644 index 0000000..e350ce8 --- /dev/null +++ b/docs/src/quick-start-testing.md @@ -0,0 +1,185 @@ +# Quick Start Testing Guide + +## Overview + +This guide provides the fastest way to test the FetchML multi-user authentication system. + +## Prerequisites + +- Docker and Docker Compose installed +- CLI built: `cd cli && zig build` +- Test configs available in `~/.ml/` + +## 5-Minute Test + +### 1. Clean Environment +```bash +make self-cleanup +``` + +### 2. Start Services +```bash +docker-compose -f docker-compose.prod.yml up -d +``` + +### 3. Test Authentication +```bash +make test-auth +``` + +### 4. Check Results +You should see: +- Admin user: Full access, shows all jobs +- Researcher user: Own jobs only +- Analyst user: Read-only access + +### 5. Clean Up +```bash +make self-cleanup +``` + +## Detailed Testing + +### Multi-User Authentication Test + +```bash +# Test each user role +cp ~/.ml/config-admin.toml ~/.ml/config.toml && ./cli/zig-out/bin/ml status +cp ~/.ml/config-researcher.toml ~/.ml/config.toml && ./cli/zig-out/bin/ml status +cp ~/.ml/config-analyst.toml ~/.ml/config.toml && ./cli/zig-out/bin/ml status +``` + +### Job Queueing Test + +```bash +# Admin can queue jobs +cp ~/.ml/config-admin.toml ~/.ml/config.toml +echo "admin job" | ./cli/zig-out/bin/ml queue admin-test + +# Researcher can queue jobs +cp ~/.ml/config-researcher.toml ~/.ml/config.toml +echo "research job" | ./cli/zig-out/bin/ml queue research-test + +# Analyst cannot queue jobs (should fail) +cp ~/.ml/config-analyst.toml ~/.ml/config.toml +echo "analysis job" | ./cli/zig-out/bin/ml queue analysis-test +``` + +### Status Verification + +```bash +# Check what each user can see +make test-auth +``` + +## Expected Results + +### Admin User Output +``` +Status retrieved for user: admin_user (admin: true) +Tasks: X total, X queued, X running, X failed, X completed +``` + +### Researcher User Output +``` +Status retrieved for user: researcher1 (admin: false) +Tasks: X total, X queued, X running, X failed, X completed +``` + +### Analyst User Output +``` +Status retrieved for user: analyst1 (admin: false) +Tasks: X total, X queued, X running, X failed, X completed +``` + +## Troubleshooting + +### Server Not Running +```bash +# Check containers +docker ps --filter "name=ml-" + +# Start services +docker-compose -f docker-compose.prod.yml up -d + +# Check logs +docker logs ml-prod-api +``` + +### Authentication Failures +```bash +# Check config files +ls ~/.ml/config-*.toml + +# Verify API keys +cat ~/.ml/config-admin.toml +``` + +### Connection Issues +```bash +# Test API directly +curl -I http://localhost:9103/health + +# Check ports +netstat -an | grep 9103 +``` + +## Advanced Testing + +### Full Test Suite +```bash +make test-full +``` + +### Performance Testing +```bash +./scripts/benchmarks/run-benchmarks-local.sh +``` + +### Cleanup Status +```bash +make test-status +``` + +## Configuration Files + +### Test Configs Location +- `~/.ml/config-admin.toml` - Admin user +- `~/.ml/config-researcher.toml` - Researcher user +- `~/.ml/config-analyst.toml` - Analyst user + +### Server Configs +- `configs/environments/config-multi-user.yaml` - Multi-user setup +- `configs/environments/config-local.yaml` - Local development + +## Next Steps + +1. **Review Documentation** + - [Testing Protocol](testing-protocol.md) + - [Configuration Reference](configuration-reference.md) + - [Testing Guide](testing-guide.md) + +2. **Explore Features** + - Job queueing and management + - WebSocket communication + - Role-based permissions + +3. **Production Setup** + - TLS configuration + - Security hardening + - Monitoring setup + +## Help + +### Common Commands +```bash +make help # Show all commands +make test-auth # Quick auth test +make self-cleanup # Clean environment +make test-status # Check system status +``` + +### Get Help +- Check logs: `docker logs ml-prod-api` +- Review documentation in `docs/src/` +- Use `--debug` flag with CLI commands \ No newline at end of file