Jeremie Fraeys
90ea18555c
docs: add vLLM workflow and cross-link documentation
...
Security Scan / Security Analysis (push) Waiting to run
Security Scan / Native Library Security (push) Waiting to run
Verification & Maintenance / V.1 - Schema Drift Detection (push) Waiting to run
Verification & Maintenance / V.4 - Custom Go Vet Analyzers (push) Waiting to run
Verification & Maintenance / V.7 - Audit Chain Integrity (push) Waiting to run
Verification & Maintenance / V.6 - Extended Security Scanning (push) Waiting to run
Verification & Maintenance / V.10 - OpenSSF Scorecard (push) Waiting to run
Verification & Maintenance / Verification Summary (push) Blocked by required conditions
Build Pipeline / Build Binaries (push) Failing after 2m4s
Build Pipeline / Build Docker Images (push) Has been skipped
Build Pipeline / Sign HIPAA Config (push) Has been skipped
Build Pipeline / Generate SLSA Provenance (push) Has been skipped
Checkout test / test (push) Successful in 5s
CI Pipeline / Test (push) Failing after 1s
CI Pipeline / Dev Compose Smoke Test (push) Has been skipped
CI Pipeline / Security Scan (push) Has been skipped
CI Pipeline / Test Scripts (push) Has been skipped
CI Pipeline / Test Native Libraries (push) Has been skipped
CI Pipeline / Native Library Build Matrix (push) Has been skipped
Contract Tests / Spec Drift Detection (push) Failing after 16s
Contract Tests / API Contract Tests (push) Has been skipped
Deploy API Docs / Build API Documentation (push) Failing after 5s
Deploy API Docs / Deploy to GitHub Pages (push) Has been skipped
Documentation / build-and-publish (push) Failing after 44s
CI Pipeline / Trigger Build Workflow (push) Failing after 0s
- Add new vLLM workflow documentation (vllm-workflow.md)
- Update scheduler-architecture.md with Plugin GPU Quota and audit logging
- Add See Also sections to jupyter-workflow.md, quick-start.md,
configuration-reference.md for better navigation
- Update landing page and index with vLLM and scheduler links
- Cross-link all documentation for improved discoverability
2026-02-26 13:04:39 -05:00
Jeremie Fraeys
dddc2913e1
chore(tools): update scripts, native libs, and documentation
...
Update tooling and documentation:
- Smoke test script with scheduler health checks
- Release cleanup script
- Native test scripts with Redis integration
- TUI SSH test script
- Performance regression detector with scheduler metrics
- Profiler with distributed tracing
- Native CMake with test targets
- Dataset hash tests
- Storage symlink resistance tests
- Configuration reference documentation updates
2026-02-26 12:08:58 -05:00
Jeremie Fraeys
9b2d5986a3
docs(architecture): add technical documentation for scheduler and security
...
Add comprehensive architecture documentation:
- scheduler-architecture.md - Design of distributed job scheduler
- Hub coordination model
- Gang scheduling algorithm
- Service discovery mechanisms
- Failure recovery strategies
- multi-tenant-security.md - Security isolation patterns
- Tenant boundary enforcement
- Resource quota management
- Cross-tenant data protection
- runtime-security.md - Operational security guidelines
- Container security configurations
- Network policy enforcement
- Audit logging requirements
2026-02-26 12:04:33 -05:00
Jeremie Fraeys
799afb9efa
docs: update coverage map and development documentation
...
Comprehensive documentation updates for 100% test coverage:
- TEST_COVERAGE_MAP.md: 49/49 requirements marked complete (100% coverage)
- CHANGELOG.md: Document Phase 8 test coverage implementation
- DEVELOPMENT.md: Add testing strategy and property-based test guidelines
- README.md: Add Testing & Security section with coverage highlights
All security and reproducibility requirements now tracked and tested
2026-02-23 20:26:13 -05:00
Jeremie Fraeys
fe75b6e27a
build(verification): Add Makefile targets and CI for verification suite
...
Add verification targets to Makefile:
- verify-schema: Check manifest schema hasn't drifted (V.1)
- test-schema-validation: Test schema validation with examples
- lint-custom: Build and run fetchml-vet analyzers (V.4)
- verify-audit: Run audit chain verification tests (V.7)
- verify-audit-chain: CLI tool for verifying specific log files
- verify-all: Run all verification checks (CI target)
- verify-quick: Fast checks for development
- verify-full: Comprehensive verification with unit/integration tests
Add install targets for verification tools:
- install-property-test-deps: gopter for property-based testing (V.2)
- install-mutation-test-deps: go-mutesting for mutation testing (V.3)
- install-security-scan-deps: gosec, nancy for supply chain (V.6)
- install-scorecard: OpenSSF Scorecard (V.10)
Add Forgejo CI workflow (.forgejo/workflows/verification.yml):
- Runs on every push and PR
- Schema drift detection
- Custom linting
- Audit chain verification
- Security scanning integration
Add verification documentation (docs/src/verification.md):
- V.1: Schema validation details
- V.4: Custom linting rules
- V.7: Audit chain verification
- CI integration guide
2026-02-23 19:44:25 -05:00
Jeremie Fraeys
b00439b86e
docs(security): document comprehensive security hardening
...
Updates documentation with new security features and hardening guide:
**CHANGELOG.md:**
- Added detailed security hardening section (2026-02-23)
- Documents all phases: file ingestion, sandbox, secrets, audit logging, tests
- Lists specific files changed and security controls implemented
**docs/src/security.md:**
- Added Overview section with defense-in-depth layers
- Added Comprehensive Security Hardening section with:
- File ingestion security with code examples
- Sandbox hardening with complete YAML config
- Secrets management with env expansion syntax
- HIPAA audit logging with tamper-evident chain hashing
2026-02-23 18:03:25 -05:00
Jeremie Fraeys
2fdc9b9218
docs: Update privacy/security and research runner docs
...
Update native test storage
2026-02-23 14:13:35 -05:00
Jeremie Fraeys
3b194ff2e8
feat: GPU detection transparency and artifact scanner improvements
...
Build CLI with Embedded SQLite / build (arm64, aarch64-linux) (push) Waiting to run
Build CLI with Embedded SQLite / build (x86_64, x86_64-linux) (push) Waiting to run
Build CLI with Embedded SQLite / build-macos (arm64) (push) Waiting to run
Build CLI with Embedded SQLite / build-macos (x86_64) (push) Waiting to run
Security Scan / Security Analysis (push) Waiting to run
Security Scan / Native Library Security (push) Waiting to run
Checkout test / test (push) Successful in 6s
CI/CD Pipeline / Test (push) Failing after 1s
CI/CD Pipeline / Dev Compose Smoke Test (push) Has been skipped
CI/CD Pipeline / Build (push) Has been skipped
CI/CD Pipeline / Test Scripts (push) Has been skipped
CI/CD Pipeline / Test Native Libraries (push) Has been skipped
CI/CD Pipeline / GPU Golden Test Matrix (push) Has been skipped
Documentation / build-and-publish (push) Failing after 39s
CI/CD Pipeline / Docker Build (push) Has been skipped
- Surface GPUDetectionInfo from parseGPUCountFromConfig for detection metadata
- Document FETCH_ML_TOTAL_CPU and FETCH_ML_GPU_SLOTS_PER_GPU env vars
- Add debug logging for all env var overrides to stderr
- Track config-layer auto-detection in GPUDetectionInfo.ConfigLayerAutoDetected
- Add --include-all flag to artifact scanner (includeAll parameter)
- Add AMD production mode enforcement (error in non-local mode)
- Add GPU detector unit tests for env overrides and AMD aliasing
2026-02-23 12:29:34 -05:00
Jeremie Fraeys
d6265df0bd
docs: update all documentation to use build tags instead of deprecated env var
...
- README.md: Replace FETCHML_NATIVE_LIBS with -tags native_libs
- docs/src/native-libraries.md: Update all examples to use build tags
- .forgejo/workflows/ci-native.yml: Use -tags native_libs in all test steps
- Remove deprecated FETCHML_NATIVE_LIBS=1/0 env var references
2026-02-21 15:11:27 -05:00
Jeremie Fraeys
a3b957dcc0
refactor(cli): Update build system and core infrastructure
...
- Makefile: Update build targets for native library integration
- build.zig: Add SQLite linking and native hash library support
- scripts/build_rsync.sh: Update rsync embedded binary build process
- scripts/build_sqlite.sh: Add SQLite constants generation script
- src/assets/README.md: Document embedded asset structure
- src/utils/rsync_embedded_binary.zig: Update for new build layout
2026-02-20 21:39:51 -05:00
Jeremie Fraeys
f357624685
docs: Update CHANGELOG and add feature documentation
...
Update documentation for new features:
- Add CHANGELOG entries for research features and privacy enhancements
- Update README with new CLI commands and security features
- Add privacy-security.md documentation for PII detection
- Add research-features.md for narrative and outcome tracking
2026-02-18 21:28:25 -05:00
Jeremie Fraeys
3187ff26ea
refactor: complete maintainability phases 1-9 and fix all tests
...
Test fixes (all 41 test packages now pass):
- Fix ComputeTaskProvenance - add dataset_specs JSON output
- Fix EnforceTaskProvenance - populate all metadata fields in best-effort mode
- Fix PrewarmNextOnce - preserve prewarm state when queue empty
- Fix RunManifest directory creation in SetupJobDirectories
- Add ManifestWriter to test worker (simpleManifestWriter)
- Fix worker ID mismatch (use cfg.WorkerID)
- Fix WebSocket binary protocol responses
- Implement all WebSocket handlers: QueueJob, QueueJobWithSnapshot, StatusRequest,
CancelJob, Prune, ValidateRequest (with run manifest validation), LogMetric,
GetExperiment, DatasetList/Register/Info/Search
Maintainability phases completed:
- Phases 1-6: Domain types, error system, config boundaries, worker/API/queue splits
- Phase 7: TUI cleanup - reorganize model package (jobs.go, messages.go, styles.go, keys.go)
- Phase 8: MLServer unification - consolidate worker + TUI into internal/network/mlserver.go
- Phase 9: CI enforcement - add scripts/ci-checks.sh with 5 checks:
* No internal/ -> cmd/ imports
* domain/ has zero internal imports
* File size limit (500 lines, rigid)
* No circular imports
* Package naming conventions
Documentation:
- Add docs/src/file-naming-conventions.md
- Add make ci-checks target
Lines changed: +756/-36 (WebSocket fixes), +518/-320 (TUI), +263/-20 (Phase 8-9)
2026-02-17 20:32:14 -05:00
Jeremie Fraeys
6580917ba8
refactor: extract domain types and consolidate error system (Phases 1-2)
...
Phase 1: Extract Domain Types
=============================
- Create internal/domain/ package with canonical types:
- domain/task.go: Task, Attempt structs
- domain/tracking.go: TrackingConfig and MLflow/TensorBoard/Wandb configs
- domain/dataset.go: DatasetSpec
- domain/status.go: JobStatus constants
- domain/errors.go: FailureClass system with classification functions
- domain/doc.go: package documentation
- Update queue/task.go to re-export domain types (backward compatibility)
- Update TUI model/state.go to use domain types via type aliases
- Simplify TUI services: remove ~60 lines of conversion functions
Phase 2: Delete ErrorCategory System
====================================
- Remove deprecated ErrorCategory type and constants
- Remove TaskError struct and related functions
- Remove mapping functions: ClassifyError, IsRetryable, GetUserMessage, RetryDelay
- Update all queue implementations to use domain.FailureClass directly:
- queue/metrics.go: RecordTaskFailure/Retry now take FailureClass
- queue/queue.go: RetryTask uses domain.ClassifyFailure
- queue/filesystem_queue.go: RetryTask and MoveToDeadLetterQueue updated
- queue/sqlite_queue.go: RetryTask and MoveToDeadLetterQueue updated
Lines eliminated: ~190 lines of conversion and mapping code
Result: Single source of truth for domain types and error classification
2026-02-17 12:34:28 -05:00
Jeremie Fraeys
e286fd7769
chore: clean up temporary and build artifacts
...
- Remove old mermaid-init.js from assets directory (moved to static)
- Remove mem.prof profiling artifact
2026-02-16 20:39:34 -05:00
Jeremie Fraeys
fabb8fa1ee
docs: remove debug command from CLI reference
2026-02-16 20:39:20 -05:00
Jeremie Fraeys
6cc02b5efc
docs: add native libraries documentation and smoke tests
...
- Add comprehensive native-libraries.md documentation
- Add smoke-test-native.sh for testing native library builds
- Document build process, architecture, and testing strategy
2026-02-16 20:38:46 -05:00
Jeremie Fraeys
d673bce216
docs: fix mermaid graphs and update outdated content
...
- Fix mermaid graph syntax errors (escape parentheses in node labels)
- Move mermaid-init.js to Hugo static directory for correct MIME type
- Update Future Extensions section in cli-tui-ux-contract-v1.md to match current roadmap
- Add ADR-004 through ADR-007 documenting C++ native optimization strategy
2026-02-16 20:37:38 -05:00
Jeremie Fraeys
5144d291cb
docs: comprehensive documentation updates
...
- Add architecture, CI/CD, CLI reference documentation
- Update installation, operations, and quick-start guides
- Add Jupyter workflow and queue documentation
- New landing page and research runner plan
2026-02-12 12:05:27 -05:00
Jeremie Fraeys
3fa4f6ae51
docs: update architecture/queue pages and remove legacy development setup
2026-01-05 13:26:39 -05:00
Jeremie Fraeys
f6e506a632
chore(docs): remove legacy Jekyll docs/_pages after Hugo migration
2026-01-05 12:41:09 -05:00
Jeremie Fraeys
1aed78839b
docs(dev): document validate workflow, CLI/TUI UX contract, and consolidate dev/testing docs
2026-01-05 12:37:46 -05:00
Jeremie Fraeys
8157f73a70
docs(ops): consolidate deployment and performance monitoring docs for Caddy-based setup
2026-01-05 12:37:40 -05:00
Jeremie Fraeys
c0eeeda940
feat(experiment): improve experiment lifecycle and update first-experiment guide
2026-01-05 12:37:34 -05:00
Jeremie Fraeys
6b771e4a50
feat(jupyter): improve runtime management and update security/workflow docs
2026-01-05 12:37:27 -05:00
Jeremie Fraeys
17e5b9b71a
docs: migrate to Hugo Book theme and publish via GitHub Pages
2026-01-05 12:22:24 -05:00
Jeremie Fraeys
cd5640ebd2
Slim and secure: move scripts, clean configs, remove secrets
...
- Move ci-test.sh and setup.sh to scripts/
- Trim docs/src/zig-cli.md to current structure
- Replace hardcoded secrets with placeholders in configs
- Update .gitignore to block .env*, secrets/, keys, build artifacts
- Slim README.md to reflect current CLI/TUI split
- Add cleanup trap to ci-test.sh
- Ensure no secrets are committed
2025-12-07 13:57:51 -05:00
Jeremie Fraeys
9dd4261873
Add Jupyter workflow documentation to docs/
...
- Create comprehensive Jupyter Workflow Integration guide
- Include ML tools usage examples (MLflow, Streamlit, Dash, Panel, Bokeh)
- Add CLI integration instructions and troubleshooting
- Update docs index to reference new workflow guide
- Provide complete setup and usage instructions
2025-12-06 16:02:49 -05:00
Jeremie Fraeys
69dc9e6af4
Clean up README files and enhance testing documentation
...
- Remove 14 duplicate README files from test fixtures
- Clean up and restructure docs/src/testing.md with comprehensive testing guide
- Update main README.md to highlight testing and reference docs/ structure
- Remove empty .new files
- Keep only valuable, directory-specific README files (reduced from 34 to 20)
2025-12-06 15:43:51 -05:00
Jeremie Fraeys
605829dfc3
Update documentation for new features and fix outdated references
...
- Update deployment.md to reference new deployments/ directory structure
- Update CLI reference with new multi-user authentication system
- Add roles and permissions examples to configuration
- Fix docker-compose paths in testing documentation
- Remove references to non-existent docker-compose.test.yml
- Update troubleshooting with correct test commands
- Remove misplaced README files from test directories
2025-12-06 13:46:40 -05:00
Jeremie Fraeys
1d55fefd52
Add comprehensive documentation and testing framework
...
- Add configuration reference with all options and examples
- Add quick start testing guide for 5-minute setup
- Document all environment and worker configurations
- Add CLI configuration examples for all user roles
- Include permission system documentation
- Add troubleshooting guides and validation commands
- Complete testing framework with multiple documentation levels
Documentation now includes:
- Quick start testing guide (5-minute setup)
- Comprehensive configuration reference
- Testing protocol and guide
- Troubleshooting and validation
- Permission system reference
- Environment variable documentation
2025-12-06 13:09:56 -05:00
Jeremie Fraeys
5a19358d00
Organize configs and scripts, create testing protocol
...
- Reorganize configs into environments/, workers/, deprecated/ folders
- Reorganize scripts into testing/, deployment/, maintenance/, benchmarks/ folders
- Add comprehensive testing guide documentation
- Add new Makefile targets: test-full, test-auth, test-status
- Update script paths in Makefile to match new organization
- Create testing protocol documentation
- Add cleanup status checking functionality
Testing framework now includes:
- Quick authentication tests (make test-auth)
- Full test suite runner (make test-full)
- Cleanup status monitoring (make test-status)
- Comprehensive documentation and troubleshooting guides
2025-12-06 13:08:15 -05:00
Jeremie Fraeys
ea15af1833
Fix multi-user authentication and clean up debug code
...
- Fix YAML tags in auth config struct (json -> yaml)
- Update CLI configs to use pre-hashed API keys
- Remove double hashing in WebSocket client
- Fix port mapping (9102 -> 9103) in CLI commands
- Update permission keys to use jobs:read, jobs:create, etc.
- Clean up all debug logging from CLI and server
- All user roles now authenticate correctly:
* Admin: Can queue jobs and see all jobs
* Researcher: Can queue jobs and see own jobs
* Analyst: Can see status (read-only access)
Multi-user authentication is now fully functional.
2025-12-06 12:35:32 -05:00
Jeremie Fraeys
385d2cf386
docs: add comprehensive documentation with MkDocs site
...
- Add complete API documentation and architecture guides
- Include quick start, installation, and deployment guides
- Add troubleshooting and security documentation
- Include CLI reference and configuration schema docs
- Add production monitoring and operations guides
- Implement MkDocs configuration with search functionality
- Include comprehensive user and developer documentation
Provides complete documentation for users and developers
covering all aspects of the FetchML platform.
2025-12-04 16:54:57 -05:00
