jfraeysd/fetch_ml
0
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
443 commits 1 branch 0 tags 38 MiB
Commit graph

14 commits

Author SHA1 Message Date
Jeremie Fraeys
cf84246115
refactor: co-locate config, container, envpool, errors, experiment, jupyter tests 
Move unit tests from tests/unit/ to internal/ following Go conventions:
- tests/unit/config/* -> internal/config/* (constants, mode_paths, paths, validation)
- tests/unit/container/* -> internal/container/* (podman, security tests)
- tests/unit/envpool/* -> internal/envpool/* (envpool tests)
- tests/unit/errors/* -> internal/errtypes/* (errors_test.go moved to errtypes package)
- tests/unit/experiment/* -> internal/experiment/* (manager tests)
- tests/unit/jupyter/* -> internal/jupyter/* (config, package_blacklist, service_manager, trash_restore)

Update import paths in test files to reflect new locations.

Note: errors_test.go moved from tests/unit/errors/ to internal/errtypes/ to match the package structure.
 2026-03-12 16:35:15 -04:00
Jeremie Fraeys
4b2782f674
feat(domain): add task visibility and supporting infrastructure 
Core domain and utility updates:

- domain/task.go: Task model with visibility system
  * Visibility enum: private, lab, institution, open
  * Group associations for lab-scoped access
  * CreatedBy tracking for ownership
  * Sharing metadata with expiry

- config/paths.go: Group-scoped data directories and audit log paths
- crypto/signing.go: Key management for audit sealing, token signature verification
- container/supply_chain.go: Image provenance tracking, vulnerability scanning
- fileutil/filetype.go: MIME type detection and security validation
- fileutil/secure.go: Protected file permissions, secure deletion
- jupyter/: Package and service manager updates
- experiment/manager.go: Visibility cascade from experiments to tasks
- network/ssh.go: SSH tunneling improvements
- queue/: Filesystem queue enhancements
 2026-03-08 13:03:27 -04:00
Jeremie Fraeys
66f262d788
security: improve audit, crypto, and config handling 
- Enhance audit checkpoint system
- Update KMS provider and tenant key management
- Refine configuration constants
- Improve TUI config handling
 2026-03-04 13:23:42 -05:00
Jeremie Fraeys
420de879ff
feat(api): integrate scheduler protocol and WebSocket enhancements 
Update API layer for scheduler integration:
- WebSocket handlers with scheduler protocol support
- Jobs WebSocket endpoint with priority queue integration
- Validation middleware for scheduler messages
- Server configuration with security hardening
- Protocol definitions for worker-scheduler communication
- Dataset handlers with tenant isolation checks
- Response helpers with audit context
- OpenAPI spec updates for new endpoints
 2026-02-26 12:05:57 -05:00
Jeremie Fraeys
23e5f3d1dc
refactor(api): internal refactoring for TUI and worker modules 
- Refactor internal/worker and internal/queue packages
- Update cmd/tui for monitoring interface
- Update test configurations
 2026-02-20 15:51:23 -05:00
Jeremie Fraeys
6028779239
feat: update CLI, TUI, and security documentation 
- Add safety checks to Zig build
- Add TUI with job management and narrative views
- Add WebSocket support and export services
- Add smart configuration defaults
- Update API routes with security headers
- Update SECURITY.md with comprehensive policy
- Add Makefile security scanning targets
 2026-02-19 15:35:05 -05:00
Jeremie Fraeys
e4d286f2e5
feat: add security monitoring and validation framework 
- Implement anomaly detection monitor (brute force, path traversal, etc.)
- Add input validation framework with safety rules
- Add environment-based secrets manager with redaction
- Add security test suite for path traversal and injection
- Add CI security scanning workflow
 2026-02-19 15:34:25 -05:00
Jeremie Fraeys
4756348c48
feat: Worker sandboxing and security configuration 
Add security hardening features for worker execution:
- Worker config with sandboxing options (network_mode, read_only, secrets)
- Execution setup with security context propagation
- Podman container runtime security enhancements
- Security configuration management in config package
- Add homelab-sandbox.yaml example configuration

Supports running jobs in isolated, restricted environments.
 2026-02-18 21:27:59 -05:00
Jeremie Fraeys
e127f97442
chore: implement centralized path registry and file organization conventions 
Add PathRegistry for centralized path management:
- Create internal/config/paths.go with PathRegistry type
- Binary paths: BinDir(), APIServerBinary(), WorkerBinary(), etc.
- Data paths: DataDir(), JupyterStateDir(), ExperimentsDir()
- Config paths: ConfigDir(), APIServerConfig()
- Helper methods: EnsureDir(), EnsureDirSecure(), FileExists()
- Auto-detect repo root by looking for go.mod

Update .gitignore for root protection:
- Add explicit /api-server, /worker, /tui, /data_manager rules
- Add /coverage.out and .DS_Store to root protection
- Prevents accidental commits of binaries to root

Add path verification script:
- Create scripts/verify-paths.sh
- Checks for binaries in root directory
- Checks for .DS_Store files
- Checks for coverage.out in root
- Verifies data/ is gitignored
- Returns exit code 1 on violations

Cleaned .DS_Store files from repository
 2026-02-18 16:48:50 -05:00
Jeremie Fraeys
3775bc3ee0
refactor: replace panic with error returns and update maintenance 
- Replace 9 panic() calls in smart_defaults.go with error returns
- Add ErrUnknownProfile error type for better error handling
- Update all callers (worker/config.go, tui/config.go, tui/cli_config.go, tui/main.go)
- Update CHANGELOG.md with recent WebSocket handler improvements
- Add metrics persistence, dataset handlers, and test organization notes
- Config validation passes (make configlint)
- All tests pass (go test ./tests/unit/api/ws)
 2026-02-18 14:44:21 -05:00
Jeremie Fraeys
d1bef0a450
refactor: Phase 3 - fix config/storage boundaries 
Move schema ownership to infrastructure layer:

- Redis keys: config/constants.go -> queue/keys.go (TaskQueueKey, TaskPrefix, etc.)

- Filesystem paths: config/paths.go -> storage/paths.go (JobPaths)

- Create config/shared.go with RedisConfig, SSHConfig

- Update all imports: worker/, api/helpers, api/ws_jobs, api/ws_validate

- Clean up: remove duplicates from queue/task.go, queue/queue.go, config/paths.go

Build status: Compiles successfully
 2026-02-17 12:49:53 -05:00
Jeremie Fraeys
add4a90e62 feat(api): refactor websocket handlers; add health and prometheus middleware 2026-01-05 12:31:07 -05:00
Jeremie Fraeys
ea15af1833 Fix multi-user authentication and clean up debug code 
- Fix YAML tags in auth config struct (json -> yaml)
- Update CLI configs to use pre-hashed API keys
- Remove double hashing in WebSocket client
- Fix port mapping (9102 -> 9103) in CLI commands
- Update permission keys to use jobs:read, jobs:create, etc.
- Clean up all debug logging from CLI and server
- All user roles now authenticate correctly:
  * Admin: Can queue jobs and see all jobs
  * Researcher: Can queue jobs and see own jobs
  * Analyst: Can see status (read-only access)

Multi-user authentication is now fully functional.
 2025-12-06 12:35:32 -05:00
Jeremie Fraeys
803677be57 feat: implement Go backend with comprehensive API and internal packages 
- Add API server with WebSocket support and REST endpoints
- Implement authentication system with API keys and permissions
- Add task queue system with Redis backend and error handling
- Include storage layer with database migrations and schemas
- Add comprehensive logging, metrics, and telemetry
- Implement security middleware and network utilities
- Add experiment management and container orchestration
- Include configuration management with smart defaults
 2025-12-04 16:53:53 -05:00