jfraeysd/fetch_ml
0
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
443 commits 1 branch 0 tags 38 MiB
Commit graph

9 commits

Author SHA1 Message Date
Jeremie Fraeys
ca6ad970c3
refactor: co-locate logging, manifest, network, privacy, prommetrics tests 
Move unit tests from tests/unit/ to internal/ following Go conventions:
- tests/unit/logging/* -> internal/logging/* (logging tests)
- tests/unit/manifest/* -> internal/manifest/* (run_manifest, schema tests)
- tests/unit/network/* -> internal/network/* (retry, ssh_pool, ssh tests)
- tests/unit/privacy/* -> internal/privacy/* (pii tests)
- tests/unit/metrics/* -> internal/prommetrics/* (metrics tests)

Update import paths in test files to reflect new locations.

Note: metrics_test.go moved from tests/unit/metrics/ to internal/prommetrics/ to match the actual package name.
 2026-03-12 16:35:37 -04:00
Jeremie Fraeys
4b2782f674
feat(domain): add task visibility and supporting infrastructure 
Core domain and utility updates:

- domain/task.go: Task model with visibility system
  * Visibility enum: private, lab, institution, open
  * Group associations for lab-scoped access
  * CreatedBy tracking for ownership
  * Sharing metadata with expiry

- config/paths.go: Group-scoped data directories and audit log paths
- crypto/signing.go: Key management for audit sealing, token signature verification
- container/supply_chain.go: Image provenance tracking, vulnerability scanning
- fileutil/filetype.go: MIME type detection and security validation
- fileutil/secure.go: Protected file permissions, secure deletion
- jupyter/: Package and service manager updates
- experiment/manager.go: Visibility cascade from experiments to tasks
- network/ssh.go: SSH tunneling improvements
- queue/: Filesystem queue enhancements
 2026-03-08 13:03:27 -04:00
Jeremie Fraeys
4cdb68907e
refactor(utilities): update supporting modules for scheduler integration 
Update utility modules:
- File utilities with secure file operations
- Environment pool with resource tracking
- Error types with scheduler error categories
- Logging with audit context support
- Network/SSH with connection pooling
- Privacy/PII handling with tenant boundaries
- Resource manager with scheduler allocation
- Security monitor with audit integration
- Tracking plugins (MLflow, TensorBoard) with auth
- Crypto signing with tenant keys
- Database init with multi-user support
 2026-02-26 12:07:15 -05:00
Jeremie Fraeys
320e6fd409
refactor(dependency-hygiene): Move path functions from config to storage 
Move ExpandPath function and path-related utilities from internal/config to internal/storage where they belong.

Files updated:
- internal/worker/config.go: use storage.ExpandPath
- internal/network/ssh.go: use storage.ExpandPath
- cmd/data_manager/data_manager_config.go: use storage.ExpandPath
- internal/api/server_config.go: use storage.ExpandPath

internal/storage/paths.go already contained the canonical implementation.

Result: Path utilities now live in storage layer, config package focuses on configuration structs.
 2026-02-17 21:15:23 -05:00
Jeremie Fraeys
fb2bbbaae5
refactor: Phase 7 - TUI cleanup - reorganize model package 
Phase 7 of the monorepo maintainability plan:

New files created:
- model/jobs.go - Job type, JobStatus constants, list.Item interface
- model/messages.go - tea.Msg types (JobsLoadedMsg, StatusMsg, TickMsg, etc.)
- model/styles.go - NewJobListDelegate(), JobListTitleStyle(), SpinnerStyle()
- model/keys.go - KeyMap struct, DefaultKeys() function

Modified files:
- model/state.go - reduced from 226 to ~130 lines
  - Removed: Job, JobStatus, KeyMap, Keys, inline styles
  - Kept: State struct, domain re-exports, ViewMode, DatasetInfo, InitialState()
- controller/commands.go - use model. prefix for message types
- controller/controller.go - use model. prefix for message types
- controller/settings.go - use model.SettingsContentMsg

Deleted files:
- controller/keys.go (moved to model/keys.go since State references KeyMap)

Result:
- No file >150 lines in model/ package
- Single concern per file: state, jobs, messages, styles, keys
- All 41 test packages pass
 2026-02-17 20:22:04 -05:00
Jeremie Fraeys
6b771e4a50 feat(jupyter): improve runtime management and update security/workflow docs 2026-01-05 12:37:27 -05:00
Jeremie Fraeys
cd5640ebd2 Slim and secure: move scripts, clean configs, remove secrets 
- Move ci-test.sh and setup.sh to scripts/
- Trim docs/src/zig-cli.md to current structure
- Replace hardcoded secrets with placeholders in configs
- Update .gitignore to block .env*, secrets/, keys, build artifacts
- Slim README.md to reflect current CLI/TUI split
- Add cleanup trap to ci-test.sh
- Ensure no secrets are committed
 2025-12-07 13:57:51 -05:00
Jeremie Fraeys
ea15af1833 Fix multi-user authentication and clean up debug code 
- Fix YAML tags in auth config struct (json -> yaml)
- Update CLI configs to use pre-hashed API keys
- Remove double hashing in WebSocket client
- Fix port mapping (9102 -> 9103) in CLI commands
- Update permission keys to use jobs:read, jobs:create, etc.
- Clean up all debug logging from CLI and server
- All user roles now authenticate correctly:
  * Admin: Can queue jobs and see all jobs
  * Researcher: Can queue jobs and see own jobs
  * Analyst: Can see status (read-only access)

Multi-user authentication is now fully functional.
 2025-12-06 12:35:32 -05:00
Jeremie Fraeys
803677be57 feat: implement Go backend with comprehensive API and internal packages 
- Add API server with WebSocket support and REST endpoints
- Implement authentication system with API keys and permissions
- Add task queue system with Redis backend and error handling
- Include storage layer with database migrations and schemas
- Add comprehensive logging, metrics, and telemetry
- Implement security middleware and network utilities
- Add experiment management and container orchestration
- Include configuration management with smart defaults
 2025-12-04 16:53:53 -05:00