Commit graph

2 commits

Author SHA1 Message Date
Jeremie Fraeys
37aad7ae87
feat: add manifest signing and native hashing support
- Integrate RunManifest.Validate with existing Validator
- Add manifest Sign() and Verify() methods
- Add native C++ hashing libraries (dataset_hash, queue_index)
- Add native bridge for Go/C++ integration
- Add deduplication support in queue
2026-02-19 15:34:39 -05:00
Jeremie Fraeys
3248279c01
refactor: Phase 3 - Extract data integrity layer
Created integrity package with extracted data utilities:

1. internal/worker/integrity/hash.go (113 lines)
   - FileSHA256Hex() - SHA256 hash of single file
   - NormalizeSHA256ChecksumHex() - Checksum normalization
   - DirOverallSHA256Hex() - Directory hash (sequential)
   - DirOverallSHA256HexParallel() - Directory hash (parallel workers)

2. internal/worker/integrity/validate.go (76 lines)
   - DatasetVerifier type for dataset validation
   - VerifyDatasetSpecs() method for checksum validation
   - ProvenanceCalculator type for provenance computation
   - ComputeProvenance() method for task provenance

Note: Used 'integrity' instead of 'data' due to .gitignore conflict
(data/ directory is ignored for experiment artifacts)

Functions extracted from data_integrity.go:
- fileSHA256Hex → FileSHA256Hex
- normalizeSHA256ChecksumHex → NormalizeSHA256ChecksumHex
- dirOverallSHA256HexGo → DirOverallSHA256Hex
- dirOverallSHA256HexParallel → DirOverallSHA256HexParallel
- verifyDatasetSpecs logic → DatasetVerifier
- computeTaskProvenance logic → ProvenanceCalculator

Build status: Compiles successfully
2026-02-17 14:20:41 -05:00