Dockerfile targets systems without GPUs:
- Add -DBUILD_NVML_GPU=OFF to cmake in simple.Dockerfile
- Add BUILD_NVML_GPU option to native/CMakeLists.txt (default ON)
- Conditionally include nvml_gpu subdirectory
- Update all_native_libs target to exclude nvml_gpu when disabled
This allows native libraries (dataset_hash, queue_index) to build
without requiring NVIDIA drivers/libraries.
- Add stub implementation in nvml_gpu.cpp when NVML not available
- CMakeLists.txt checks for both NVML library and headers
- Build succeeds on macOS/non-NVIDIA systems with stub
- Runtime detection via gpu_is_available() prevents runtime errors
- Add native/nvml_gpu/ C++ library wrapping NVIDIA Management Library
- Add Go bindings in internal/worker/gpu_nvml_native.go and gpu_nvml_stub.go
- Update gpu_detector.go to use NVML for accurate GPU count detection
- Update native/CMakeLists.txt to build nvml_gpu library
- Provides real-time GPU utilization, memory, temperature, clocks, power
- Falls back to environment variable when NVML unavailable
Replace FETCHML_NATIVE_LIBS=1 environment variable with -tags native_libs:
Changes:
- internal/queue/native_queue.go: UseNativeQueue is now const true
- internal/queue/native_queue_stub.go: UseNativeQueue is now const false
- build/docker/simple.Dockerfile: Add -tags native_libs to go build
- deployments/docker-compose.dev.yml: Remove FETCHML_NATIVE_LIBS env var
- native/README.md: Update documentation for build tags
- scripts/test-native-with-redis.sh: New test script with Redis via docker-compose
Benefits:
- Compile-time enforcement (no runtime checks needed)
- Cleaner deployment (no env var management)
- Type safety (const vs var)
- Simpler testing with docker-compose Redis integration
Add comprehensive explanation of the reproducibility problem and fix:
- Document readdir filesystem-dependent ordering issue
- Explain std::sort fix for lexicographic ordering
- Clarify recursive traversal with cycle detection
- Document hidden file and special file exclusions
- Warn researchers about silent omissions and empty hash edge cases
This addresses the core concern that researchers need to understand
the hash is computed over sorted paths to trust cross-machine verification.
