diff --git a/.gitignore b/.gitignore index 7d65994..7b9b1ec 100644 --- a/.gitignore +++ b/.gitignore @@ -209,7 +209,7 @@ secrets/ cli/src/assets/rsync_release.bin # Test files -# test_*.go +test_*.go *_test_output/ # Build artifacts diff --git a/README.md b/README.md index 282293f..98ed063 100644 --- a/README.md +++ b/README.md @@ -107,14 +107,17 @@ make dev # Fast dev build make prod # Optimized production build ``` -### Test +### Testing ```bash make test # All tests make test-unit # Unit tests only make test-coverage # With coverage report +make test-auth # Multi-user authentication tests ``` +**Quick Start Testing**: See **[Testing Guide](docs/src/testing.md)** for comprehensive testing documentation, including a 5-minute quick start guide. + ## Configuration ### Development (`configs/config-dev.yaml`) @@ -155,10 +158,14 @@ Access: `http://localhost:3000` (dev) or `http://YOUR_SERVER:3000` (prod) ## Documentation -- **[Getting Started](docs/getting-started.md)** - Detailed setup guide -- **[Production Deployment](docs/production-monitoring.md)** - Linux deployment -- **[WebSocket API](docs/api/)** - Protocol documentation -- **[Architecture](docs/architecture/)** - System design +- **[Testing Guide](docs/src/testing.md)** - Comprehensive testing documentation +- **[Quick Start Testing](docs/src/quick-start-testing.md)** - 5-minute testing guide +- **[Installation](docs/src/installation.md)** - Setup instructions +- **[Architecture](docs/src/architecture.md)** - System design +- **[Configuration Reference](docs/src/configuration-reference.md)** - Configuration options +- **[CLI Reference](docs/src/cli-reference.md)** - Command-line interface +- **[Deployment](docs/src/deployment.md)** - Production deployment +- **[Troubleshooting](docs/src/troubleshooting.md)** - Common issues ## Makefile Targets diff --git a/docs/src/testing.md b/docs/src/testing.md index 8895eff..76a8bb5 100644 --- a/docs/src/testing.md +++ b/docs/src/testing.md @@ -1,46 +1,129 @@ # Testing Guide -How to run and write tests for FetchML. +Comprehensive testing documentation for FetchML platform. -## Running Tests +## Quick Start Testing -### Quick Test +For a fast 5-minute testing experience, see the **[Quick Start Testing Guide](quick-start-testing.md)**. +## Test Types + +### Unit Tests ```bash -# All tests -make test +make test-unit # Go unit tests only +cd cli && zig build test # Zig CLI tests +``` -# Unit tests only -make test-unit - -# Integration tests -make test-integration - -# With coverage -make test-coverage - - -## Quick Test +### Integration Tests ```bash -make test # All tests -make test-unit # Unit only -. -make test. -make test$ -make test; make test # Coverage - # E2E tests +make test-integration # API and database integration +``` + +### End-to-End Tests +```bash +make test-e2e # Full workflow testing +``` + +### All Tests +```bash +make test # Run complete test suite +make test-coverage # With coverage report ``` ## Docker Testing + +### Development Environment ```bash -docker-compose up -d (testing only) +docker-compose up -d make test docker-compose down ``` +### Production Environment Testing +```bash +docker-compose -f docker-compose.prod.yml up -d +make test-auth # Multi-user auth test +make self-cleanup # Clean up after testing +``` + +## Performance Testing + +### Benchmark Suite +```bash +./scripts/benchmarks/run-benchmarks-local.sh +``` + +### Load Testing +```bash +make test-load # API load testing +``` + +## Authentication Testing + +Multi-user authentication testing is fully covered in the **[Quick Start Testing Guide](quick-start-testing.md)**. + +```bash +make test-auth # Quick auth role testing +``` + ## CLI Testing + +### Build and Test CLI ```bash cd cli && zig build dev ./cli/zig-out/dev/ml --help zig build test -``` +``` + +### CLI Integration Tests +```bash +make test-cli # CLI-specific integration tests +``` + +## Troubleshooting Tests + +### Common Issues +- **Server not running**: Check with `docker ps --filter "name=ml-"` +- **Authentication failures**: Verify configs in `~/.ml/config-*.toml` +- **Connection issues**: Test API with `curl -I http://localhost:9103/health` + +### Debug Mode +```bash +make test-debug # Run tests with verbose output +``` + +## Test Configuration + +### Test Configs Location +- `~/.ml/config-admin.toml` - Admin user +- `~/.ml/config-researcher.toml` - Researcher user +- `~/.ml/config-analyst.toml` - Analyst user + +### Test Data +- `tests/fixtures/` - Test data and examples +- `tests/benchmarks/` - Performance test data + +## Continuous Integration + +Tests run automatically on: +- Pull requests (full suite) +- Main branch commits (unit + integration) +- Releases (full suite + benchmarks) + +## Writing Tests + +### Go Tests +- Unit tests: `*_test.go` files +- Integration tests: `tests/e2e/` directory +- Benchmark tests: `tests/benchmarks/` directory + +### Zig Tests +- CLI tests: `cli/tests/` directory +- Follow Zig testing conventions + +## See Also + +- **[Quick Start Testing Guide](quick-start-testing.md)** - Fast 5-minute testing +- **[Testing Protocol](testing-protocol.md)** - Detailed testing procedures +- **[Configuration Reference](configuration-reference.md)** - Test setup +- **[Troubleshooting](troubleshooting.md)** - Common issues diff --git a/tests/fixtures/examples/pytorch_project/README.md b/tests/fixtures/examples/pytorch_project/README.md deleted file mode 100644 index 02057b1..0000000 --- a/tests/fixtures/examples/pytorch_project/README.md +++ /dev/null @@ -1,11 +0,0 @@ -# PyTorch Experiment - -Neural network classification project using PyTorch. - -## Usage -```bash -python train.py --epochs 10 --batch_size 32 --learning_rate 0.001 --hidden_size 64 --output_dir ./results -``` - -## Results -Results are saved in JSON format with training metrics and PyTorch model checkpoint. diff --git a/tests/fixtures/examples/sklearn_project/README.md b/tests/fixtures/examples/sklearn_project/README.md deleted file mode 100644 index 36b353f..0000000 --- a/tests/fixtures/examples/sklearn_project/README.md +++ /dev/null @@ -1,11 +0,0 @@ -# Scikit-learn Experiment - -Random Forest classification project using scikit-learn. - -## Usage -```bash -python train.py --n_estimators 100 --output_dir ./results -``` - -## Results -Results are saved in JSON format with accuracy and model metrics. diff --git a/tests/fixtures/examples/standard_ml_project/README.md b/tests/fixtures/examples/standard_ml_project/README.md deleted file mode 100644 index 77fca96..0000000 --- a/tests/fixtures/examples/standard_ml_project/README.md +++ /dev/null @@ -1,11 +0,0 @@ -# Standard ML Experiment - -Minimal PyTorch neural network classification experiment. - -## Usage -```bash -python train.py --epochs 5 --batch_size 32 --learning_rate 0.001 --output_dir ./results -``` - -## Results -Results are saved in JSON format with training metrics and PyTorch model checkpoint. diff --git a/tests/fixtures/examples/statsmodels_project/README.md b/tests/fixtures/examples/statsmodels_project/README.md deleted file mode 100644 index 1d45b6a..0000000 --- a/tests/fixtures/examples/statsmodels_project/README.md +++ /dev/null @@ -1,11 +0,0 @@ -# Statsmodels Experiment - -Linear regression experiment using statsmodels for statistical analysis. - -## Usage -```bash -python train.py --output_dir ./results -``` - -## Results -Results are saved in JSON format with statistical metrics and model summary. diff --git a/tests/fixtures/examples/tensorflow_project/README.md b/tests/fixtures/examples/tensorflow_project/README.md deleted file mode 100644 index e6e0f2d..0000000 --- a/tests/fixtures/examples/tensorflow_project/README.md +++ /dev/null @@ -1,11 +0,0 @@ -# TensorFlow Experiment - -Deep learning experiment using TensorFlow/Keras for classification. - -## Usage -```bash -python train.py --epochs 10 --batch_size 32 --learning_rate 0.001 --output_dir ./results -``` - -## Results -Results are saved in JSON format with training metrics and TensorFlow SavedModel. diff --git a/tests/fixtures/examples/xgboost_project/README.md b/tests/fixtures/examples/xgboost_project/README.md deleted file mode 100644 index fe9e773..0000000 --- a/tests/fixtures/examples/xgboost_project/README.md +++ /dev/null @@ -1,11 +0,0 @@ -# XGBoost Experiment - -Gradient boosting experiment using XGBoost for binary classification. - -## Usage -```bash -python train.py --n_estimators 100 --max_depth 6 --learning_rate 0.1 --output_dir ./results -``` - -## Results -Results are saved in JSON format with accuracy metrics and XGBoost model file. diff --git a/tests/fixtures/podman/workspace/pytorch_project/README.md b/tests/fixtures/podman/workspace/pytorch_project/README.md deleted file mode 100644 index 02057b1..0000000 --- a/tests/fixtures/podman/workspace/pytorch_project/README.md +++ /dev/null @@ -1,11 +0,0 @@ -# PyTorch Experiment - -Neural network classification project using PyTorch. - -## Usage -```bash -python train.py --epochs 10 --batch_size 32 --learning_rate 0.001 --hidden_size 64 --output_dir ./results -``` - -## Results -Results are saved in JSON format with training metrics and PyTorch model checkpoint. diff --git a/tests/fixtures/podman/workspace/sklearn_project/README.md b/tests/fixtures/podman/workspace/sklearn_project/README.md deleted file mode 100644 index 36b353f..0000000 --- a/tests/fixtures/podman/workspace/sklearn_project/README.md +++ /dev/null @@ -1,11 +0,0 @@ -# Scikit-learn Experiment - -Random Forest classification project using scikit-learn. - -## Usage -```bash -python train.py --n_estimators 100 --output_dir ./results -``` - -## Results -Results are saved in JSON format with accuracy and model metrics. diff --git a/tests/fixtures/podman/workspace/standard_ml_project/README.md b/tests/fixtures/podman/workspace/standard_ml_project/README.md deleted file mode 100644 index 77fca96..0000000 --- a/tests/fixtures/podman/workspace/standard_ml_project/README.md +++ /dev/null @@ -1,11 +0,0 @@ -# Standard ML Experiment - -Minimal PyTorch neural network classification experiment. - -## Usage -```bash -python train.py --epochs 5 --batch_size 32 --learning_rate 0.001 --output_dir ./results -``` - -## Results -Results are saved in JSON format with training metrics and PyTorch model checkpoint. diff --git a/tests/fixtures/podman/workspace/statsmodels_project/README.md b/tests/fixtures/podman/workspace/statsmodels_project/README.md deleted file mode 100644 index 1d45b6a..0000000 --- a/tests/fixtures/podman/workspace/statsmodels_project/README.md +++ /dev/null @@ -1,11 +0,0 @@ -# Statsmodels Experiment - -Linear regression experiment using statsmodels for statistical analysis. - -## Usage -```bash -python train.py --output_dir ./results -``` - -## Results -Results are saved in JSON format with statistical metrics and model summary. diff --git a/tests/fixtures/podman/workspace/tensorflow_project/README.md b/tests/fixtures/podman/workspace/tensorflow_project/README.md deleted file mode 100644 index e6e0f2d..0000000 --- a/tests/fixtures/podman/workspace/tensorflow_project/README.md +++ /dev/null @@ -1,11 +0,0 @@ -# TensorFlow Experiment - -Deep learning experiment using TensorFlow/Keras for classification. - -## Usage -```bash -python train.py --epochs 10 --batch_size 32 --learning_rate 0.001 --output_dir ./results -``` - -## Results -Results are saved in JSON format with training metrics and TensorFlow SavedModel. diff --git a/tests/fixtures/podman/workspace/xgboost_project/README.md b/tests/fixtures/podman/workspace/xgboost_project/README.md deleted file mode 100644 index fe9e773..0000000 --- a/tests/fixtures/podman/workspace/xgboost_project/README.md +++ /dev/null @@ -1,11 +0,0 @@ -# XGBoost Experiment - -Gradient boosting experiment using XGBoost for binary classification. - -## Usage -```bash -python train.py --n_estimators 100 --max_depth 6 --learning_rate 0.1 --output_dir ./results -``` - -## Results -Results are saved in JSON format with accuracy metrics and XGBoost model file.