# ADR-003: Use Redis for Job Queue

## Status
Accepted

## Context
For the ML experiment job queue system, we needed a solution that:
- Provides reliable job queuing and distribution
- Supports multiple workers consuming jobs concurrently
- Offers persistence and durability
- Handles job priorities and retries
- Integrates well with our Go-based API server
- Can scale horizontally with multiple workers

## Decision
We chose Redis as the job queue backend using its list data structures and pub/sub capabilities.

## Consequences

### Positive
- Excellent performance with sub-millisecond latency
- Built-in persistence options (AOF, RDB)
- Simple and reliable queue operations (LPUSH/RPOP)
- Good Go client library support
- Supports job priorities through multiple lists
- Easy to monitor and debug
- Can handle high throughput workloads
- Low memory overhead for queue operations

### Negative
- Additional infrastructure component to manage
- Memory-based (requires sufficient RAM)
- Limited built-in job scheduling features
- No complex job dependency management
- Requires careful handling of connection failures

## Options Considered

### Database-based Queuing (PostgreSQL)
**Pros:**
- No additional infrastructure
- ACID transactions
- Complex queries and joins possible
- Integrated with primary database

**Cons:**
- Higher latency for queue operations
- Database contention under high load
- More complex implementation for reliable polling
- Limited scalability for high-frequency operations

### RabbitMQ
**Pros:**
- Purpose-built message broker
- Advanced routing and filtering
- Built-in acknowledgments and retries
- Good clustering support

**Cons:**
- More complex setup and configuration
- Higher resource requirements
- Steeper learning curve
- Overkill for simple queue needs

### Apache Kafka
**Pros:**
- Extremely high throughput
- Built-in partitioning and replication
- Good for event streaming

**Cons:**
- Complex setup and operations
- Designed for streaming, not job queuing
- Higher latency for individual job processing
- More resource intensive

### In-memory Queuing (Go channels)
**Pros:**
- Zero external dependencies
- Very fast
- Simple implementation

**Cons:**
- No persistence (jobs lost on restart)
- Limited to single process
- No monitoring or observability
- Not suitable for distributed systems

## Rationale

Redis provides the optimal balance of simplicity, performance, and reliability for our job queue needs. The list-based queue implementation (LPUSH/RPOP) is straightforward and highly performant. Redis's persistence options ensure jobs aren't lost during restarts, and the pub/sub capabilities enable real-time notifications for workers.

The Go client library is excellent and provides connection pooling, automatic reconnection, and good error handling. Redis's low memory footprint and fast operations make it ideal for high-frequency job queuing scenarios common in ML workloads.

While RabbitMQ offers more advanced features, Redis is sufficient for our current needs and much simpler to operate. The simple queue model also makes it easier to understand and debug when issues arise.
