CUDA Out of Memory - DeepSeek R1 32B on 24GB GPU
TL;DR
The error "CUDA out of memory" on a 24GB GPU with DeepSeek R1 32B indicates VRAM exhaustion during inference.
This is almost always caused by insufficient memory headroom for the 32B model variant.
This is a physical memory ceiling, not a configuration bug.
Symptoms
When running DeepSeek R1 32B on 24GB VRAM GPUs (RTX 3090, RTX 4090, A10), you may encounter:
CUDA out of memory
RuntimeError: CUDA error: out of memory
torch.OutOfMemoryError: CUDA out of memory
Tried to allocate 2.00 GiB (GPU 0; 24.00 GiB total capacity; 22.50 GiB already allocated)
Inference fails during token generation. Memory usage approaches 100% before crash. The error may occur mid-conversation rather than at startup.
Memory Requirements
DeepSeek R1 32B VRAM requirements by quantization:
| Quantization | Model Size | Minimum VRAM | Stable VRAM |
|---|---|---|---|
| 4-bit (Q4_K_M) | ~18 GB | 22 GB | 24 GB+ |
| 4-bit (Q4_0) | ~19 GB | 24 GB | 32 GB+ |
| 5-bit (Q5_K_M) | ~22 GB | 28 GB | 32 GB+ |
| 8-bit | ~35 GB | 40 GB | 48 GB+ |
24GB VRAM provides minimal headroom for 4-bit quantization only.
Why It Happens
Three factors cause OOM errors with DeepSeek R1 32B on 24GB GPUs:
-
DeepSeek R1 32B requires significant VRAM for model weights. 4-bit quantization consumes ~18-19 GB, leaving 5-6 GB for inference overhead.
-
KV cache expands with context length. Each token adds to memory allocation. A 4K context consumes ~3-4 GB additional VRAM.
-
Memory fragmentation and runtime overhead trigger OOM when available memory drops below threshold. CUDA allocation fails when contiguous memory blocks are insufficient.
Fast Check
Verify your hardware limits before adjusting configuration.
This diagnostic confirms whether your GPU meets stable requirements for the 32B model.
Hard Limits
The following table shows stable VRAM requirements for DeepSeek R1 variants:
| Model | Stable VRAM Requirement |
|---|---|
| 8B | 12–16 GB |
| 32B | 24 GB+ (32 GB recommended) |
If your GPU has 24GB VRAM, 32B model operation is marginal. This failure is expected behavior under load.
24GB GPU Reality
RTX 3090 / RTX 4090 / A10 constraints:
| Resource | Available | 32B Requirement | Status |
|---|---|---|---|
| Total VRAM | 24 GB | 24+ GB | Marginal |
| After model load | ~5 GB | ~6+ GB | Insufficient |
| Context headroom | 2-4 GB | 4+ GB | Limited |
24GB is the absolute minimum for 32B inference. Any additional load causes failure.
Local Mitigation (Hard Way)
If running 32B on 24GB VRAM, these steps may enable marginal operation:
1. Reduce context window.
Limit context to 2K-4K tokens maximum.
OLLAMA_NUM_CTX=2048 ollama run deepseek-r1:32b
2. Lower batch size.
Process single requests only.
# In vLLM
--max-num-seqs 1 --gpu-memory-utilization 0.95
3. Switch to smaller quantization.
Use Q3_K_M or Q3_K_S if available.
ollama run deepseek-r1:32b-q3_K_M
GPU Memory Configuration
Aggressive Memory Allocation
# PyTorch
export PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:128
Pre-allocate Memory
# vLLM
--gpu-memory-utilization 0.98
Monitor Memory
watch -n 0.5 'nvidia-smi --query-gpu=memory.used,memory.free --format=csv,noheader,nounits'
Alternative: Use 8B Model
The 8B model runs comfortably on 24GB VRAM:
| Metric | 32B on 24GB | 8B on 24GB |
|---|---|---|
| Model memory | 19 GB | 6 GB |
| KV cache available | 4 GB | 17 GB |
| Max context | 4K tokens | 32K+ tokens |
| Stability | Marginal | Stable |
If 32B OOM persists, the 8B variant provides reliable operation.
Hardware Verdict
When CUDA OOM occurs on 24GB VRAM with 32B model:
Required VRAM (stable): 32 GB Detected VRAM: 24 GB
Conclusion: No configuration change can bypass physical memory limits.
24GB supports marginal 32B operation only. Stable inference requires 32GB+ VRAM.
Escape Option
For reliable 32B model inference:
Run DeepSeek R1 on a pre-configured 24GB+ GPU instanceHourly billing. No long-term commitment.
Cloud GPU instances provide:
- A100 40GB/80GB options
- Stable 32B inference
- Full context support
- No memory constraints