TL;DR: The 30-Second Triage
Before you spend hours installing drivers, check your hardware against this matrix.
Less than 12GB VRAM: You will likely hit OOM or run at single-digit tokens/sec. Solution: Cloud GPU
12GB to 24GB VRAM: Usable for quantized 7B-14B models. Full-size reasoning models (30B+) will require heavy quantization.
24GB or more VRAM: You are in the green zone for local development.
📉 The Reality of Local LLMs
Running OpenClaw locally isn't just about CPU speed. The bottleneck is almost always Memory Bandwidth and VRAM Capacity.
1. The VRAM Bottleneck
Modern reasoning models (like the DeepSeek R1 family or Llama-3 variants) are memory-hungry.
| Precision | VRAM per 1B Parameters |
|---|---|
| FP16 (Full) | ~2 GB |
| 4-bit Quantized | ~0.7 GB |
Reality: A consumer card with 8GB VRAM simply cannot load a 30B+ parameter model, no matter how fast your CPU is.
2. The Speed Trade-off
Even if you fit the model into system RAM (CPU offloading), inference speed drops drastically.
| Offloading Method | Speed | Usability |
|---|---|---|
| GPU Offloading | Real-time interaction | Interactive, usable |
| CPU/System RAM | 0.5-5 t/s | Painfully slow, often unusable |
🛠️ Hardware Tiers
Tier C: Consumer Laptops (Integrated Graphics)
Typical Specs: 4-8GB unified memory, shared with CPU
Experience:
- High latency, low throughput
- Suitable for testing API connectivity or very small models (TinyLlama, <1B params)
- Not recommended for daily use with reasoning models
Verdict: API or cloud only
Tier B: Gaming Desktops (12GB to 16GB VRAM)
Typical Specs: RTX 3060/3070/4060/4070, 8GB to 12GB VRAM
Experience:
- Capable of running quantized 7B/8B models comfortably
- Larger reasoning models (30B+) will OOM or require extreme quantization
- May need to reduce context window (num_ctx < 4096)
Limitations:
- Multi-turn conversations may slow as context fills
- Model switching requires VRAM management
Verdict: Good for learning, limiting for production workflows
Tier A: Workstation / Cloud (24GB or more VRAM)
Typical Specs: RTX 3090/4090/5090, Apple Silicon (32GB+ Unified), or Cloud H100/A100
Experience:
- Full access to larger models (30B-70B ranges) with usable speeds
- Can run multiple models simultaneously
- Sufficient VRAM for full context windows (8k+ tokens)
Verdict: Required for serious local development
💡 Decision: Upgrade or Rent?
If your local hardware falls into Tier C or Tier B, you have a decision to make.
Option 1: The Cloud Route (Immediate)
If you need to run large models now without buying new hardware.
Pros:
- Instant access to H100/A100 class GPUs
- Pay only for uptime
- No hardware management
Cons:
- Not offline
- Recurring cost
New users may qualify for promotional credit. Terms apply.
Option 2: The Local Route (Long-term)
If you plan to run models 24/7 and value privacy above all.
Pros:
- Total data sovereignty
- One-time cost
Cons:
- High upfront investment (GPU, Power Supply, Cooling)
- Electricity costs
Hardware Recommendations:
- NVIDIA RTX 4090 (24GB VRAM) - Best consumer option
- NVIDIA RTX 3090 (24GB VRAM) - Good value on used market
- Apple Mac Studio (64GB+ Unified Memory) - Best Mac option
🙋 FAQ
Why is my inference so slow (Painfully slow (seconds per word))?
A: You are likely offloading layers to your CPU/System RAM because your GPU VRAM is full. Check your num_gpu or n_gpu_layers settings. Reduce model size, increase VRAM, or switch to GPU offloading.
Can I run DeepSeek R1 on my Mac?
A: Yes, if you have an M-series chip with sufficient Unified Memory (32GB+ recommended for decent quantization). Apple Silicon uses unified memory, so all available RAM can be used for model inference. However, memory bandwidth is still a bottleneck compared to discrete GPUs.
Does OpenClaw support multi-GPU?
A: OpenClaw relies on the underlying inference engine (e.g., Ollama/Llama.cpp). Multi-GPU support depends on their specific configuration. Check the inference engine documentation for multi-GPU setup instructions.
What if I have 8GB VRAM but want to run 30B models?
A: You have two options: 1) Use extreme quantization (2-bit or less) which degrades model quality significantly, or 2) Offload to CPU/system RAM which will be painfully slow (0.5-2 t/s). For 30B+ models, the practical solution is cloud GPU with 24GB+ VRAM.
Is unified memory (Mac) better than discrete VRAM?
A: Unified memory (Apple Silicon) offers flexibility but has lower bandwidth than discrete GPU VRAM (100-400 GB/s vs 500-1000 GB/s). For large models, discrete GPUs with high-bandwidth memory (HBM) will significantly outperform unified memory systems.
How much VRAM do I need for 70B models?
A: At 4-bit quantization, ~49GB VRAM. At 8-bit, ~98GB VRAM. Current consumer cards top out at 24GB (RTX 3090/4090), so 70B models require cloud GPUs (H100/A100 with 80GB+ VRAM) or multi-GPU setups.
Related Articles
-
Fix OpenClaw CUDA OOM Errors - VRAM optimization guide
-
Torch CUDA Out of Memory - PyTorch-specific OOM fix
-
Fix OpenClaw Slow Inference - Bandwidth explained
-
OpenClaw Error Index - Master error dictionary
Bottom Line: Hardware physics doesn't negotiate.
Check your VRAM against model requirements before investing time in setup.