What LLMs Can Run on NVIDIA RTX A6000 (48GB VRAM)?
NVIDIA RTX A6000 features 48 GB VRAM and 768 GB/s memory bandwidth (Workstation GPU). Below is the exact compatibility and token/sec speed table across all 36 open-source LLMs.
| LLM Model | Params | Q4_K_M VRAM (8K) | FP16 VRAM (8K) | Status on NVIDIA RTX A6000 | Est. Speed | Exact Config |
|---|---|---|---|---|---|---|
| DeepSeek-R1 671B (MoE) | 671B | 420.2 GB | 1383.46 GB | Exceeds 48GB | ~2.7 tok/s | Calculate → |
| DeepSeek-V3 671B (MoE) | 671B | 420.2 GB | 1383.46 GB | Exceeds 48GB | ~2.7 tok/s | Calculate → |
| DeepSeek-R1-Distill-Llama-70B | 70.6B | 47.25 GB | 148.6 GB | Fits 4-bit Q4_K_M | ~11.1 tok/s | Calculate → |
| DeepSeek-R1-Distill-Qwen-32B | 32.8B | 23.14 GB | 70.23 GB | Fits 4-bit Q4_K_M | ~23.9 tok/s | Calculate → |
| DeepSeek-R1-Distill-Qwen-14B | 14.8B | 11.4 GB | 32.65 GB | Fits FP16 & Q4 | ~53.1 tok/s | Calculate → |
| DeepSeek-R1-Distill-Llama-8B | 8.03B | 6.67 GB | 18.2 GB | Fits FP16 & Q4 | ~97.8 tok/s | Calculate → |
| DeepSeek-R1-Distill-Qwen-7B | 7.62B | 5.85 GB | 16.79 GB | Fits FP16 & Q4 | ~103.1 tok/s | Calculate → |
| Qwen3-235B-A22B (MoE) | 235B | 148.87 GB | 486.23 GB | Exceeds 48GB | ~4.6 tok/s | Calculate → |
| Qwen3-30B-A3B (MoE) | 30.5B | 20.45 GB | 64.23 GB | Fits 4-bit Q4_K_M | ~238 tok/s | Calculate → |
| Qwen3-32B | 32.8B | 23.14 GB | 70.23 GB | Fits 4-bit Q4_K_M | ~23.9 tok/s | Calculate → |
| Qwen3-14B | 14.8B | 11.15 GB | 32.4 GB | Fits FP16 & Q4 | ~53.1 tok/s | Calculate → |
| Qwen3-8B | 8.2B | 6.9 GB | 18.67 GB | Fits FP16 & Q4 | ~95.8 tok/s | Calculate → |
| Qwen2.5-72B-Instruct | 72.7B | 48.56 GB | 152.93 GB | Exceeds 48GB | ~1.4 tok/s | Calculate → |
| Qwen2.5-Coder-32B-Instruct | 32.8B | 23.14 GB | 70.23 GB | Fits 4-bit Q4_K_M | ~23.9 tok/s | Calculate → |
| Qwen2.5-14B-Instruct | 14.8B | 11.4 GB | 32.65 GB | Fits FP16 & Q4 | ~53.1 tok/s | Calculate → |
| Qwen2.5-7B-Instruct | 7.62B | 5.85 GB | 16.79 GB | Fits FP16 & Q4 | ~103.1 tok/s | Calculate → |
| QwQ-32B Reasoning | 32.8B | 23.14 GB | 70.23 GB | Fits 4-bit Q4_K_M | ~23.9 tok/s | Calculate → |
| Llama-4-Maverick-17B-128E (400B MoE) | 400B | 251.93 GB | 826.16 GB | Exceeds 48GB | ~6 tok/s | Calculate → |
| Llama-4-Scout-17B-16E (109B MoE) | 109B | 70.22 GB | 226.7 GB | Exceeds 48GB | ~6 tok/s | Calculate → |
| Llama-3.3-70B-Instruct | 70.6B | 47.25 GB | 148.6 GB | Fits 4-bit Q4_K_M | ~11.1 tok/s | Calculate → |
| Llama-3.1-405B-Instruct | 405B | 257.51 GB | 838.92 GB | Exceeds 48GB | ~1.2 tok/s | Calculate → |
| Llama-3.1-70B-Instruct | 70.6B | 47.25 GB | 148.6 GB | Fits 4-bit Q4_K_M | ~11.1 tok/s | Calculate → |
| Llama-3.1-8B-Instruct | 8.03B | 6.67 GB | 18.2 GB | Fits FP16 & Q4 | ~97.8 tok/s | Calculate → |
| Llama-3.2-3B-Instruct | 3.21B | 3.54 GB | 8.14 GB | Fits FP16 & Q4 | ~244.7 tok/s | Calculate → |
| Mistral-Large-2-123B | 123B | 80.23 GB | 256.8 GB | Exceeds 48GB | ~1.2 tok/s | Calculate → |
| Mistral-Small-3.1-24B-Instruct | 24B | 16.9 GB | 51.35 GB | Fits 4-bit Q4_K_M | ~32.7 tok/s | Calculate → |
| Mixtral-8x22B-Instruct (MoE) | 141B | 90.46 GB | 292.87 GB | Exceeds 48GB | ~2.6 tok/s | Calculate → |
| Mixtral-8x7B-Instruct (MoE) | 46.7B | 30.82 GB | 97.86 GB | Fits 4-bit Q4_K_M | ~60.9 tok/s | Calculate → |
| Codestral-22B | 22.2B | 16.27 GB | 48.14 GB | Fits 4-bit Q4_K_M | ~35.4 tok/s | Calculate → |
| Google Gemma-3-27B-IT | 27.4B | 21.64 GB | 60.98 GB | Fits 4-bit Q4_K_M | ~28.7 tok/s | Calculate → |
| Google Gemma-3-12B-IT | 12.2B | 11.28 GB | 28.79 GB | Fits FP16 & Q4 | ~64.4 tok/s | Calculate → |
| Google Gemma-3-4B-IT | 4.3B | 4.4 GB | 10.58 GB | Fits FP16 & Q4 | ~182.7 tok/s | Calculate → |
| Microsoft Phi-4 14B | 14.7B | 11.4 GB | 32.5 GB | Fits FP16 & Q4 | ~53.4 tok/s | Calculate → |
| Cohere Command-A 111B | 111B | 71.99 GB | 231.33 GB | Exceeds 48GB | ~1.2 tok/s | Calculate → |
| Yi-1.5-34B-Chat | 34.4B | 24.02 GB | 73.4 GB | Fits 4-bit Q4_K_M | ~22.8 tok/s | Calculate → |
| GLM-4-32B-0414 | 32B | 22.55 GB | 68.49 GB | Fits 4-bit Q4_K_M | ~24.5 tok/s | Calculate → |