What LLMs Run on NVIDIA RTX 3090 (24GB VRAM)?
NVIDIA RTX 3090 (24GB VRAM, 936 GB/s bandwidth, 71 TFLOPS FP16).
| 模型名称 | 总参数 (激活参数) | Q4_K_M (4位推荐) | FP16 无损显存 | Status on NVIDIA RTX 3090 | Decode / Prefill | Report |
|---|---|---|---|---|---|---|
| DeepSeek-R1 671B (MoE) | 671B | 378.7 GB | 1250.87 GB | 显存不足 | ~2.8 / 499 tok/s | Calculate → |
| DeepSeek-V3-0324 671B (MoE) | 671B | 378.7 GB | 1250.87 GB | 显存不足 | ~2.8 / 499 tok/s | Calculate → |
| DeepSeek-R1-Distill-Llama-70B | 70.55B | 43.09 GB | 134.41 GB | 显存不足 | ~2.8 / 262 tok/s | Calculate → |
| DeepSeek-R1-Distill-Qwen-32B | 32.76B | 21.23 GB | 63.51 GB | Fits Q4_K_M | ~32.5 / 563 tok/s | Calculate → |
| DeepSeek-R1-Distill-Qwen-14B | 14.77B | 10.61 GB | 29.5 GB | Fits Q4_K_M | ~72.1 / 1250 tok/s | Calculate → |
| DeepSeek-R1-Distill-Llama-8B | 8.03B | 6.22 GB | 16.45 GB | FP16 & Q4 | ~132.7 / 2299 tok/s | Calculate → |
| DeepSeek-R1-Distill-Qwen-7B | 7.62B | 5.43 GB | 15.12 GB | FP16 & Q4 | ~139.8 / 2423 tok/s | Calculate → |
| Qwen3-235B-A22B (MoE) | 235.1B | 134.4 GB | 439.87 GB | 显存不足 | ~5.3 / 832 tok/s | Calculate → |
| Qwen3-30B-A3B (MoE) | 30.53B | 18.53 GB | 58.1 GB | Fits Q4_K_M | ~319 / 5527 tok/s | Calculate → |
| Qwen3-32B | 32.76B | 21.22 GB | 63.51 GB | Fits Q4_K_M | ~32.5 / 563 tok/s | Calculate → |
| Qwen3-14B | 14.77B | 10.36 GB | 29.25 GB | Fits Q4_K_M | ~72.1 / 1250 tok/s | Calculate → |
| Qwen3-8B | 8.19B | 6.47 GB | 16.87 GB | FP16 & Q4 | ~130.1 / 2254 tok/s | Calculate → |
| Qwen3-4B | 4.02B | 3.61 GB | 8.68 GB | FP16 & Q4 | ~265 / 4592 tok/s | Calculate → |
| Qwen2.5-72B-Instruct | 72.71B | 44.38 GB | 138.43 GB | 显存不足 | ~2.6 / 254 tok/s | Calculate → |
| Qwen2.5-Coder-32B-Instruct | 32.76B | 21.23 GB | 63.51 GB | Fits Q4_K_M | ~32.5 / 563 tok/s | Calculate → |
| Qwen2.5-14B-Instruct | 14.77B | 10.61 GB | 29.5 GB | Fits Q4_K_M | ~72.1 / 1250 tok/s | Calculate → |
| Qwen2.5-7B-Instruct | 7.62B | 5.43 GB | 15.12 GB | FP16 & Q4 | ~139.8 / 2423 tok/s | Calculate → |
| QwQ-32B Reasoning | 32.76B | 21.23 GB | 63.51 GB | Fits Q4_K_M | ~32.5 / 563 tok/s | Calculate → |
| Llama-4-Maverick-17B-128E (400B MoE) | 400B | 227.32 GB | 747.05 GB | 显存不足 | ~6.3 / 1073 tok/s | Calculate → |
| Llama-4-Scout-17B-16E (109B MoE) | 109B | 63.7 GB | 205.02 GB | 显存不足 | ~8.6 / 1073 tok/s | Calculate → |
| Llama-3.3-70B-Instruct | 70.55B | 43.09 GB | 134.41 GB | 显存不足 | ~2.8 / 262 tok/s | Calculate → |
| Llama-3.1-405B-Instruct | 405B | 233.02 GB | 758.82 GB | 显存不足 | ~1.2 / 46 tok/s | Calculate → |
| Llama-3.1-70B-Instruct | 70.55B | 43.09 GB | 134.41 GB | 显存不足 | ~2.8 / 262 tok/s | Calculate → |
| Llama-3.1-8B-Instruct | 8.03B | 6.22 GB | 16.45 GB | FP16 & Q4 | ~132.7 / 2299 tok/s | Calculate → |
| Llama-3.2-3B-Instruct | 3.21B | 3.33 GB | 7.34 GB | FP16 & Q4 | ~331.9 / 5751 tok/s | Calculate → |
| Mistral-Large-2-123B | 122.6B | 72.84 GB | 231.62 GB | 显存不足 | ~1.2 / 151 tok/s | Calculate → |
| Mistral-Small-3.1-24B-Instruct | 23.6B | 15.28 GB | 45.7 GB | Fits Q4_K_M | ~45.1 / 782 tok/s | Calculate → |
| Mixtral-8x22B-Instruct (MoE) | 140.6B | 81.38 GB | 264.13 GB | 显存不足 | ~3.4 / 472 tok/s | Calculate → |
| Mixtral-8x7B-Instruct (MoE) | 46.7B | 27.8 GB | 88.47 GB | 显存不足 | ~35.1 / 1431 tok/s | Calculate → |
| Codestral-22B | 22.2B | 14.81 GB | 43.59 GB | Fits Q4_K_M | ~48 / 832 tok/s | Calculate → |
| Google Gemma-3-27B-IT | 27.4B | 20.34 GB | 55.4 GB | Fits Q4_K_M | ~38.9 / 674 tok/s | Calculate → |
| Google Gemma-3-12B-IT | 12.2B | 10.75 GB | 26.21 GB | Fits Q4_K_M | ~87.3 / 1513 tok/s | Calculate → |
| Google Gemma-3-4B-IT | 4.3B | 4.24 GB | 9.56 GB | FP16 & Q4 | ~247.8 / 4293 tok/s | Calculate → |
| Microsoft Phi-4 14B | 14.66B | 10.5 GB | 29.36 GB | Fits Q4_K_M | ~72.7 / 1259 tok/s | Calculate → |
| Cohere Command-A 111B | 111B | 66.19 GB | 209.26 GB | 显存不足 | ~1.3 / 166 tok/s | Calculate → |
| GLM-4-32B-0414 | 32B | 20.77 GB | 62 GB | Fits Q4_K_M | ~33.3 / 577 tok/s | Calculate → |