What LLMs Run on NVIDIA L40S 48GB (48GB VRAM)?
NVIDIA L40S 48GB (48GB VRAM, 864 GB/s bandwidth, 362 TFLOPS FP16).
| 模型名称 | 总参数 (激活参数) | Q4_K_M (4位推荐) | FP16 无损显存 | Status on NVIDIA L40S 48GB | Decode / Prefill | Report |
|---|---|---|---|---|---|---|
| DeepSeek-R1 671B (MoE) | 671B | 378.7 GB | 1250.87 GB | 显存不足 | ~3 / 2544 tok/s | Calculate → |
| DeepSeek-V3-0324 671B (MoE) | 671B | 378.7 GB | 1250.87 GB | 显存不足 | ~3 / 2544 tok/s | Calculate → |
| DeepSeek-R1-Distill-Llama-70B | 70.55B | 43.09 GB | 134.41 GB | Fits Q4_K_M | ~13.9 / 1334 tok/s | Calculate → |
| DeepSeek-R1-Distill-Qwen-32B | 32.76B | 21.23 GB | 63.51 GB | Fits Q4_K_M | ~30 / 2873 tok/s | Calculate → |
| DeepSeek-R1-Distill-Qwen-14B | 14.77B | 10.61 GB | 29.5 GB | FP16 & Q4 | ~66.6 / 6372 tok/s | Calculate → |
| DeepSeek-R1-Distill-Llama-8B | 8.03B | 6.22 GB | 16.45 GB | FP16 & Q4 | ~122.5 / 11721 tok/s | Calculate → |
| DeepSeek-R1-Distill-Qwen-7B | 7.62B | 5.43 GB | 15.12 GB | FP16 & Q4 | ~129.1 / 12352 tok/s | Calculate → |
| Qwen3-235B-A22B (MoE) | 235.1B | 134.4 GB | 439.87 GB | 显存不足 | ~6.5 / 4240 tok/s | Calculate → |
| Qwen3-30B-A3B (MoE) | 30.53B | 18.53 GB | 58.1 GB | Fits Q4_K_M | ~294.4 / 18000 tok/s | Calculate → |
| Qwen3-32B | 32.76B | 21.22 GB | 63.51 GB | Fits Q4_K_M | ~30 / 2873 tok/s | Calculate → |
| Qwen3-14B | 14.77B | 10.36 GB | 29.25 GB | FP16 & Q4 | ~66.6 / 6372 tok/s | Calculate → |
| Qwen3-8B | 8.19B | 6.47 GB | 16.87 GB | FP16 & Q4 | ~120.1 / 11492 tok/s | Calculate → |
| Qwen3-4B | 4.02B | 3.61 GB | 8.68 GB | FP16 & Q4 | ~244.6 / 18000 tok/s | Calculate → |
| Qwen2.5-72B-Instruct | 72.71B | 44.38 GB | 138.43 GB | Fits Q4_K_M | ~13.5 / 1294 tok/s | Calculate → |
| Qwen2.5-Coder-32B-Instruct | 32.76B | 21.23 GB | 63.51 GB | Fits Q4_K_M | ~30 / 2873 tok/s | Calculate → |
| Qwen2.5-14B-Instruct | 14.77B | 10.61 GB | 29.5 GB | FP16 & Q4 | ~66.6 / 6372 tok/s | Calculate → |
| Qwen2.5-7B-Instruct | 7.62B | 5.43 GB | 15.12 GB | FP16 & Q4 | ~129.1 / 12352 tok/s | Calculate → |
| QwQ-32B Reasoning | 32.76B | 21.23 GB | 63.51 GB | Fits Q4_K_M | ~30 / 2873 tok/s | Calculate → |
| Llama-4-Maverick-17B-128E (400B MoE) | 400B | 227.32 GB | 747.05 GB | 显存不足 | ~7 / 5472 tok/s | Calculate → |
| Llama-4-Scout-17B-16E (109B MoE) | 109B | 63.7 GB | 205.02 GB | 显存不足 | ~17.7 / 5472 tok/s | Calculate → |
| Llama-3.3-70B-Instruct | 70.55B | 43.09 GB | 134.41 GB | Fits Q4_K_M | ~13.9 / 1334 tok/s | Calculate → |
| Llama-3.1-405B-Instruct | 405B | 233.02 GB | 758.82 GB | 显存不足 | ~1.2 / 232 tok/s | Calculate → |
| Llama-3.1-70B-Instruct | 70.55B | 43.09 GB | 134.41 GB | Fits Q4_K_M | ~13.9 / 1334 tok/s | Calculate → |
| Llama-3.1-8B-Instruct | 8.03B | 6.22 GB | 16.45 GB | FP16 & Q4 | ~122.5 / 11721 tok/s | Calculate → |
| Llama-3.2-3B-Instruct | 3.21B | 3.33 GB | 7.34 GB | FP16 & Q4 | ~306.4 / 18000 tok/s | Calculate → |
| Mistral-Large-2-123B | 122.6B | 72.84 GB | 231.62 GB | 显存不足 | ~2 / 768 tok/s | Calculate → |
| Mistral-Small-3.1-24B-Instruct | 23.6B | 15.28 GB | 45.7 GB | FP16 & Q4 | ~41.7 / 3988 tok/s | Calculate → |
| Mixtral-8x22B-Instruct (MoE) | 140.6B | 81.38 GB | 264.13 GB | 显存不足 | ~5.3 / 2407 tok/s | Calculate → |
| Mixtral-8x7B-Instruct (MoE) | 46.7B | 27.8 GB | 88.47 GB | Fits Q4_K_M | ~76.2 / 7296 tok/s | Calculate → |
| Codestral-22B | 22.2B | 14.81 GB | 43.59 GB | FP16 & Q4 | ~44.3 / 4240 tok/s | Calculate → |
| Google Gemma-3-27B-IT | 27.4B | 20.34 GB | 55.4 GB | Fits Q4_K_M | ~35.9 / 3435 tok/s | Calculate → |
| Google Gemma-3-12B-IT | 12.2B | 10.75 GB | 26.21 GB | FP16 & Q4 | ~80.6 / 7715 tok/s | Calculate → |
| Google Gemma-3-4B-IT | 4.3B | 4.24 GB | 9.56 GB | FP16 & Q4 | ~228.7 / 18000 tok/s | Calculate → |
| Microsoft Phi-4 14B | 14.66B | 10.5 GB | 29.36 GB | FP16 & Q4 | ~67.1 / 6420 tok/s | Calculate → |
| Cohere Command-A 111B | 111B | 66.19 GB | 209.26 GB | 显存不足 | ~2.5 / 848 tok/s | Calculate → |
| GLM-4-32B-0414 | 32B | 20.77 GB | 62 GB | Fits Q4_K_M | ~30.7 / 2941 tok/s | Calculate → |