Can AMD Instinct MI300X 192GB (192GB) run Mixtral-8x22B-Instruct (MoE)?

AMD Instinct MI300X 192GB (192GB) fits Mixtral-8x22B-Instruct (MoE) smoothly in FP8 (W8A8 / E4M3). Unquantized FP16 (264.58 GB) exceeds 192 GB, but 4-bit Q4_K_M requires 81.38 GB (42% VRAM utilization) at ~154.3 tokens/sec.

How much VRAM does Mixtral-8x22B-Instruct (MoE) require in Q4_K_M, FP8, and FP16?

At 8,192 context length, Mixtral-8x22B-Instruct (MoE) requires 81.38 GB in GGUF Q4_K_M, 137.74 GB in FP8, and 264.58 GB in FP16.

✅ PASST IN DEN VRAM (FP8 (W8A8 / E4M3))

Mixtral-8x22B-Instruct (MoE) on AMD Instinct MI300X 192GB (192GB VRAM)

Unquantized FP16 (264.58 GB) exceeds 192 GB, but 4-bit Q4_K_M requires 81.38 GB (42% VRAM utilization) at ~154.3 tokens/sec.

🎛️ Interaktiver VRAM-, Kontext- & Offload-Konfigurator

Quantisierungsformat (Reale Bits pro Gewicht)

Inferenz-Engine

Kontextfenster (Tokens) 8,192 tokens

Gleichzeitige Batch-Größe 1

KV-Cache Präzision

Anzahl der GPUs (Tensor-Parallelität)

CPU-RAM Offload (% der Layer auf CPU) 0% (100% GPU)

📊 Echtzeit-Speicher- & Performance-Telemetrie

81.38 GB / 192 GB

🟦 Modellgewichte: 79.14 GB 🟪 KV-Cache: 1.75 GB 🟨 Engine-Puffer: 0.49 GB

Benötigter GPU-VRAM81.38 GB

Benötigter System-RAM (CPU)0.00 GB

Verfügbarer Hardware-VRAM192 GB (1x AMD Instinct MI300X 192GB)

VRAM-Reserve / Defizit110.62 GB

Geschätzte Decode-Geschwindigkeit~154.3 tokens/sec

Geschätzte Prefill-Geschwindigkeit~8691 tokens/sec

Time-to-First-Token (4K Prompt)~471 ms

Transformer-TopologieMoE-GQA (56L, 8/48 KV/Q heads)

🚀 Startbefehl (vLLM / llama.cpp / Ollama)

ollama run mixtral:8x22b
# Or llama.cpp server (8K context, Flash Attention):
llama-server -hf mistralai/Mixtral-8x22B-Instruct-v0.1-GGUF -c 8192 -ngl 99 -fa

CLOUD-GPU PREISVERGLEICH & MIETEN-VS-KAUFEN RECHNER

Nicht genug lokaler VRAM? Cloud-GPU in 60 Sekunden starten (Mixtral-8x22B-Instruct (MoE): 81.38 GB in 4-bit)

Recommended Cloud Tier: Multi-GPU H100 / H200 Node · Full FP16 Needs: 264.58 GB

Geschätzte Nutzungsstunden pro Tag: 4 hrs/day

Amortisationszeit (Kauf vs. Miete): ~32 months (vs $19,500 hardware)

Cloud Provider	RTX 4090 (24GB)	A100 (80GB)	H100 (80GB)	Est. Monthly Cost (4h/day)	Free Credit / Trial	Deploy
RunPod	$0.44 / hr	$1.39 / hr	$2.39 / hr	$52.80 / mo	Instant PyTorch / vLLM Templates + Serverless GPU	Auf RunPod starten →
Vast.ai	$0.29 / hr	$0.95 / hr	$1.79 / hr	$34.80 / mo	Save 50–70% on RTX 4090 / 5090 / A100 rentals	Spot-GPU mieten →
Vultr Cloud GPU	—	$1.79 / hr	$2.69 / hr	$214.80 / mo (A100)	$300 Free Credit	$300 Guthaben sichern →
DigitalOcean GPU	—	—	$2.99 / hr	$358.80 / mo (H100)	$200 Free Credit (60d)	$200 Startguthaben →

Alle Quantisierungsstufen im Detail: Mixtral-8x22B-Instruct (MoE) × AMD Instinct MI300X 192GB

Quantization Format	Core / Emb BPW	Weights (GiB)	KV Cache (8K)	Total VRAM	1x AMD Instinct MI300X 192GB (192GB)	Decode / Prefill	Min GPUs
FP16 / BF16 (Unquantized)	16 / 16	261.89 GB	1.75 GB	264.13 GB	Nein (+72.1GB)	~2.7 / 8691 tok/s	2x
FP8 (W8A8 / E4M3)	8.25 / 8.5	135.05 GB	1.75 GB	137.29 GB	Ja (72%)	~90.3 / 8691 tok/s	1x
GGUF Q8_0 (8.50 bpw)	8.5 / 8.5	139.13 GB	1.75 GB	141.37 GB	Ja (74%)	~87.7 / 8691 tok/s	1x
GGUF Q6_K (6.56 bpw)	6.56 / 6.56	107.37 GB	1.75 GB	109.62 GB	Ja (57%)	~113.6 / 8691 tok/s	1x
GGUF Q5_K_M (5.69 bpw)	5.69 / 6.56	93.17 GB	1.75 GB	95.42 GB	Ja (50%)	~131 / 8691 tok/s	1x
GGUF Q4_K_M / AWQ (4.83 bpw)	4.83 / 6.56	79.14 GB	1.75 GB	81.38 GB	Ja (42%)	~154.3 / 8691 tok/s	1x
GGUF IQ4_XS (4.25 bpw)	4.25 / 5.5	69.62 GB	1.75 GB	71.87 GB	Ja (37%)	~175.3 / 8691 tok/s	1x
GGUF Q3_K_M (3.91 bpw)	3.91 / 5	64.05 GB	1.75 GB	66.29 GB	Ja (35%)	~190.6 / 8691 tok/s	1x

Dieses Modell auf anderen GPUs testen

Weitere Modelle auf dieser GPU