LLMs on NPU¶
End-to-end decoder-only LLM inference (prefill + autoregressive decode) mapped to the AMD NPU2 in bf16 via MLIR-AIR. Coverage and performance below are refreshed nightly. Per-model details and source are in programming_examples/llms/.
Nightly LLM Benchmark (NPU2)¶
End-to-end LLM inference performance on the AMD Ryzen AI 5 PRO 340 (Krackan Point, NPU2) benchmark runner — 2×32 GB DDR5-5600 SODIMM — refreshed nightly. TTFT is time to first token (prefill latency); Decode is steady-state generation throughput.
| Model | Context | TTFT (ms) | Decode (tok/s) | Measured | Verify |
|---|---|---|---|---|---|
| llama32_1b (HF) | 2048 | 1130.0 | 11.1 | 2026-08-28 | 🟢 |
| llama32_1b_int4 (HF) | 2048 | 1090.0 | 16.31 | 2026-08-28 | 🟢 |
| llama32_3b (HF) | 2048 | 4990.0 | 5.13 | 2026-08-28 | 🟢 |
| qwen25_0_5b (HF) | 2048 | 860.0 | 12.65 | 2026-08-28 | 🟢 |
| qwen25_1_5b (HF) | 2048 | 2220.0 | 7.78 | 2026-08-28 | 🟢 |
| qwen25_3b (HF) | 2048 | 4130.0 | 4.72 | 2026-08-28 | 🟢 |
| qwen3_0_6b (HF) | 2048 | 1460.0 | 10.55 | 2026-08-28 | 🟢 |
| qwen3_1_7b (HF) | 2048 | 2070.0 | 7.12 | 2026-08-28 | 🟢 |
| qwen3_4b (HF) | 2048 | 5300.0 | 3.93 | 2026-08-28 | 🟢 |
| smollm2_1_7b (HF) | 2048 | 1770.0 | 7.44 | 2026-08-28 | 🟢 |
| smolvla | 241 | 786.4 | — | 2026-08-28 | 🟢 |
Verify: 🟢 pass 🔴 fail ⚪ skipped. — not measured.
Decode throughput vs context (tok/s)¶
Steady-state decode throughput at increasing KV-cache depth.
The models below reimplement AMD NPU LLM designs originally developed by the FastFlowLM team, using the higher-level abstractions of the MLIR-AIR dialect.
| Model | 1k | 2k | 4k | 8k | 16k | 32k | 64k | 128k | Verify |
|---|---|---|---|---|---|---|---|---|---|
| gemma3_4b_q4nx (HF) | 20.73 | 19.34 | 17.07 | 13.85 | 10.05 | 6.49 | 3.80 | 2.08 | 🟢 |
| lfm2_1_2b_q4nx (HF) | 67.48 | 62.19 | 53.78 | 42.33 | 29.75 | 18.65 | 10.68 | 5.76 | 🔴 |
| llama31_8b_q4nx (HF) | 13.14 | 12.49 | 11.34 | 9.58 | 7.33 | 4.98 | 3.04 | 1.71 | 🔴 |
| llama32_1b_q4nx (HF) | 68.98 | 63.09 | 54.80 | 42.79 | 29.96 | 18.73 | 10.71 | 5.77 | 🔴 |
| llama32_3b_q4nx (HF) | 28.00 | 25.62 | 21.82 | 16.89 | 11.63 | 7.18 | 4.06 | 2.18 | 🟢 |
| phi4_mini_q4nx (HF) | 23.44 | 21.50 | 18.47 | 14.40 | 10.00 | 6.21 | 3.53 | 1.90 | 🟢 |
| qwen25_3b_q4 (HF) | 27.15 | 24.13 | 19.77 | 14.52 | 9.48 | 5.60 | 3.08 | 1.62 | 🔴 |
| qwen25_7b_q4nx (HF) | 13.30 | 12.72 | 11.69 | 10.07 | 7.88 | 5.50 | 3.43 | 1.96 | 🔴 |
| qwen3_4b_q4nx (HF) | 20.76 | 19.01 | 16.23 | 12.57 | 8.66 | 5.34 | 3.02 | 1.62 | 🔴 |
| qwen3_8b_q4nx (HF) | 12.65 | 11.94 | 10.76 | 9.00 | 6.79 | 4.56 | 2.75 | 1.53 | 🟢 |
Prefill latency vs padded prefill length (TTFT, ms)¶
Time to first token at increasing padded prefill length. Each column is a separate build of the prefill ELFs: the prompt is padded to the built length, so TTFT tracks that length rather than the number of real prompt tokens.
| Model | 512 | 1k | 2k | 4k | Verify |
|---|---|---|---|---|---|
| gemma3_4b_q4nx (HF) | 1039 | 1846 | 3674 | 7635 | 🟢 |
| lfm2_1_2b_q4nx (HF) | ✗ | ✗ | ✗ | ✗ | 🔴 |
| llama31_8b_q4nx (HF) | 1356 | 2485 | 5955 | 17702 | 🔴 |
| llama32_1b_q4nx (HF) | 291 | 498 | 905 | 1870 | 🔴 |
| llama32_3b_q4nx (HF) | 644 | 1170 | 2291 | 6139 | 🟢 |
| phi4_mini_q4nx (HF) | 745 | 1314 | 2625 | 7012 | 🟢 |
| qwen25_7b_q4nx (HF) | 2055 | 4001 | 8064 | 17861 | 🔴 |
| qwen3_8b_q4nx (HF) | 1853 | 3484 | 7112 | 15480 | 🟢 |
✗ unexpected failure.
📈 Performance history over time — per-nightly TTFT and decode throughput plotted per model.
Last updated 2026-08-28 · mlir-air 6b9792a · mlir-aie a2f8daf · llvm-aie 21.0.0.2026080601+f4a72c27