Benchmarks¶
gmlx serves the same GGUF files faster than llama.cpp at context depths from 512 to more than 200K tokens. Its prefill is faster on every model at every measured depth. Above a depth of about 4K tokens its decoding is faster too, and the gap grows as the context deepens.
Speculative decoding is measured where the model has a native head or a companion drafter. The measurements behind the performance pages' guidance are under Serving measurements and KV cache fidelity. The data behind the charts is in a JSON file, and the benchmark harness reproduces any cell.
Summary¶
The first chart shows gmlx's throughput divided by the reference engine's, for every model, against the context depth.
A second chart shows how much faster speculative decoding is than plain decoding on the same server, for each model, against the context depth.
Methodology¶
Every number in the per-model tables is server throughput. Both engines ran the same GGUF weights, sampler settings and chat prompts on the setup in this table. oMLX loads MLX checkpoints, so it ran an MLX checkpoint of the same weights. MTPLX ran its own checkpoint of the model, with the prompts and sampler settings listed under Qwen3.8-27B against MTPLX.
| Item | Setting |
|---|---|
| Hardware | Apple M5 Max with 128 GB of unified memory, in a MacBook Pro. |
| gmlx | 0.1.0 for most models. |
| mlx-kquant | 0.3.5 for most models, with the K-quant and performance kernels. |
| llama.cpp | b9967 |
| Build overrides | Models measured again on newer releases list their own builds under Model provenance. |
| DeepSeek-V4 reference | The dwarfstar ds4-server by antirez with the ignore-eos patch, at b030961 for DeepSeek-V4-Flash IQ2_XXS and 8db1d1d for DeepSeek-V4.1-Flash Q2. |
| oMLX reference | oMLX 0.7.2 at 3df94603 with its custom kernels built, its prefix caches off and forced-length decode. |
| MTPLX reference | MTPLX 2.12.2 with the ignore-eos patch, for forced-length decode. |
| Dates | 2026-07-05 to 2026-10-25. |
| Prompt corpus | HuggingFaceH4/ultrachat_200k:train_sft, with the chat template applied. |
| Sampling | Temperature 0.6, top-p 0.95, top-k 20 and seed 1234, with the same random sequence on both engines. |
| Speculative draft | Three draft tokens a round, two on Qwen3.8-Flash-Next, from the native MTP head or gemma-4's companion drafter. |
| Aggregation | Four requests a cell in two rounds that alternate the engines. Median, then lowest and highest sample in parentheses. |
| Thermal protocol | A cooldown to 50 C or below between engines, 20 seconds of baseline cooldown and one warmup request. |
| Decode metric | Median decode tokens per second over samples of at least 150 output tokens. |
| Prefill metric | Median prefill tokens per second over all successful samples. |
MTP@N in the tables means speculative decoding with N draft tokens in each round on both engines. The baseline column is the same server with it off.
Model provenance¶
Chart labels give the base model, even for an abliterated community build, so each row names the GGUF file behind a label for reproduction. The Builds column names the gmlx and mlx-kquant builds that each model's rows were measured on. Models are measured again independently, so a newer build on one row does not apply to the others. The Measured column gives the date of the newest run still contributing cells to the row, because a partial rerun replaces an older depth series one cell at a time.
| Model | GGUF file | Source | MTP | Builds | Measured |
|---|---|---|---|---|---|
| Qwen3.5-122B-A10B UD-Q5_K_M | Qwen3.5-122B-A10B-UD-Q5_K_M-00001-of-00003.gguf |
HF | Native | gmlx 0.1.0 / kq 0.3.5 | 2026-07-18 |
| Qwen3.6-35B-A3B Q6_K | Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-Q6_K.gguf |
HF | Native | gmlx 0.1.0 / kq 0.3.5 | 2026-07-18 |
| Qwen3.6-27B Q6_K | Qwen_Qwen3.6-27B-Q6_K.gguf |
HF | Native | gmlx 0.1.0 / kq 0.3.5 | 2026-07-18 |
| Qwen3.5-9B Q6_K | Qwen3.5-9B-Q6_K.gguf |
HF | Native | gmlx 0.1.0 / kq 0.3.5 | 2026-07-18 |
| gemma-4-31B-it Q6_K | gemma-4-31B-it-Q6_K.gguf |
- | Drafter | gmlx 0.1.0 / kq 0.3.5 | 2026-07-18 |
| gemma-4-26B-A4B-it Q6_K | google_gemma-4-26B-A4B-it-Q6_K.gguf |
HF | Drafter | gmlx 0.1.0 / kq 0.3.5 | 2026-07-18 |
| gemma-4-12B-it Q6_K | gemma-4-12b-it-Q6_K.gguf |
- | Drafter | gmlx 0.1.0 / kq 0.3.5 | 2026-07-18 |
| gemma-4-E4B-it Q6_K | gemma-4-E4B-it-Q6_K.gguf |
- | - | gmlx 0.1.0 / kq 0.3.5 | 2026-07-18 |
| gemma-4-E2B-it UD-Q6_K_XL | gemma-4-E2B-it-UD-Q6_K_XL.gguf |
- | - | gmlx 0.1.0 / kq 0.3.5 | 2026-07-18 |
| gpt-oss-120b MXFP4 | gpt-oss-120b-heretic-v2-MXFP4.gguf |
HF | - | gmlx 0.1.0 / kq 0.3.5 | 2026-07-18 |
| gpt-oss-20b MXFP4 | gpt-oss-20b-mxfp4.gguf |
- | - | gmlx 0.1.0 / kq 0.3.5 | 2026-07-18 |
| Dolphin3.0-Llama3.1-8B Q6_K | Dolphin3.0-Llama3.1-8B-abliterated.Q6_K.gguf |
HF | - | gmlx 0.1.0 / kq 0.3.5 | 2026-07-18 |
| DeepSeek-V4-Flash UD-IQ3_XXS | DeepSeek-V4-Flash-UD-IQ3_XXS-00001-of-00004.gguf |
HF | - | gmlx 0.1.0 / kq 0.3.5 | 2026-07-13 |
| DeepSeek-V4-Flash IQ2_XXS | DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf |
HF | - | gmlx 0.2.2 / kq 0.3.11 | 2026-08-09 |
| DeepSeek-V4.1-Flash Q2 | DeepSeek-V4.1-Flash-Q2.gguf |
HF | - | gmlx 0.4.13+ds41 / kq 0.4.11 | 2026-09-18 |
| Flint INT4 Q4_1 | flint-int4-q4_1.gguf |
HF | - | gmlx 0.5.1+flint-perf / kq 0.5.6 | 2026-10-24 |
| Flint INT4 | flint-int4-q4_1.gguf |
HF | - | gmlx 0.5.1+flint-perf / kq 0.5.6 / oMLX 0.7.2 | 2026-10-24 |
| Qwen3.8-27B 4-bit | Qwen3.8-27B-Q4_0.gguf |
HF | Native, drafter | gmlx 0.5.1+#225 / kq 0.5.6 / MTPLX 2.12.2 | 2026-10-25 |
| Qwen3.8-Flash-Next UD-Q3_K_XL | Qwen3.8-Flash-Next-UD-Q3_K_XL-00001-of-00003.gguf |
HF | Native | gmlx 0.4.5+#98 / kq 0.4.3 | 2026-08-29 |
Per-model detail¶
Qwen3.5-122B-A10B UD-Q5_K_M¶
| KV depth | gmlx decode (baseline) | gmlx decode (MTP@3) | MTP lift | llama.cpp decode (MTP@3) | gmlx/llama.cpp decode | gmlx prefill | llama.cpp prefill |
|---|---|---|---|---|---|---|---|
| 512 | 47.8 (47.4-48) | 59 (55.6-67.3) | 1.23x | 47.8 (42.3-51) | 1.23x | 643.9 (482.2-706.7) | 568.8 (546.6-612.4) |
| 4.3k | 45.9 (45.7-46.2) | 58.3 (55.6-62.8) | 1.27x | 46.4 (44.7-48) | 1.26x | 950.3 (935.3-956.4) | 555 (496.5-607.6) |
| 17k | 42.2 (42.1-42.4) | 56 (46.6-61.2) | 1.33x | 38.6 (37.4-40.7) | 1.45x | 957.6 (945.2-968.5) | 476.3 (449.9-491.7) |
| 67k | 33.6 (32.8-33.9) | 42 (38.5-44.7) | 1.25x | 25.4 (24.4-31.2) | 1.65x | 649.5 (628-654.5) | 284.8 (276-289.7) |
| 110k | 28.4 (27.9-28.8) | 36.7 (34.4-38.2) | 1.29x | 21.9 (21.1-24.7) | 1.68x | 474.2 (463.2-478.4) | 202.2 (201.2-207.3) |
| 200k | 21.9 (21.8-22) | 22 (20.6-23.8) | 1.00x | 14.4 (13.9-15.9) | 1.53x | 300.3 (297.5-301.3) | 129.9 (129.3-130) |
Qwen3.6-35B-A3B Q6_K¶
| KV depth | gmlx decode (baseline) | gmlx decode (MTP@3) | MTP lift | llama.cpp decode (MTP@3) | gmlx/llama.cpp decode | gmlx prefill | llama.cpp prefill |
|---|---|---|---|---|---|---|---|
| 512 | 128.8 (126.7-130) | 138.1 (131.4-143.1) | 1.07x | 110.1 (99.9-115.3) | 1.25x | 2106.6 (1554.9-2319.9) | 1737.9 (1166.7-1756.3) |
| 4.3k | 128.1 (127.5-128.7) | 139 (123.1-151.3) | 1.09x | 104 (96.9-111.1) | 1.34x | 2764.5 (2636.6-2849.5) | 1722.3 (1478.8-1823.7) |
| 17k | 114.3 (113.9-114.5) | 118.8 (117.1-130.5) | 1.04x | 87.4 (83.4-103.3) | 1.36x | 2736.4 (2685.6-2761.8) | 1440.5 (1403.8-1459.5) |
| 67k | 83.4 (82.8-84.7) | 85 (80.7-89.8) | 1.02x | 63.2 (52.4-70.1) | 1.34x | 1841.4 (1834.8-1849.8) | 739 (727.1-798.2) |
| 110k | 68.5 (68.4-69.1) | 72.6 (66-80.3) | 1.06x | 48.6 (40.9-50.8) | 1.49x | 1364.7 (1338.3-1374.4) | 537.6 (517-561.6) |
| 200k | 46.8 (46.5-48.2) | 48.3 (46.2-50.7) | 1.03x | 31.3 (27.2-36.3) | 1.54x | 805.1 (785-835) | 321.1 (315.8-324.3) |
Qwen3.6-27B Q6_K¶
| KV depth | gmlx decode (baseline) | gmlx decode (MTP@3) | MTP lift | llama.cpp decode (MTP@3) | gmlx/llama.cpp decode | gmlx prefill | llama.cpp prefill |
|---|---|---|---|---|---|---|---|
| 512 | 20.7 (20.7-20.8) | 36.8 (35.1-39.9) | 1.78x | 31.5 (31.1-32.6) | 1.17x | 430.2 (407.4-489.8) | 394.6 (335.7-446.8) |
| 4.3k | 19.8 (19.7-20.1) | 37.1 (33.9-47) | 1.87x | 28.7 (27.5-56.4) | 1.29x | 600.9 (596-606.2) | 443.9 (412.5-464.4) |
| 17k | 18.8 (18.7-18.8) | 34.1 (29.5-39.3) | 1.81x | 25.6 (24.1-29.8) | 1.33x | 575.9 (570.1-587.6) | 405.4 (399.7-425.8) |
| 67k | 15.8 (15.5-16.2) | 24.6 (21.8-31.3) | 1.56x | 18.8 (17.6-21.6) | 1.31x | 454.5 (436.6-466.1) | 252.1 (247.9-255.7) |
| 110k | 14.5 (14.5-14.6) | 22.2 (18.5-23.8) | 1.53x | 16.9 (13.1-18.5) | 1.31x | 387.7 (384.6-389.3) | 182.6 (160.3-185) |
| 200k | 12.1 (11.7-12.4) | 16.3 (15.3-17.1) | 1.35x | 13.5 (11.2-14.8) | 1.21x | 280.3 (273.1-285.8) | 112.8 (112.5-114.5) |
Qwen3.5-9B Q6_K¶
| KV depth | gmlx decode (baseline) | gmlx decode (MTP@3) | MTP lift | llama.cpp decode (MTP@3) | gmlx/llama.cpp decode | gmlx prefill | llama.cpp prefill |
|---|---|---|---|---|---|---|---|
| 512 | 70.3 (69.1-72.4) | 111.9 (102.9-126.7) | 1.59x | 75.8 (72.8-80.9) | 1.48x | 1595 (1407.5-2267.2) | 1136.4 (1075.4-1142.4) |
| 4.3k | 69.1 (69-69.6) | 96.6 (85.7-103) | 1.40x | 76.8 (69.5-86.7) | 1.26x | 1817.4 (1786.1-1916.4) | 1424 (1342-1582) |
| 17k | 62.7 (62-63.5) | 85.1 (76.7-91.9) | 1.36x | 68.4 (61.3-73.6) | 1.24x | 1962.4 (1920-1997.6) | 1350.5 (1306-1375.9) |
| 67k | 49.8 (49.3-50.7) | 58.5 (52.5-64.1) | 1.17x | 49.4 (43.9-59.4) | 1.18x | 1551.7 (1540.2-1600.1) | 758.7 (740.7-798.9) |
gemma-4-31B-it Q6_K¶
| KV depth | gmlx decode (baseline) | gmlx decode (MTP@3) | MTP lift | llama.cpp decode (MTP@3) | gmlx/llama.cpp decode | gmlx prefill | llama.cpp prefill |
|---|---|---|---|---|---|---|---|
| 512 | 17.8 (17.6-18) | 34.5 (32.5-37.5) | 1.94x | 28.4 (25.1-28.8) | 1.21x | 367.5 (360.3-384.2) | 281.1 (261-283.8) |
| 4.3k | 16.1 (15.2-17) | 30 (26.5-31.7) | 1.86x | 24.8 (21.5-27.1) | 1.21x | 461.8 (439-490.4) | 341.7 (317.1-354.5) |
| 17k | 15.2 (13.9-16.1) | 29.1 (27.4-34.4) | 1.91x | 21.5 (18.6-25.6) | 1.35x | 439 (407.9-463.6) | 292.4 (284.9-295.8) |
| 67k | 12.4 (11.1-13.2) | 23.6 (19.9-25.5) | 1.90x | 14.1 (13.2-18.2) | 1.67x | 343.1 (319.2-363.7) | 166 (163.9-166.7) |
| 110k | 11.3 (11.2-11.4) | 18.2 (17-19.6) | 1.61x | 10.3 (9.6-13.5) | 1.77x | 292.3 (290.5-293.3) | 118.7 (116.9-122.8) |
| 200k | 9 (9-9.1) | 12.4 (11.8-13.5) | 1.38x | 6.4 (6-8.2) | 1.94x | 204.2 (192.2-207.7) | 74.1 (72.1-75.4) |
gemma-4-26B-A4B-it Q6_K¶
| KV depth | gmlx decode (baseline) | gmlx decode (MTP@3) | MTP lift | llama.cpp decode (MTP@3) | gmlx/llama.cpp decode | gmlx prefill | llama.cpp prefill |
|---|---|---|---|---|---|---|---|
| 512 | 97.8 (96.9-98.8) | 113.9 (104.2-127.1) | 1.16x | 93 (90.4-93.5) | 1.22x | 1821.3 (1531.2-2125) | 1268.3 (1104.3-1480.1) |
| 4.3k | 97.2 (97.1-97.3) | 107.1 (98.1-117.7) | 1.10x | 96.3 (79.2-109.9) | 1.11x | 2610.2 (2514.4-2668.2) | 1549.2 (1400-1650.7) |
| 17k | 90.4 (90.1-90.9) | 95.9 (89.5-124.4) | 1.06x | 77.2 (65.6-97) | 1.24x | 2679.4 (2647.9-2731) | 1355.5 (1281.5-1413.2) |
| 67k | 68.6 (67.9-69.3) | 69.7 (65.7-75.5) | 1.02x | 40.2 (36.1-47.8) | 1.73x | 1992.6 (1964-2010.6) | 676.3 (622.2-722.3) |
| 110k | 57.9 (57.5-58.5) | 53.8 (49.9-70.2) | 0.93x | 32.8 (30-33.6) | 1.64x | 1558 (1541.4-1564.8) | 502 (491.3-533.6) |
gemma-4-12B-it Q6_K¶
| KV depth | gmlx decode (baseline) | gmlx decode (MTP@3) | MTP lift | llama.cpp decode (MTP@3) | gmlx/llama.cpp decode | gmlx prefill | llama.cpp prefill |
|---|---|---|---|---|---|---|---|
| 512 | 43.9 (43.3-44) | 71.9 (66.3-87.3) | 1.64x | 54 (51.4-55.3) | 1.33x | 854.4 (808.8-965.4) | 726.2 (589.4-773.2) |
| 4.3k | 42.9 (42.8-43.3) | 67.1 (62.9-76.9) | 1.56x | 57.5 (52.4-65.4) | 1.17x | 1208 (1185.8-1228.8) | 856.8 (772.5-899.7) |
| 17k | 40.2 (40-40.5) | 67.6 (63.8-77) | 1.68x | 52.7 (46.9-56.5) | 1.28x | 1219.7 (1209.2-1239.5) | 771.2 (761.8-811) |
| 67k | 34.4 (33.9-34.9) | 50.1 (47.1-66.6) | 1.46x | 27.5 (23.7-33.2) | 1.82x | 961.2 (944.2-976.6) | 431.9 (411.5-438) |
| 110k | 30.1 (30-30.7) | 46.2 (38.6-53.1) | 1.53x | 21.5 (17.9-22.8) | 2.15x | 821 (806.1-827.9) | 310.1 (306.7-326.3) |
gemma-4-E4B-it Q6_K¶
| KV depth | gmlx decode | llama.cpp decode | gmlx/llama.cpp decode | gmlx prefill | llama.cpp prefill | gmlx/llama.cpp prefill |
|---|---|---|---|---|---|---|
| 512 | 91.9 (91.2-92.1) | 92.6 (92.3-93.5) | 0.99x | 2317.7 (2223.7-2456.6) | 2168.9 (1974.8-2547.6) | 1.07x |
| 4.3k | 91.3 (90.2-91.8) | 89.9 (85.9-90.8) | 1.02x | 4564.5 (4367.5-4613.2) | 2984.5 (2769.6-3247.5) | 1.53x |
| 17k | 84.1 (82.9-85) | 75.5 (74.3-81.1) | 1.11x | 4810.2 (4662.7-4954) | 2221.4 (2200.4-2314.9) | 2.17x |
| 67k | 62.8 (61.1-63.5) | 57.1 (54.2-60.9) | 1.10x | 4095.1 (4031.9-4239.9) | 1119.9 (1084.2-1159.4) | 3.66x |
| 110k | 51.9 (51.2-52.2) | 45.7 (45.2-46) | 1.14x | 3326.1 (3266.9-3420.2) | 754.2 (717.8-783.8) | 4.41x |
gemma-4-E2B-it UD-Q6_K_XL¶
| KV depth | gmlx decode | llama.cpp decode | gmlx/llama.cpp decode | gmlx prefill | llama.cpp prefill | gmlx/llama.cpp prefill |
|---|---|---|---|---|---|---|
| 512 | 155.3 (155-156) | 145.1 (143.7-146) | 1.07x | 5217 (5037.3-5527) | 3273.8 (3080.6-3759.2) | 1.59x |
| 4.3k | 153.9 (153.7-163) | 140.1 (138.9-141.5) | 1.10x | 16620.1 (16156.5-17589) | 5212.7 (5061.9-5636.2) | 3.19x |
| 17k | 140.4 (139.6-141.8) | 123.4 (120.5-124.1) | 1.14x | 16188.3 (15211.5-19019.1) | 3469.4 (3412.9-3619.4) | 4.67x |
| 67k | 107.2 (106.6-107.7) | 84.6 (83.4-85) | 1.27x | 10403.4 (9910.2-11056.5) | 1460.4 (1417.8-1464.6) | 7.12x |
| 110k | 88.6 (84.6-90.2) | 65 (64.3-65.5) | 1.36x | 7706.8 (7288.2-8190.8) | 975.4 (966.9-983.2) | 7.90x |
gpt-oss-120b MXFP4¶
| KV depth | gmlx decode | llama.cpp decode | gmlx/llama.cpp decode | gmlx prefill | llama.cpp prefill | gmlx/llama.cpp prefill |
|---|---|---|---|---|---|---|
| 512 | 89.9 (87.6-91.2) | 83.7 (83.7-84.1) | 1.07x | 875.5 (764.8-1078.1) | 601.1 (581.9-622.7) | 1.46x |
| 4.3k | 88.5 (86.8-91.6) | 66.3 (65.6-69.6) | 1.33x | 1502.4 (1423.3-1558.5) | 827.5 (780.3-870.2) | 1.82x |
| 17k | 77.9 (76.7-78.5) | 59.2 (58.5-59.7) | 1.32x | 1658.1 (1579.1-1676.7) | 709.5 (683-737.3) | 2.34x |
| 67k | 57 (54-65.9) | 40.9 (39.6-41.4) | 1.39x | 1374.5 (1340.1-1386.8) | 391.8 (386.9-395.4) | 3.51x |
| 90k | 49.8 (48.7-51.3) | 35.9 | 1.39x | 1229 (1219.9-1241.3) | 315.2 (308.4-321.2) | 3.90x |
gpt-oss-20b MXFP4¶
| KV depth | gmlx decode | llama.cpp decode | gmlx/llama.cpp decode | gmlx prefill | llama.cpp prefill | gmlx/llama.cpp prefill |
|---|---|---|---|---|---|---|
| 512 | 128.4 (126.5-130.5) | 122 (116-122) | 1.05x | 2307.5 (2149.1-2689.5) | 1235.5 (1055.5-1666.8) | 1.87x |
| 4.3k | 133.4 (131.3-135.1) | 119.1 (112.9-121.6) | 1.12x | 3441.8 (3269.6-3525.8) | 1937.4 (1717.7-1973.9) | 1.78x |
| 17k | 121.7 (119.7-122.6) | 107 (104-107.8) | 1.14x | 3520.6 (3354.2-3607.7) | 1529.7 (1507.3-1553) | 2.30x |
| 67k | 88.2 (84.2-92.7) | 65.8 (64.6-68.6) | 1.34x | 2636.2 (2453-2655.5) | 662.8 (653.4-703.2) | 3.98x |
| 90k | 77.5 (74.8-82.8) | 56.4 (55.7-57.9) | 1.37x | 2285.5 (2258.9-2333.2) | 520.9 (517.3-547) | 4.39x |
Dolphin3.0-Llama3.1-8B Q6_K¶
| KV depth | gmlx decode | llama.cpp decode | gmlx/llama.cpp decode | gmlx prefill | llama.cpp prefill | gmlx/llama.cpp prefill |
|---|---|---|---|---|---|---|
| 512 | 80.2 (78.9-82.1) | 76.7 (75.3-78.7) | 1.05x | 2384.4 (2279.7-2462.6) | 2290.8 (2206.5-2481) | 1.04x |
| 4.3k | 71.3 (70.5-71.8) | 67.9 (65.3-70.2) | 1.05x | 2120.1 (1940.1-2236.3) | 1669.2 (1566.4-1810.8) | 1.27x |
| 17k | 53.4 (52.7-54.6) | 50.1 (48.9-50.4) | 1.07x | 1846.2 (1752.2-1902.3) | 980.1 (943.4-1006.4) | 1.88x |
| 67k | 28.7 (26.7-29.1) | 26.6 (23.3-27.4) | 1.08x | 1065.5 (1009.8-1087.3) | 326.4 (308.1-345.4) | 3.26x |
| 110k | 20.7 (19.4-21.9) | 17.4 (16.6-17.6) | 1.19x | 765.6 (742.2-776.3) | 205.9 (200.7-208.8) | 3.72x |
DeepSeek-V4-Flash UD-IQ3_XXS¶
| KV depth | gmlx decode | llama.cpp decode | gmlx/llama.cpp decode | gmlx prefill | llama.cpp prefill | gmlx/llama.cpp prefill |
|---|---|---|---|---|---|---|
| 512 | 34.9 (34.1-35) | 12.9 (12.7-13.1) | 2.71x | 377.9 (334.7-438.4) | 312.5 (280.6-336.8) | 1.21x |
| 4.3k | 32.5 (32.3-32.7) | 12.5 (12.4-12.7) | 2.60x | 572.3 (567.3-582.8) | 356.7 (342.2-370.7) | 1.60x |
| 17k | 31 (30.6-31.8) | 12.4 (12-12.5) | 2.50x | 614.1 (610-619.6) | 265.7 (248.9-279.8) | 2.31x |
| 67k | 30.1 (29.1-30.5) | 10.9 (10.6-10.9) | 2.76x | 597.4 (591-610.5) | 152.1 (142.2-157.1) | 3.93x |
| 110k | 28.3 (28-28.9) | 9.9 (9.7-9.9) | 2.86x | 543.8 (492.2-569.6) | 116.5 (112-118.7) | 4.67x |
| 200k | 24.9 (24.8-25.1) | - | - | 484.3 (450-493.2) | - | - |
| 384k | 22.3 (20.7-22.9) | - | - | 411.1 (397.2-431.8) | - | - |
Flint INT4 Q4_1¶
| KV depth | gmlx decode | llama.cpp decode | gmlx/llama.cpp decode | gmlx prefill | llama.cpp prefill | gmlx/llama.cpp prefill |
|---|---|---|---|---|---|---|
| 512 | 157.2 (150-163.4) | 120.2 (118.3-122.9) | 1.31x | 3227.1 (2711.2-3695.8) | 1743 (1517.9-2440) | 1.85x |
| 4.3k | 153.4 (151.8-156.3) | 116 (115.9-117) | 1.32x | 5177.2 (4876-5539.7) | 2343.7 (2131.6-2387.6) | 2.21x |
| 17k | 145.2 (142.2-148.3) | 101.5 (98.7-103) | 1.43x | 4708.7 (4410.1-4984) | 2001.1 (1986.6-2041.7) | 2.35x |
| 67k | 107.7 (103.4-109.5) | 63.8 (62.9-64.5) | 1.69x | 2755.7 (2577.5-3030.6) | 1140.2 (1131.5-1152.5) | 2.42x |
| 110k | 86.4 (85.5-87.6) | 48.7 (48.2-49.2) | 1.77x | 1963.1 (1902.7-2031.8) | 849.3 (838.9-852.4) | 2.31x |
| 200k | 60.9 (60.9-61.2) | 31.8 (31.5-31.9) | 1.92x | 1242.8 (1238.2-1247.8) | 537.3 (535.8-538.3) | 2.31x |
Qwen3.8-Flash-Next UD-Q3_K_XL¶
| KV depth | gmlx decode (baseline) | gmlx decode (MTP@2) | MTP lift | llama.cpp decode (MTP@2) | gmlx/llama.cpp decode | gmlx prefill | llama.cpp prefill |
|---|---|---|---|---|---|---|---|
| 512 | 50.4 (48.1-50.8) | 61 (51.7-67.5) | 1.21x | - | - | 743 (637-842.2) | 536.1 (402.5-567.3) |
| 4.3k | 47.8 (46.2-48.4) | 54.3 (47.5-57.5) | 1.14x | - | - | 1338.6 (1249.5-1392.2) | 688.6 (666.5-710.7) |
| 17k | 44.9 (44.4-45.9) | 54.1 (49.6-58.3) | 1.20x | - | - | 1516.1 (1485.9-1559.4) | 600.3 (573.5-614.6) |
| 67k | 36.8 (36.4-37.6) | 51.4 (47.7-58.1) | 1.40x | - | - | 1447.8 (1408.1-1489.7) | 394.7 (386-412) |
| 110k | 31.7 (31-32.4) | 49.8 (46-55.4) | 1.57x | - | - | 1379.9 (1354.6-1461.2) | 288.7 (282.4-292.4) |
| 200k | 25.3 (25.3-25.6) | 46.8 (44.9-49.7) | 1.85x | - | - | 1250 (1234.9-1275.6) | 210.8 (207-217.1) |
DeepSeek-V4 against ds4-server¶
The two antirez files are compared with ds4-server, a DeepSeek-V4 server, instead of llama.cpp. Each ratio is gmlx divided by ds4-server, and Methodology lists each model's ds4-server build.
DeepSeek-V4-Flash IQ2_XXS¶
| KV depth | gmlx decode | ds4-server decode | gmlx/ds4-server decode | gmlx prefill | ds4-server prefill | gmlx/ds4-server prefill |
|---|---|---|---|---|---|---|
| 512 | 42.6 (42.3-43.1) | 40.5 (39.6-40.7) | 1.05x | 541.6 (541.5-548.3) | 489.7 (482-492.1) | 1.11x |
| 4.3k | 38.8 (37.9-39.7) | 33.4 (33.4-34.6) | 1.16x | 783.5 (733.9-803.4) | 609.5 (572.3-612.9) | 1.29x |
| 17k | 36.9 (34.8-37.4) | 31.4 (30.8-31.8) | 1.18x | 774.3 (726.3-789.7) | 567.9 (524.6-568.4) | 1.36x |
| 50k | 34.7 (34.6-35.2) | 28.5 (27.8-28.5) | 1.22x | 765.7 (756.4-767.4) | 525.7 (520.6-525.9) | 1.46x |
| 67k | 33.8 (32.2-34.3) | 27.3 (27.2-28.1) | 1.24x | 718.2 (651.3-725.6) | 490.6 (450.2-492.4) | 1.46x |
| 110k | 32.2 (31.8-32.9) | 25.1 (24.1-25.7) | 1.28x | 672 (629.1-680) | 432.4 (411.8-443.4) | 1.55x |
| 200k | 30.1 (29.5-30.2) | 21.9 (21.3-21.9) | 1.37x | 602.9 (583.8-619.1) | 357.1 (347.8-365.8) | 1.69x |
| 300k | 27.6 (27.5-27.7) | 18 (18-18.5) | 1.53x | 542.4 (540.8-542.8) | 315.8 (312.4-321.2) | 1.72x |
| 384k | 24.9 (24.8-25) | 16.2 | 1.54x | 495.4 (493.8-496.9) | 275.9 (275.8-276.1) | 1.80x |
| 500k | 22.6 (22.6-22.7) | 14.2 | 1.59x | 440.7 (438.9-442.5) | 236.9 (236.5-237.3) | 1.86x |
DeepSeek-V4.1-Flash Q2¶
| KV depth | gmlx decode | ds4-server decode | gmlx/ds4-server decode | gmlx prefill | ds4-server prefill | gmlx/ds4-server prefill |
|---|---|---|---|---|---|---|
| 512 | 20.8 (18.3-20.9) | 15.5 (15-16.1) | 1.34x | 53.3 (47.1-84.6) | 34.8 (30.4-59.2) | 1.53x |
| 4.3k | 18.1 (17.6-19) | 15.1 (14.2-16.1) | 1.20x | 332.6 (322.8-354.5) | 113.9 (76-118.8) | 2.92x |
| 17k | 18.6 (18.2-20.6) | 14.5 (14-16.1) | 1.28x | 611.4 (485.1-620.6) | 189.9 (176.5-335) | 3.22x |
| 50k | 18.7 (17-20.5) | 14.7 (13.3-15.8) | 1.27x | 668.8 (662.9-685) | 507.5 (489.5-519) | 1.32x |
| 67k | 19.2 (17.1-21) | 14.9 (14.5-15.6) | 1.29x | 719.7 (716.4-727.8) | 511 (483-548) | 1.41x |
| 110k | 19.6 (18.9-20.5) | 14.8 (13.4-15.1) | 1.32x | 798.5 (759.8-835.1) | 626.8 (585.8-632) | 1.27x |
| 200k | 18.7 (18.1-18.9) | 13.1 (12.9-13.4) | 1.43x | 807.9 (782.1-831.4) | 556.5 (536.1-564.9) | 1.45x |
| 384k | 17.4 (12.9-18.2) | 12.2 (10.6-12.7) | 1.43x | 796.9 (790.7-807.7) | 612.5 (592.9-633.5) | 1.30x |
Flint against oMLX¶
oMLX ran RecursiveSI/flint-int4,
the MLX checkpoint of the same weights as the GGUF file gmlx ran. Its
tensors were relabeled from U8 to uint32 storage and its 8-bit embedding
declared in config.json, with the bytes unchanged. Each ratio is gmlx
divided by oMLX, and Methodology lists the oMLX build.
Flint INT4¶
| KV depth | gmlx decode | oMLX decode | gmlx/oMLX decode | gmlx prefill | oMLX prefill | gmlx/oMLX prefill |
|---|---|---|---|---|---|---|
| 512 | 160.2 (148.9-163.6) | 147.9 (142.9-149.7) | 1.08x | 3139.2 (2825.2-3354.5) | 2091.5 (1144.8-2546.3) | 1.50x |
| 4.3k | 155.8 (154.9-156.5) | 141.9 (141.1-142.4) | 1.10x | 5497.3 (5198.9-5536) | 3721 (3428.9-3855.8) | 1.48x |
| 17k | 148 (147.3-148.9) | 127.3 (124.1-127.8) | 1.16x | 4967.6 (4887.6-5006.6) | 3578.9 (3558.3-3605.7) | 1.39x |
| 67k | 109.1 (108.3-109.3) | 87.7 (86.6-89.2) | 1.24x | 2945.2 (2853.1-3038.5) | 2338.9 (2269-2423.4) | 1.26x |
| 110k | 87 (85.9-87.4) | 66.5 (66.1-67.3) | 1.31x | 1990.6 (1965.7-2052.6) | 1692 (1672.8-1704.2) | 1.18x |
| 200k | 60.9 (60.6-61.3) | 45.4 (44.9-46.2) | 1.34x | 1243.2 (1213.3-1248.8) | 1113.7 (1108.3-1117.3) | 1.12x |
Qwen3.8-27B against MTPLX¶
MTPLX by Youssof Altoukhi serves MLX checkpoints with speculative decoding from the model's native MTP head. It ran its Bare Speed checkpoint, Youssofal/Qwen3.8-27B-MTPLX-Bare-Speed, with MLX 4-bit affine weights in groups of 64 and a 4-bit MTP head. gmlx ran the Q4_0 GGUF file under Model provenance with its Q6_K output tensor requantized to Q4_0, so both engines store 4.5 bits per weight.
gmlx ran two speculative arms: the native MTP head, as MTPLX does, and
the DFlash 2
drafter
Qwen3.8-27B-DFlash2-Q4_0.gguf. Each ratio is gmlx divided by MTPLX.
This comparison has its own settings in place of those under
Methodology:
- Prompts are code requests from
ise-uiuc/Magicoder-Evol-Instruct-110K, the same ones on every arm, with thinking on at medium effort. - Sampling is temperature 1.0, top-p 0.95, top-k 20 and seed 1234, with 256 output tokens a request.
- Eight requests a cell up to 17K tokens and two from 67K, each in two rounds.
- Draft tokens a round are 5 for gmlx's native head, 7 for the DFlash 2 drafter and 3 for MTPLX, the most MTPLX allows for this checkpoint. Each was the fastest length tried at 4.3K tokens.
- gmlx prefill is from its native-head arm.
- MTPLX was measured on 2026-10-14 and gmlx on 2026-10-25. MTPLX repeated four of its 17K requests on 2026-10-25 with the same tokens and acceptance, about 3% slower.
Qwen3.8-27B 4-bit¶
| KV depth | gmlx decode (MTP) | gmlx decode (DFlash 2) | MTPLX decode (MTP) | gmlx/MTPLX decode (MTP) | gmlx/MTPLX decode (DFlash 2) | gmlx prefill | MTPLX prefill | gmlx/MTPLX prefill |
|---|---|---|---|---|---|---|---|---|
| 512 | 66.2 (53.9-84.8) | 80.9 (63.9-140) | 61.7 (54.9-90.9) | 1.07x | 1.31x | 577.1 (547.5-618.8) | 490 (418.6-556.9) | 1.18x |
| 4.3k | 69.8 (58.4-89.7) | 76.3 (45.1-113.2) | 54.5 (46.5-60.9) | 1.28x | 1.40x | 737 (726.9-748.4) | 670.6 (660.3-684.1) | 1.10x |
| 17k | 59 (51.8-83.9) | 71.8 (38.5-124.8) | 52.6 (43.8-61.7) | 1.12x | 1.37x | 712.6 (687.6-726.5) | 659.6 (643.5-679.9) | 1.08x |
| 67k | 48.8 (42-61) | 63.8 (52.8-85.1) | 38 (37.3-53.1) | 1.28x | 1.68x | 548.2 (542.2-551.3) | 534.3 (526.5-536.5) | 1.03x |
| 110k | 44.5 (40.6-46.3) | 58.9 (40.7-61) | 35.5 (33.8-37.7) | 1.25x | 1.66x | 472.2 (471-480.5) | 460.4 (436.8-463.1) | 1.03x |
| 200k | 37.3 (33.2-38.2) | 47.3 (41.9-55.4) | 23.7 (21.3-26.1) | 1.57x | 2.00x | 358.7 (355.9-361.2) | 318.5 (313-353.9) | 1.13x |
Image generation¶
gmlx draws a 1024x1024 image faster than ComfyUI and stable-diffusion.cpp for Z-Image-Turbo and both Qwen-Image 2.1 models, faster than ComfyUI for Krea 2 Turbo, and faster than mflux for Z-Image-Turbo, on the M5 Max listed under Methodology. Each speedup is the other engine's time divided by gmlx's, with the same GGUF file on both sides, except for mflux, which runs its own MLX quants against the gmlx file nearest in bits.
- Each engine ran 1024x1024 images with seed 42, cfg 1, euler and the model's default steps, one engine at a time in its own process, over three rounds in alternating order. The tables give the median of rounds 2 and 3.
- Seconds per step leave out the first step. Seconds per image cover the prompt encode, the steps and the VAE decode, without the model load; for mflux they cover the steps alone.
- ComfyUI 0.37.0 ran with ComfyUI-GGUF and torch 2.14 on MPS in bf16,
started with
--use-pytorch-cross-attention. For the Qwen-Image and Krea 2 models it took the prompt conditioning from gmlx, so its times cover the DiT, the sampler and the VAE. - stable-diffusion.cpp f89d9b13 was built with Metal and run with
--diffusion-fa. - mflux 0.20.0 ran its 4-bit and 8-bit Z-Image-Turbo quants.
Z-Image-Turbo, 8 steps¶
| Engine | Weights | Seconds per step | gmlx | Seconds per image | gmlx | Speedup per step | Speedup per image |
|---|---|---|---|---|---|---|---|
| ComfyUI with ComfyUI-GGUF | z-image-turbo-Q4_K_M.gguf | 2.74 | 1.48 | 22.4 | 12.2 | 1.86x | 1.84x |
| stable-diffusion.cpp | z-image-turbo-Q4_K_M.gguf | 4.65 | 1.48 | 41.0 | 12.2 | 3.15x | 3.36x |
| mflux | mflux 4-bit, gmlx z-image-turbo-Q4_K_M.gguf | 2.22 | 1.48 | 17.8 | 12.2 | 1.50x | 1.46x |
| mflux | mflux 8-bit, gmlx z-image-turbo-Q8_0.gguf | 2.40 | 1.50 | 19.2 | 12.4 | 1.60x | 1.54x |
Qwen-Image 2.1 Turbo, 8 steps¶
| Engine | Weights | Seconds per step | gmlx | Seconds per image | gmlx | Speedup per step | Speedup per image |
|---|---|---|---|---|---|---|---|
| ComfyUI with ComfyUI-GGUF | Qwen-Image-2.1-Turbo-AD-Q4_K.gguf | 3.19 | 1.76 | 28.0 | 14.7 | 1.81x | 1.91x |
| stable-diffusion.cpp | Qwen-Image-2.1-Turbo-AD-Q4_K.gguf | 5.62 | 1.76 | 49.5 | 14.7 | 3.20x | 3.37x |
Qwen-Image 2.1, 40 steps¶
| Engine | Weights | Seconds per step | gmlx | Seconds per image | gmlx | Speedup per step | Speedup per image |
|---|---|---|---|---|---|---|---|
| ComfyUI with ComfyUI-GGUF | qwen-image-2.1-Q4_K_M.gguf | 3.06 | 1.66 | 125.1 | 66.5 | 1.84x | 1.88x |
| stable-diffusion.cpp | qwen-image-2.1-Q4_K_M.gguf | 5.39 | 1.66 | 221.1 | 66.5 | 3.24x | 3.32x |
Krea 2 Turbo, 8 steps¶
| Engine | Weights | Seconds per step | gmlx | Seconds per image | gmlx | Speedup per step | Speedup per image |
|---|---|---|---|---|---|---|---|
| ComfyUI with ComfyUI-GGUF | Krea-2-Turbo-Q4_K_M.gguf | 5.55 | 2.89 | 45.1 | 23.4 | 1.92x | 1.93x |
Serving measurements¶
These measurements back the performance pages' guidance. Unless a row names another machine, they come from the M5 Max listed under Methodology.
| Measurement | Result |
|---|---|
| Uniform against mixed quant | A uniform Q6_K decoded 64% faster than the mixed UD build on the dense Qwen3.6-27B, and 15% faster on the MoE Qwen3.6-35B-A3B, with equal or better output. |
| Batched decoding | Three streams on Qwen3.6-35B-A3B Q6_K gave 1.3x to 1.7x the total throughput of one, falling as the context grew. |
| Admission pacing | A client arriving at 14K tokens left the running stream 4% of its decode speed under strict alternation, and 80% with pacing. |
| Shared-prompt cascade | Four streams on a 12K-token system prompt decoded about 1.4x faster in total. |
| Sparse attention | On Llama-3.1-8B Q6_K at 32K, decoding ran 1.4x faster for one stream and 1.8x in total for three, with Q6-level divergence. |
| Stochastic acceptance | Acceptance rose by a few points on a Q6 dense model and by about 14 points on a low-bit MoE quant. |
| DFlash 2 drafter | On Qwen3.8-27B at Q6, the drafter about tripled decoding speed over plain decoding, and was about 1.5x the native head. |
| Thermal behavior | A 14-inch M5 Max ran about twenty minutes of streamed MoE decoding before settling about 20% lower. The 16-inch model holds its boost clocks longer. |
KV cache fidelity¶
The fidelity measure is the teacher-forced logit KL divergence against an
fp16 cache on wikitext, from scripts/kld_harness.py. One leg scores the
chunked prefill logits, and the other scores decoding token by token from
the full prefill depth. KL divergence is in nats, and lower is better. The
median is the typical position. The decode p99 is the worst hundredth,
where a quantizer's outliers show, and top-1 is the share of
generated positions whose most likely token matches the fp16 cache.
Qwen3.5-9B Q4_K_M at 16K¶
The model has a head dimension of 256, and 7 of its 32 layers quantize.
| Cache | Prefill median | Decode median | Decode p99 | Decode top-1 |
|---|---|---|---|---|
| affine 2 | 0.02778 | 0.02745 | 0.5596 | 89.0% |
| kvarn 2 | 0.01503 | 0.00612 | 0.2301 | 94.7% |
| affine 3 | 0.00648 | 0.00606 | 0.1057 | 94.3% |
| kvarn 3 | 0.00291 | 0.00139 | 0.0313 | 97.4% |
| affine 4 | 0.00190 | 0.00183 | 0.0276 | 96.9% |
| kvarn 4 | 0.00117 | 0.00060 | 0.0071 | 98.3% |
| kvarn 5 | 0.00055 | 0.00029 | 0.0042 | 98.2% |
| kvarn k6 v5 | 0.00046 | 0.00028 | 0.0039 | 98.6% |
| affine 6 | 0.00046 | 0.00038 | 0.0045 | 98.7% |
| kvarn 6 | 0.00036 | 0.00027 | 0.0036 | 98.7% |
| affine 8 | 0.00029 | 0.00020 | 0.0027 | 98.7% |
| kvarn 8 | 0.00027 | 0.00020 | 0.0030 | 99.2% |
Qwen3.8-27B Q6_K_XL at 16K¶
The model has a head dimension of 256, and 15 of its 65 layers quantize.
| Cache | Prefill median | Decode median | Decode p99 | Decode top-1 |
|---|---|---|---|---|
| affine 2 | 0.01975 | 0.02314 | 0.4697 | 90.3% |
| kvarn 2 | 0.01009 | 0.00491 | 0.1280 | 95.1% |
| affine 3 | 0.00383 | 0.00419 | 0.1034 | 96.1% |
| kvarn 3 | 0.00212 | 0.00113 | 0.0318 | 97.3% |
| affine 4 | 0.00138 | 0.00136 | 0.0268 | 97.5% |
| kvarn 4 | 0.00084 | 0.00045 | 0.0078 | 97.4% |
| kvarn 5 | 0.00041 | 0.00025 | 0.0039 | 98.4% |
| kvarn k6 v5 | 0.00034 | 0.00019 | 0.0039 | 98.5% |
| affine 6 | 0.00033 | 0.00030 | 0.0055 | 98.7% |
| kvarn 6 | 0.00027 | 0.00019 | 0.0030 | 98.8% |
| affine 8 | 0.00023 | 0.00019 | 0.0032 | 98.7% |
| kvarn 8 | 0.00021 | 0.00015 | 0.0037 | 98.9% |
Qwen3.8-27B Q6_K_XL at 32K¶
This run uses the same model and layers at twice the context.
| Cache | Prefill median | Decode median | Decode p99 | Decode top-1 |
|---|---|---|---|---|
| affine 4 | 0.00162 | 0.00205 | 0.0176 | 97.6% |
| kvarn 4 | 0.00104 | 0.00070 | 0.0068 | 98.1% |
| affine 6 | 0.00039 | 0.00049 | 0.0037 | 98.5% |
| kvarn 6 | 0.00033 | 0.00032 | 0.0027 | 99.4% |
| affine 8 | 0.00027 | 0.00030 | 0.0038 | 98.8% |
| kvarn 8 | 0.00026 | 0.00028 | 0.0026 | 99.0% |
Nemotron-3.5-Lightning-30B-A3B at 16K¶
This Mamba2 hybrid has a head dimension of 128.
| Cache | Prefill median | Decode median | Decode p99 | Decode top-1 |
|---|---|---|---|---|
| kvarn 4 | 0.00270 | 0.00163 | 0.0508 | 98.1% |
| kvarn 6 | 0.00125 | 0.00103 | 0.0266 | 98.8% |
| affine 8 | 0.00123 | 0.00094 | 0.0335 | 98.2% |
| kvarn 8 | 0.00111 | 0.00095 | 0.0298 | 98.6% |
Reading the tables¶
The kvarn cache beats the affine cache of the same width on both legs at every width below 8, by 3 to 5x on the decode median at 2 to 4 bits, and the two converge at 8. At 6 bits, kvarn sits between affine 6 and affine 8 on the 9B model and matches affine 8 on the 27B model, in three quarters of the memory of affine 8. At 32K, its decode median can trail affine 8 by a few percent while its p99 and top-1 stay ahead. The split width k6 v5 keeps kvarn 6's decode median, and its p99 and top-1 are at least as good as kvarn 5's.
When two caches differ by a few percent on one measure, prefer the one with the lower p99 and the higher top-1. Top-1 is closest to what a greedy or low-temperature user sees. The median measures how far the whole next-token distribution moved, which is what sampling draws from. The p99 bounds the outliers, which matter because one badly wrong position can change a reasoning chain or a tool call, and a long generation feeds its errors back in. The corpus is wikitext under teacher forcing, so these tables rank caches against each other and do not predict a task score.
TurboQuant, mlx-vlm's scheme (arXiv:2504.19874), was measured on the same models and legs. It was ahead of affine at 2 and 3 bits, level at 4 and behind at 6 and 8. It trailed kvarn at each width on all measures.
Speed was measured on Qwen3-0.6B Q8 with 27 of 28 layers quantized, a dense model whose decoding is limited by the KV read. At 16K, kvarn 6 decoded at 0.81x fp16 and 0.69x affine 8, and at 32K at 0.98x and 0.75x. Prefill stayed within 10% of both. On GDN hybrids and gemma-4, all three caches ran within the spread between runs.