Speculative decoding

Speculative decoding makes a model generate faster without changing its output. A small, fast drafter proposes the next few tokens, and the model checks all of them in one pass. It keeps the tokens it agrees with and adds one of its own, so a round can produce several tokens for about the cost of one.

gmlx run Qwen3.8-27B-UD-Q6_K.gguf                          # a native head turns it on
gmlx run model.gguf --draft-gguf drafter.gguf              # a separate drafter file
gmlx run model.gguf --bench-depths "0,4096" --speculative  # measure the gain

Turning it on

Some GGUF files carry a native head, a small extra layer that drafts tokens, as the Qwen3.5, Qwen3.6 and Qwen3.8 models do. run and chat turn speculation on for these without a flag. DeepSeek-V4, Qwen3.8-Flash-Next, Muse Glimmer and Nemotron 3.5 Super VL turn it on the same way when their companion drafter sits in the model's folder, or in an MTP folder inside or beside it. Nemotron 3.5 Bolt and MiniCPM-V 4.7 35B-A3B turn it on when their drafter repo, <repo>-LiLiCorr-GGUF or <repo>-DSpark-gguf, sits beside the model's repo. Other families, such as gemma-4 with its assistant drafter, need --draft-gguf.

Flag	Config key	Effect
--speculative	speculative: true	Turns it on with a native head or a companion that the loader finds.
--draft-gguf PATH	draft_gguf	Drafts with this file. Turns speculation on, and wins over a native head.
--native-mtp	native_mtp: true	Drafts with the native head even when a drafter file is set.
--no-mtp	speculative: false	Turns it off.

On the server, a model speculates when its speculative key is true or auto. gmlx init sets auto for models with a head, and gmlx pull and a discover scan pair a drafter that they find with its model. A model pulled by name, as in gmlx pull qwen3.8-27b, comes with its drafter and registers it, unless you pass --no-drafter. For a model already in the config, gmlx sync-models --add-companions adds the drafter to its entry.

Speculation works with some other features and stays off with others:

• --stream-experts: off by default, and --speculative turns it on.
• A MoE whose experts stay in memory: on in run and chat and for speculative: auto, off with an [mtp] line when the drafter does not fit beside them. --speculative or speculative: true streams the experts to keep it, as When a MoE streams its experts describes.
• --stream-cpu: run and chat refuse --speculative.
• The lossy MoE settings: off, and chat refuses --speculative.
• --mmproj: text turns speculate. Turns with images or audio speculate on the server. In run and chat, audio turns on a model with its own speech encoder speculate too, and other media turns decode without it.
• --adapter: the adapted model checks each draft, so the output matches plain decoding with the adapter.

Settings that speculation drops

In run and chat, speculation drops a few settings with a warning for each, such as:

• --max-kv-size and --quantized-kv-start
• on run, also --stop and --prefill-step-size

Pass --no-mtp to keep them.

Logit bias, the penalties and XTC work with speculation. They lower the share of accepted drafts, XTC the most, so a request that sets them gains less from speculation.

How much it gains

The gain depends on how many drafts the model accepts. A dense model decodes about 1.6 to 1.9 times as fast at short contexts and keeps a smaller gain deep into long ones. Most MoE models gain less, and some become slower at depth. Code gains more than free prose, because it is easier to predict. Benchmarks has each model's speedup curves, and --bench-depths measures your own model, as in the example above.

Qwen3.8-Flash-Next sizes each round by how sure its head is of the first draft: five drafts when it is sure, three when it is fairly sure, and one when it is not, with at most three after a round where the model rejected a draft. Code decodes up to a fifth faster this way than with a fixed three drafts, and prose about as fast.

The native head of Nemotron 3.5 Transcribe drafts 7 tokens a round, where other Nemotron-H heads draft 2. --draft-block-size sets a smaller block, and GMLX_NEM_MTP_BLOCK sets the block of every Nemotron-H head, drafts plus one. Below 48 GB of memory, gmlx pull registers this model with speculative: false.

A quantized KV cache lowers the share of accepted drafts, and 4 bits costs the most. With speculation on, keep the KV cache at full precision if you can, and use 8 bits if memory requires it.

Several requests at once

Checking a draft costs little while one request decodes and much more when several do. The server therefore gives each model a width cap, the number of requests that speculate together. Past the cap, the oldest requests keep speculating and the others decode without speculation in the same step, and when a speculating request finishes, the next oldest takes its place. This needs a drafter that keeps its own state for each request and a model that undoes each request's rejected drafts separately. In these cases the whole batch decodes without speculation past the cap instead, and speculates again when the batch shrinks:

• Any mixture-of-experts model, because the padded positions of the other requests widen the set of experts each step reads. GMLX_MTP_MIXED=1 gives such a model mixed batches anyway.
• The gemma-4 assistant drafter, which reads the model's own KV cache.
• gemma-4 and MiniMax-M3 models, and the MTP heads of GLM-DSA, Inkling and Hy-V4 models, which accept the same number of drafts for every request in a batch.
• Hy3, DeepSeek-V4, Muse Glimmer, Nemotron-H, Qwen3.8-Flash-Next and GLM-5.3-Flash models, which undo the rejected drafts of one request at a time.
• A quantized KV cache that cannot roll back each request separately.

Drafter	Default cap
A native head on a dense Qwen model	No cap
The gemma-4 assistant drafter, and any family without a default of its own	2
Any mixture-of-experts model	1
The heads of Hy3, DeepSeek-V4, Muse Glimmer, Nemotron-H, Qwen3.8-Flash-Next and GLM-5.3-Flash, and every DFlash drafter	1, which no setting raises

speculative_width_cap sets the cap for one model, and 0 removes it. gmlx serve --speculative-width-cap N sets it for every model and overrides each model's key. Speculative batching describes how the batch switches between these modes.

DFlash drafters

DFlash 2 is a block-diffusion drafter, with checkpoints for Qwen3.8-27B and Muse-Glimmer-30B. One drafter pass proposes a whole block of tokens, where a native head runs one pass for each token it drafts.

gmlx run Qwen3.8-27B-UD-Q6_K.gguf --draft-gguf Qwen3.8-27B-DFlash2-Q8_0.gguf

A DFlash 2 file's header names its base model, so a discover scan pairs the two even in different folders. Muse Glimmer also finds a drafter in its own folder with --speculative, or when --mmproj loads its vision encoder. Qwen3.8-27B keeps its native head until you pass --draft-gguf.

The block is the size the checkpoint was trained with, 8 on Qwen3.8 and 16 on Muse Glimmer, so a round drafts 7 or 15 tokens. --draft-block-size makes it smaller. The drafter handles one request at a time, so its server width cap is 1.

gmlx also runs the community DSpark drafters for Ternary Bonsai 2 27B, which draft a block in one pass too. The loader reports them as dflash_dspark. On Bonsai, the Qwen3.8-27B DFlash 2 drafter is faster for most prompts, and the Bonsai-trained drafters lead only on long code output. Start with the Qwen3.8 drafter and measure the others on your own work.

Gemma 4 models run plain DFlash drafters too. unsloth ships one beside its NVIDIA-Nemotron-3.5-Bolt-E4B quants. gmlx pull nemotron-3.5-bolt-e4b downloads and pairs the two, and a pull by reference pairs them when you name both files:

gmlx pull hf:unsloth/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF \
  NVIDIA-Nemotron-3.5-Bolt-E4B-UD-Q4_K_XL.gguf \
  dflash-NVIDIA-Nemotron-3.5-Bolt-E4B-Q8_0.gguf

Its block is 8, so a round drafts 7 tokens.

MiniCPM-V 4.7 ships a DSpark drafter beside each model. For the 35B-A3B, a pull that names the drafter pairs it, as Supported families shows, and gmlx chat then speculates on text turns. Its block is 16, but a round drafts 3 tokens, because longer drafts cost more than they gain. --draft-block-size sets another size. The 1B's drafter decodes slower than the 1B alone, so gmlx never pairs it. Pass it with --draft-gguf to try it.

Context copy

When the output repeats the prompt, as when the model edits code it was given or quotes a document, the prompt itself is a good draft. Context copy looks for the newest generated tokens in the prompt, and when it finds them, the round checks the prompt tokens that follow in place of the drafter's draft. The model accepts them by the same rule as any draft. The drafter learns the accepted tokens and drafts the next round as usual.

A copy can run longer than the drafter's draft, up to 24 tokens. The model checks a longer copy with other kernels than a drafter round uses, and their rounding can change which token wins where the model's top two candidates are almost equally likely. From such a token on, the output can differ from a run without context copy. For a block drafter, GMLX_CONTEXT_COPY_MAX set to its draft length, 7 for a DFlash 2 drafter with blocks of 8, keeps the output token-identical to a run without copies. A copy shorter than 4 tokens is not worth a round, so a value under 4 turns copying off. A native head that drafts 2 or 3 tokens therefore has no copy length that keeps the output identical, other than no copies at all. DeepSeek-V4 and GLM-5.3-Flash copy at most 5 tokens a round, the most they can take back after a rejected draft.

When the model rejects most of a copy, or a copy round makes tokens more slowly than the drafter's rounds, copying pauses for a while, longer after each such round in a row. At temperature 0, a copy starts at 8 tokens, or at one less than twice the tokens of an average drafter round when that is more, and doubles each time the model accepts one whole, up to that limit. So a copy round can add at least twice the tokens of an average drafter round, and a drafter that already predicts the text well keeps its rounds. When sampling, the model keeps a copied token only as often as it would pick that token, so the copy length follows the chance that a copied token is kept. A copy stops where that chance gets low, and a copy runs only when the tokens it is expected to add beat a drafter round by half. That chance comes from the request's recent copies and from how far back the generated text already repeats the prompt, so text that the output repeats word for word still copies beside a strong drafter.

On the server, a request copies when it starts while no other request decodes, and stops copying when another request joins it.

context_copy: true turns it on for a model. GMLX_CONTEXT_COPY=1 turns it on for every model, in run and chat too, GMLX_CONTEXT_COPY=0 turns it off everywhere, and GMLX_CONTEXT_COPY_MAX sets the longest copy. Context copy gains on grounded answers, edits and rewrites, and does nothing on free prose that does not repeat the prompt.

Stochastic acceptance

On a request with a temperature above zero, gmlx accepts drafts by rejection sampling. The output is a true sample from the model's distribution, and more drafts pass than by exact match, so decoding is faster. The gain is largest on low-bit quants and on text where the model is unsure. Greedy requests match a run without speculation. On Nemotron-H, an explicit --stochastic-mtp or stochastic_mtp: true takes a faster verify, and a greedy request can then differ on a near tie.

A seeded sampled run is not token-identical to the same run without speculation. To accept only drafts that match the token the model would sample, turn stochastic acceptance off:

gmlx run model.gguf --no-stochastic-mtp --temp 0.7 --seed 7 --prompt "Hello"

In a server config, set server.stochastic_mtp: false.