Concurrent requests

The gmlx server decodes several clients' requests together as one batch. It paces a new request's admission so that the running streams do not stall, and when several requests share a prompt, it reads that prompt only once.

Batched decoding

The server decodes all active requests of a model as one batch. Decoding is limited by memory bandwidth, and a batched step reads the weights once for all requests. Total throughput therefore rises with the number of clients, and each request slows by less than its share. The gain falls as the contexts grow, because each request does its own attention work. Benchmarks has measurements.

A model with speculative decoding speculates only while the batch is narrow, as Several requests at once describes. A model that streams from disk should get one request at a time, as Serving a streamed model explains.

Admitting a new request

A new request's prompt must prefill while other requests are decoding. Prefill runs in chunks of 2048 tokens, and at a deep context one chunk can take as much GPU time as hundreds of decode steps. A scheduler that alternates one decode step with one chunk therefore lets a long prompt stall the streams that are already running. When chunks are short, slowing admission only delays the new request and narrows the batch. Two settings handle these stalls, one for each symptom:

Symptom	Setting	Default	Effect
The running streams stall while a long prompt arrives.	server.decode_prefill_ratio	auto	The server slows admission only when a running stream would drop below half of its batched speed. A number fixes the ratio, and 0 alternates strictly.
A stream pauses during a long prefill chunk.	server.prefill_tick_ms	500	The server halves each chunk until its expected time fits the budget. 0 turns off the halving, which suits batch jobs.

With pacing, a new request waits at most about twice as long for its first token as it would without pacing. Prefill runs at full speed when nothing is decoding, so one client alone sees no difference under any setting. The server reads both settings at start, so change them and run gmlx restart.

Shared prompts

Requests often share the start of their prompt, such as a common system prompt or histories restored from the prompt cache. The server finds the shared part from the requests' token ids, and decodes such a batch with a cascade kernel that reads the shared part once for the whole batch. The gain grows with the length of the shared part and the number of requests. The cascade is exact and on by default, and GMLX_CASCADE_SDPA=0 turns it off.