Internals

gmlx internals cover the serving path, the caches, speculative batching, model support and the tooling for tests and upstream upgrades. The user guides start at the documentation home, and the development setup is in the contributing guide.

How it works

• Serving architecture: The path of a request through the server, and the scheduling policy gmlx adds to mlx-vlm.
• Speculative batching: How speculative decoding and continuous batching run together.
• Prompt cache internals: The cache tier of each architecture, the cache layers and the reuse counters.
• Structured reads: How /v1/systemone answers questions from a denoise read, and its parity with the vLLM example.
• Hadamard-folded GGUFs: Weights stored under a Hadamard rotation, and where gmlx undoes it.
• Distillation internals: How the teacher pass and the training head bound their memory, and the costs of the worked run.

Working on gmlx

• Adding a GGUF architecture: What supporting a model family involves, and its acceptance gate.
• Testing: The test tiers, GGUF-gated runs and the end-to-end harnesses.
• Upgrading mlx-vlm, mlx-lm and mlx: Moving the pinned upstream versions.
• Debug switches: Environment variables for isolating defects, including the prompt cache switches.

Measurements

• Streaming measurements: The data behind the lossless and lossy settings of streamed models.
• Structured read measurements: Prefill, read, sample, step and thought timings on DiffusionGemma.