Skip to content

Internals

gmlx internals cover the serving path, the caches, speculative batching, model support and the tooling for tests and upstream upgrades. The user guides start at the documentation home, and the development setup is in the contributing guide.

How it works

  • Serving architecture: The path of a request through the server, and the scheduling policy gmlx adds to mlx-vlm.
  • Speculative batching: How speculative decoding and continuous batching run together.
  • Prompt cache internals: The cache tier of each architecture, the cache layers and the reuse counters.
  • Structured reads: How /v1/systemone answers questions from a denoise read, and its parity with the vLLM example.
  • Letter readout: How /v1/systemone answers on autoregressive models, and its parity with OpenJev's helper.
  • Hadamard-folded GGUFs: Weights stored under a Hadamard rotation, and where gmlx undoes it.
  • Byte-level pre-tokenizers: How a GGUF's pre-tokenizer name picks the word split and flags that llama.cpp uses.
  • Distillation internals: How the teacher pass and the training head bound their memory, and the costs of the worked run.
  • Image serving: How image jobs share the GPU with chat replies, and how the server plans their memory.
  • Image step routes: The contract that lets a sampler in another program, such as ComfyUI, run an image model one step at a time.

Working on gmlx

  • Adding a GGUF architecture: What supporting a model family involves, and its acceptance gate.
  • Testing: The test tiers, GGUF-gated runs and the end-to-end harnesses.
  • CI and the GPU gate: What a pull request needs to merge, the GPU gate run on a Mac, and what each CI leg runs.
  • Upstream releases: The copy of mlx-vlm and mlx-lm that gmlx carries, how it takes upstream changes, and moving mlx.
  • Debug switches: Environment variables for isolating defects, including the prompt cache switches.

Measurements