Configuration¶

gmlx.yaml names the models gmlx can run, the settings each one runs with, and the server's own settings. Every key has a default, so a file holds only the keys you want to change. A model name from the file works in a request, in gmlx run and in gmlx chat.

Create the file
Where gmlx looks
What the file contains
Models, Aliases, Profiles, Rules
Sampling, Model loading, Prompt cache
Model discovery
Server
Voice, Assistant, Launch, Chat themes
Changing the file
Flags and environment variables
Complete example

Create the file¶

gmlx init scans a folder of GGUF files and writes a config with one entry for each model it finds:

gmlx init --models-dir ~/models

The file goes to ~/.config/gmlx/gmlx.yaml. With no flags on a terminal, init asks its questions as a wizard instead. The result holds the server settings you chose and the models it found:

# gmlx configuration, written by `gmlx init`.
# Every key, with its default: https://asher.github.io/gmlx/config.html
# The settings a server runs with: gmlx serve --print-config
# The sampling each model starts from: gmlx profiles

server:
  host: 127.0.0.1
  port: 8080
  model_dirs:
    - ~/models
  defaults:
    ttl_s: 900
  cache:
    enabled: true

models:
  # sampling (gemma): t=1 top_p=0.95 top_k=64
  gemma-4-12b-it-q4:
    path: gemma-4-12b-it-Q4_K_M.gguf
  # sampling (qwen3.6): t=1 top_p=0.95 top_k=20
  qwen3.8-27b-ud-q6:
    path: Qwen3.8-27B-UD-Q6_K.gguf
    speculative: true

The comment above each model shows the sampling values it starts from, taken from its family defaults and the GGUF. Start the server with gmlx serve, which finds the file on its own.

Where gmlx looks¶

A command that needs the file reads the first one it finds:

~/.config/gmlx/gmlx.yaml, where gmlx init writes.
~/.gmlx.yaml.

Pass --config FILE to read another file. gmlx never reads a gmlx.yaml in the current folder, because a config can name commands that the server runs.

gmlx reads only the first file it finds. If you already have a ~/.gmlx.yaml, a new ~/.config/gmlx/gmlx.yaml hides it completely, so add new blocks to the file you already have.

To see every setting a server would run with, including the defaults you did not set, run gmlx serve --print-config. It prints YAML and exits without loading a model.

What the file contains¶

Every block is optional. A server needs models, from models or from a discover scan.

Block	What it sets
`models`	The models gmlx can run, each with its GGUF file and settings
`aliases`	Extra model names, such as `coder` for `qwen3.8-27b-ud-q6@coding`
`image_models`	Image generation models, each with its DiT, text encoder and VAE files
`video_models`	Video generation models, each with its DiT, text encoder, VAEs and LoRA
`profiles`	Named sets of sampling, loading and prompt settings that any model can use
`rules`	A profile for every model whose id matches a pattern
`discover`	Folders scanned at each start for GGUFs that have no entry
`server`	Address, API key, model folders, memory budget and extra services
`talk`	The voice client's model, voice, wake phrase and listening thresholds
`assistant`	Tool servers and long-term memory of the built-in assistant
`launch`	How `gmlx launch` runs clients in containers
`theme`, `themes`	Colors of the terminal chat

A key is named by its full path, such as server.port. models.*.pin means the pin key of any entry under models.

Models¶

Each entry under models is one model. The entry's key is the model's id, which a request puts in its model field and which gmlx run and gmlx chat take on the command line. You can rename an entry freely, but an id cannot contain @, which separates an id from a profile name.

models:
  qwen3.8-27b-ud-q6:
    path: Qwen3.8-27B-UD-Q6_K.gguf
    speculative: true
    pin: true
  gemma-12b:
    path: gemma-4-12b-it-Q4_K_M.gguf
    mmproj: mmproj-gemma-4-12b-it-bf16.gguf
    profile: coding

An entry whose file is missing is skipped with a warning, and the server keeps running. gmlx sync-models removes such entries.

Key	Default	Meaning
`path`	required	The GGUF file: absolute, relative to a `server.model_dirs` folder, or `hf:<org>/<repo>/<file.gguf>[@rev]`
`profile`	none	Profile for requests that name none: one under `profiles` or an intent such as `coding`
`profiles`	none	Changes to a profile for this model only, keyed by profile or intent name. Each value takes the keys of `overrides`.
`overrides`	none	Settings that win over every profile, for this model only. Takes every profile key except `extends`.
`family`	detected	The family defaults to start from. Set it only when detection picks the wrong family.
`mmproj`	none	Vision or audio projector GGUF that makes the model multimodal
`asr`	found beside the model	Speech companion GGUF for a model that reads audio through one, such as Granite Switch. `false` pairs none.
`speculative`	`false`	Speculative decoding with the GGUF's MTP head, or with `draft_gguf`. On a MoE, `auto` loads the drafter only when the experts fit in memory beside it.
`draft_gguf`	none	A separate drafter GGUF. Setting it turns on `speculative`.
`native_mtp`	`false`	Draft with the GGUF's own MTP head even when `draft_gguf` is set
`speculative_width_cap`	`null`	Speculate on at most this many requests at a time. `null` takes the drafter's default, `0` removes the cap.
`context_copy`	`false`	Also draft from the prompt, as Context copy describes.
`adapters`	none	LoRA GGUFs, each a path or `{path, strength}`, strength 1 by default. A path listed twice is an error.
`adapter`	none	Shorthand for one `adapters` entry at strength 1. Set `adapter` or `adapters`, not both.
`pin`	`false`	Load at start and stay loaded. Only `POST /unload` unloads a pinned model.
`ttl_s`	`server.defaults.ttl_s`	Seconds without a request before the model unloads. `null` or `0` turns off the timeout.
`stream`	none	Run a model larger than memory. `experts` streams the routed experts from disk, `cpu` runs the whole model on the CPU.

Each drafter has its own default speculative_width_cap, which Several requests at once lists.

Ids with the same path and different adapters share one loaded copy. The strength in adapters is the default of a request, and a request can set another without a reload, as Mixing adapters per request shows.

profiles changes one profile for one model. Here @coding also sets min_p on this model, and other models keep the plain intent:

models:
  qwen3.8-27b-ud-q6:
    path: Qwen3.8-27B-UD-Q6_K.gguf
    profiles:
      coding: {sampling: {min_p: 0.05}}

Streaming keys¶

These keys apply only to a model with stream set. Models larger than memory explains how to choose a placement and how to size the moe_* keys, which all change the output.

Key	Default	Meaning
`moe_experts`	the model's count	Fixed number of experts per token, below the number the model was trained with
`moe_expert_mass`	off	Keep the smallest set of experts whose gate weights reach this share, above 0 and at most 1
`moe_miss_shed`	off	Drop experts outside the decode arena while the kept ones still cover this share of the gate weight
`moe_layer_shed`	off	Probability, between 0 and 1, of skipping a streamed layer's routed experts
`moe_prestage`	`ranked`	Experts read ahead: `ranked` reads the predicted ones, `keepers` only those that `moe_miss_shed` keeps, so it needs `moe_miss_shed`
`prefill_feeder`	`true`	Prefill reads expert weights directly from the GGUF
`decode_feeder`	`true` with `stream: experts`	Decoding keeps the most used experts in a wired arena
`stream_fast_disk`	`auto`	Read-ahead for streamed decoding. `auto` tests the drive, `on` and `off` force the choice.

Rope scaling¶

rope_scaling runs a model past its trained context. Its keys are the rope scaling flags of gmlx run, and each one wins over its key in the GGUF header:

models:
  qwen3.8-flash-next-512k:
    path: Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf
    rope_scaling:
      type: yarn
      factor: 2.0
      original_context: 262144

Key	Flag	Meaning
`type`	`--rope-scaling`	`none`, `linear` or `yarn`
`factor`	`--rope-scale`	Expand the context by this factor
`freq_scale`	`--rope-freq-scale`	The same value as `factor`, spelled as its inverse
`freq_base`	`--rope-freq-base`	Rope base frequency
`original_context`	`--yarn-orig-ctx`	The context the model was trained at
`ext_factor`	`--yarn-ext-factor`	Extrapolation mix, 1 under yarn and 0 otherwise by default
`attn_factor`	`--yarn-attn-factor`	Attention magnitude scale
`beta_fast`	`--yarn-beta-fast`	Low correction dimension, 32 by default, or a mistral3 header's
`beta_slow`	`--yarn-beta-slow`	High correction dimension, 1 by default, or a mistral3 header's

Two ids on one GGUF with different rope_scaling load as separate models, and the prompt cache on disk keeps their KV apart. A YaRN factor extends the context that /v1/models reports to original_context times the factor. A linear factor leaves it at the trained context. A constant factor can cost quality on short prompts, so pick the smallest factor that fits.

Aliases¶

An alias gives a model a second name, and can fix a profile to it. The server lists each alias in /v1/models as its own entry, so a client that picks models from a menu can choose a model and a profile together.

aliases:
  fast: gemma-12b
  coder: qwen3.8-27b-ud-q6@coding

An alias cannot contain @ or match a model id. gmlx pull <name> adds the name as an alias with the comment set by gmlx pull, and a later pull of the same name moves it to the new file.

Image models¶

Each entry under image_models is one image generation model: a DiT GGUF with its text encoder and VAE. gmlx pull writes one for a model it downloads, and model discovery adds one for a DiT it finds with its parts. The entry's key is its id, which a request to the image routes puts in its model field. Image ids, model ids, aliases and served assistants share one set of names, so an image id cannot repeat any of them, and it cannot contain @.

models:
  qwen3-4b:
    path: Qwen3-4B-Q4_K_M.gguf
image_models:
  z-image-turbo:
    path: z-image-turbo-Q4_K_M.gguf
    encoder: qwen3-4b
    vae: z-image-vae-BF16.gguf
  qwen-image:
    path: qwen-image-2.1-Q4_K_M.gguf
    encoder: Qwen3-VL-8B-Instruct-Q4_K_M.gguf
    encoder_mmproj: mmproj-Qwen3-VL-8B-Instruct-BF16.gguf
    vae: qwen-image-2.1-vae-F32.gguf
    defaults: {size: 1024x1024, steps: 30}
server:
  defaults:
    image_model: z-image-turbo

Image models load into the same memory budget as chat models, and unload the same way: by ttl_s, by POST /unload, or to make room for another model. The memory an entry counts is its weights and what one image at its default size needs while it runs, so two ids with the same files and another default size or guidance load as two entries.

Key	Default	Meaning
`path`	required	The DiT GGUF, found as a model `path` is
`encoder`	required	The text encoder: the id of an entry under `models`, whose loaded copy it shares, or a GGUF path
`encoder_mmproj`	none	The encoder's vision tower GGUF, which Qwen-Image 2.1 edits from reference images need. A shared encoder brings its model's `mmproj` instead.
`vae`	required	The VAE, a GGUF or a safetensors file
`family`	detected	The image family, such as `z-image`, `qwen-image-2.1` or `flux-3-klein`. A file of another family is refused at load.
`variant`	detected	The DiT's variant, such as `turbo`, for a file whose weights match no known variant. A Krea 2 file of unknown weights needs `variant: turbo` to load.
`adapters`	none	LoRA files applied to every request, each a path or `{path, strength}` with strength 1 by default
`upsampler`	none	The `models` id of the chat model that rewrites each prompt, as Prompt upsampling describes
`upsample_prompt`	the family's	A text file with the system prompt for the `upsampler`, found in `model_dirs` when relative. Qwen-Image 2.1 needs one, which `gmlx pull qwen-image` fetches
`upsample`	`true`	Rewrite prompts with the `upsampler`. `false` runs them as typed, unless a request sends `prompt_upsampling: true`.
`defaults`	the family's	Request defaults: `size` as `WxH`, `steps`, `guidance`, `sampler` (`euler` or `uni_pc_bh2`) and `negative_prompt`
`encoder_release`	`auto`	Release the text encoder after each encode: `always`, `never`, or `auto`, which keeps it while the whole model fits in 80 percent of the working budget
`pin`	`false`	Load at start and stay loaded, as a model's `pin`
`ttl_s`	`server.defaults.ttl_s`	Seconds without a request before the model unloads. `null` or `0` turns off the timeout.

The auto choice of encoder_release counts no request LoRAs. A shared encoder must be a model without adapters. A quantized chat GGUF used as the encoder gives slightly different images than a full precision encoder file.

Video models¶

Each entry under video_models is one video generation model for gmlx video generate: a DiT GGUF with its text encoder, its video and audio VAEs and its LoRA. gmlx video pull writes one for each recipe it pulls. Video ids share one set of names with model ids, image ids, aliases and served assistants.

video_models:
  h3:
    path: unsloth__MiniMax-H3-GGUF/minimax_h3_fl2va_pruned-Q4_K.gguf
    encoder: unsloth__Qwen3-VL-32B-Instruct-GGUF/Qwen3-VL-32B-Instruct-Q4_K_M.gguf
    vae: Comfy-Org__MiniMax-H3/vae/minimax_h3_video_vae_fp16.safetensors
    audio_vae: Comfy-Org__MiniMax-H3/vae/minimax_h3_audio_vae_fp32.safetensors
    adapters: [lightx2v__Minimax-h3-Turbo/minimax_h3_fl2v_turbo_8step_v1.1_768p_comfyui_bf16.safetensors]
    defaults: {steps: 8, shift: 6, size: 1344x768, seconds: 5, int8: off}

Key	Default	Meaning
`path`	required	The DiT GGUF, found as a model `path` is
`encoder`	required	The text encoder GGUF
`encoder_mmproj`	none	The encoder's vision tower GGUF, which reads a first or last frame
`vae`	required	The video VAE safetensors file
`audio_vae`	required	The audio VAE safetensors file
`tiny_vae`	none	The TAEH3 file that `--preview` decodes with
`adapters`	none	LoRA files applied to every clip, each a path or `{path, strength}` with strength 1 by default
`variant`	detected	The DiT's variant, such as `fl2va-turbo4` for a merged Turbo file gmlx does not know
`defaults`	none	Clip defaults: `steps`, `shift`, `size` as `WxH`, `seconds` and `int8`. Unset, steps and shift come from the variant, and a clip is 1344x768 for 5 s
`upsampler`	none	The `models` id of the chat model that rewrites each prompt, as Prompt upsampling describes
`upsample_prompt`	none	MiniMax's prompt guide, which the `upsampler` reads, found in `model_dirs` when relative. `gmlx video pull` fetches it
`upsample`	`true`	Rewrite prompts with the `upsampler`. `false` runs them as typed, unless a request sends `prompt_upsampling: true`.

defaults.int8 is off, the default, attention, linears or all, the parts of the DiT that run in int8, as Faster clips with int8 describes. linears and all take more memory.

Profiles¶

A profile is a named set of settings that any model can use. Every model starts from its family defaults, the sampling values its publisher recommends, so a new file needs no profiles at all.

Some publishers also recommend values for a task, such as a lower temperature for code. gmlx offers these as built-in profiles called intents: @coding, @instruct, @creative, @reasoning-low, @reasoning-medium, @reasoning-high, @reasoning-max and @reasoning-xhigh. Add one to the model id to use it:

gmlx run qwen3.8-27b-ud-q6@coding --prompt "Write a binary search in Go."

Your own profiles go under profiles. A profile can start from an intent with extends and change only what it sets. In the file, you write an intent without the @, as in extends: coding. A profile with an intent's name replaces that intent.

profiles:
  review:
    extends: coding
    system: "You are a terse code reviewer."
    sampling:
      temperature: 0.3
      max_tokens: 2048
  agent:
    chat_template_kwargs: {preserve_thinking: true}

How a request gets its settings¶

Settings come in layers, and a later layer wins over an earlier one:

Layer	Set by
Family defaults	Built in, for each detected family
Server default profile	`server.defaults.profile`
Rule profile	The first matching entry in `rules`
Model profile	`models.*.profile`, or `@name` on the request
Per-model profile change	`models.*.profiles`
Model overrides	`models.*.overrides`
Request fields	The request body

A request's @name replaces the model's own profile rather than adding to it. gmlx profiles <id> prints the sampling values a model resolves to under each of its profiles, with every layer applied.

Profile keys¶

Key	Default	Meaning
`extends`	none	Profile or intent to start from. Each key this profile sets replaces the inherited value.
`sampling`	none	Defaults for the sampling request fields. A field the request sends wins.
`load`	none	Model loading keys
`cache`	none	Prompt cache keys that change `server.cache` for this profile's models
`system`	none	System prompt, used only when the request has no system message
`chat_template`	the GGUF's	Inline Jinja, or a path to a `.jinja` or `.txt` file, that replaces the GGUF's chat template
`chat_template_kwargs`	none	Variables passed to the chat template on each request. Keys the request sends win.
`thinking`	the template's	Reasoning `on`, `off` or `adaptive`, on any family
`reasoning_effort`	the template's	A reasoning level the template accepts, such as `low`, `medium` or `high`

preserve_thinking: true in chat_template_kwargs keeps earlier <think> blocks in the prompt on the Qwen3.6 and Gemma 4 templates, so an agent sees its earlier reasoning. A template applies at load, so two ids with different chat_template values are two loaded copies.

thinking and reasoning_effort map to the variable that each model's template reads, so one profile works across families:

Family	Template variable	Values
Qwen3.x, GLM	`enable_thinking`	`true` or `false`
MiniMax-M3	`thinking_mode`	Three states, so `adaptive` works
MiniMax-M3.1	None	Thinking is adaptive. In `gmlx chat`, `on` ends the prompt inside the thinking block
Kimi K2.x	`thinking`	`true` or `false`
Hy3	`reasoning_effort`	Levels that include `no_think`
gpt-oss, Beam	`reasoning_effort`	`low`, `medium` or `high`

Some templates have no thinking switch, such as gpt-oss, Beam, Muse, K2 Horizon and MiniMax-M3.1. On these, off closes each thinking block as soon as it opens. A thinking_budget you set wins over it.

A request's enable_thinking, thinking or reasoning_effort field wins over the profile. gmlx run and gmlx chat take --thinking and --reasoning-effort.

A request's thinking also takes the z.ai form, {"type": "enabled"} or {"type": "disabled"}. In that form, clear_thinking: false keeps earlier reasoning in the prompt.

Rules¶

Rules give a profile to every model whose id matches a pattern, so a group of models can share settings. The first rule that matches wins. A rule ranks below a model's own profile and above server.defaults.profile.

rules:
  - {match: "*coder*", profile: review}
  - {match: "qwen3.8-*", profile: agent}

Key	Default	Meaning
`match`	required	Model ids the rule applies to, as a shell wildcard pattern such as `qwen3.8-*`, not a regular expression
`profile`	required	The profile the matching models get

Sampling¶

A sampling block sets defaults for the request fields that control generation. It goes in a profile or in a model's overrides. Each key is also a request field of the same name, and the request's value wins. A default of "family" comes from the family defaults, and gmlx profiles <id> prints it.

profiles:
  precise:
    sampling:
      temperature: 0.2
      top_p: 0.9
      max_tokens: 4096
      stop: ["</answer>"]

Key	Default	Meaning
`temperature`	family	How random each token choice is. `0` always takes the most likely token. Also accepted as `temp`.
`top_p`	family	Sample only from the most likely tokens whose probabilities add up to this value. `0` turns it off.
`top_k`	family	Sample only from this many of the most likely tokens. `0` turns it off.
`min_p`	family	Drop tokens below this share of the most likely token's probability. `0` turns it off.
`max_tokens`	no limit	Stop after this many tokens. `gmlx serve --max-tokens` caps requests that set none.
`seed`	none	Repeatable sampling for this request, when its batch and the speculation setting also match
`stop`	none	A string, or a list of strings, that ends generation. The Anthropic endpoint uses its own `stop_sequences`.
`repetition_penalty`	off	Lowers the chance of tokens seen in the last `repetition_context_size` tokens. `1` has no effect.
`repetition_context_size`	`20`	Recent tokens that `repetition_penalty` looks at
`presence_penalty`	off	Lowers the chance of every token already generated, by the same amount
`frequency_penalty`	off	Lowers the chance of each token in proportion to how often it was generated
`xtc_probability`	off	Chance that XTC removes the most likely candidates, to vary the text
`xtc_threshold`	`0.0`	XTC removes candidates whose probability is above this value
`enable_thinking`	the template's	Whether the chat template opens a thinking block. A profile's `thinking` sets it on any family.
`thinking_budget`	no limit	Close the thinking block after this many reasoning tokens. `-1` sets no budget.
`thinking_budget_message`	none	Serve only. Text written in the thinking block ahead of the end marker when the budget runs out.
`thinking_start_token`	detected	Text that opens the model's reasoning. Family defaults set it for some families. Else gmlx reads it from the chat template or the vocabulary.
`thinking_end_token`	detected	Text that closes the model's reasoning. gmlx detects it with the start text.

When gmlx detects no markers, a response_format schema applies from the first token, and the thinking budget looks for <think>. Set both markers for a model whose reasoning text gmlx does not detect.

On a model with a built-in drafter, thinking_budget can run up to one draft round over, and a request that runs in a batch with other requests drops it. gmlx run and gmlx chat ignore thinking_budget_message. Thinking budget describes the request fields and the drafters.

Model loading¶

A load block changes how a model is built, for example to quantize its KV cache so that a long context uses less memory. It goes in a profile or in a model's overrides. Two ids that differ in a load key are two loaded copies. When you serve a single GGUF with no config, each key is also a gmlx serve flag, such as --kv-bits.

profiles:
  long-context:
    load: {kv_bits: 6, kv_quant_scheme: kvarn}

Key	Default	Meaning
`kv_bits`	none, `6` with `kvarn`	Bits per KV cache value. `uniform` takes 2, 3, 4, 6 or 8, and `kvarn` takes 2, 3, 4, 5, 6 or 8.
`kv_quant_scheme`	`auto`	`auto` picks for the model, `uniform` is affine, and `kvarn` is kvarn
`kv_group_size`	`64`	Group size of `uniform` quantization
`kv_tail_tokens`	`1024`	Newest tokens that stay fp16 under `kvarn`, as a multiple of 128
`quantized_kv_start`	`0`	Tokens the cache holds in fp16 before it quantizes. Batched requests and `kvarn` quantize from the first token.
`max_kv_size`	no cap	Most context tokens a request may use. `gmlx run` and `gmlx chat` also make the cache a rotating window of this size.

prefill_step_size and dtype apply to the whole server, under Scheduling.

Prompt cache¶

The prompt cache keeps the computed start of recent prompts, so a request that begins the same way skips that part of prefill. Agents and long chats gain the most, because they resend the same history every turn. server.cache sets it for every model, and a cache block in a profile or a model's overrides changes it. Prompt cache says what it restores for each architecture.

server:
  cache:
    enabled: true
    disk: {path: ~/.cache/gmlx/apc, max_gb: 100}

Key	Default	Meaning
`enabled`	`false`	Turns the prompt cache on. `gmlx init` writes `true`.
`block_size`	chosen per model	Tokens per cache block: 16, 32, 64, 128 or 256
`num_blocks`	grows to fit	Blocks in the pool that all cached prompts share. A full pool drops the oldest prompts.
`exact_entries`	`4`	Whole-prompt copies kept for hybrid and recurrent models. Each copy is a full cache.
`hash`	`fast`	How blocks are identified. `sha256` is stable across processes and costs more per token.
`disk`	`false`	A second tier on the SSD, which survives unloads and restarts. `true` uses the default `path`.
`disk.path`	`~/.cache/gmlx/apc`	Folder of the SSD tier, under `$XDG_CACHE_HOME` when that is set. Setting it turns the tier on.
`disk.max_gb`	no limit	Space each namespace may use, in GB. By default each model has its own namespace.
`disk.namespace`	the model path	The partition on disk that the model reads and writes
`disk.workers`	`1`	Threads that write entries to disk
`disk.read_mode`	`direct`	How entries are read back: `direct` with plain file reads, `mmap` by mapping the files

Model discovery¶

The discover block lists folders that the server scans at each start and reload. Every GGUF without an entry under models becomes a model, named after its file, so a new file appears after a restart or a reload. gmlx run, gmlx list and gmlx rm scan the same folders. To write the entries into the file once instead, run gmlx init or gmlx sync-models.

discover:
  - dir: ~/models
    recursive: true

The scan names each model after its file. It drops the shard suffix, markers such as mmproj, assistant, draft and mtp, and imatrix tags, and adds the short quantization: Qwen3-8B-Q4_K_M.gguf becomes qwen3-8b-q4. When two files would get the same name, both get the full quantization, such as qwen3-8b-q4-k-m and qwen3-8b-q4-k-s, and a clash that remains gets a number, as does a name that a model or an image model already has. gmlx list shows the names.

Key	Default	Meaning
`dir`	`null`	The folder to scan. `null` scans every folder in `server.model_dirs`.
`recursive`	`false`	Also scan subfolders
`pair_mmproj`	`true`	Pair each `mmproj*.gguf` with the quants of the model in the same folder that it matches, or with the quant folders below it
`speculative`	`auto`	`auto` and `true` turn on speculative decoding for MTP models and pair drafters, as the model's `speculative` does. `false` never does.

A companion pairs with each quant of its model in the same folder. With no model beside it, it pairs with every model in the folders one level down when they are all quants of one model, the layout of repositories that keep each quant in its own folder. A named companion, such as mmproj-Qwen3.8-27B-BF16.gguf, must match the model's name. A generic one, such as mmproj-F16.gguf, stays unpaired beside models that are not all quants of one model. Files are quants of one model when their names match past the quant and their headers give one arch and hidden size. A companion whose header gives another hidden size than the model's also stays unpaired. When a folder holds several companions, the first by file name wins.

A drafter GGUF becomes a model's draft_gguf when all of these hold:

It is in the same folder as the model, or in an MTP folder beside the model's folder or inside it.
Its architecture can draft for the model, and their hidden sizes match.
Its file name matches the model's, or the model is the only one in those folders that qualifies.

A drafter whose header names its base model pairs with that model in any scanned folder. A model with stream set gets no drafter.

An image DiT becomes an image model when the scan also finds its family's text encoder and VAE. Files are matched by their weights, not their names, and the nearest match to the DiT wins. A text encoder that is also a chat model, scanned or configured without adapters, is shared by its id; for the Qwen-Image models the chat model must also have its mmproj. A DiT without its encoder or VAE is reported with the gmlx pull command that fetches them.

Server¶

The server block sets where the server listens, who may call it, where models are found, how much memory they may use, and which extra services run. With no server block, the server listens on 127.0.0.1:8080 with the prompt cache off. The keys below are under server, as in the complete example.

Address and authentication¶

Key	Default	Meaning
`host`	`127.0.0.1`	The address to listen on. Any address other than loopback needs `api_key` or `no_auth`.
`port`	`8080`	The port to listen on
`api_key`	none	Key every request must send, as `Authorization: Bearer <key>` or `x-api-key: <key>`. Read only from this file.
`no_auth`	`false`	Accept an address other than loopback without a key, for a proxy that handles authentication
`media_urls`	`false`	Let a request name an image, audio or video by an `http(s)://` URL, which the server fetches
`cors_origins`	none	Web page origins that may call the server, such as `https://chat.example.com` or `http://192.168.1.20:3000`

The client commands chat, talk, ps, systemone, launch and launch menubar take --api-key to send the key. A change to api_key, media_urls or cors_origins applies after gmlx restart.

With media_urls on, any client that reaches the port can make the Mac fetch from any public host. The server still refuses hosts on the Mac and the local network.

Pages on localhost, 127.0.0.1 and [::1], and desktop apps built on Electron, Tauri or VS Code webviews, may always call the server. curl sends no origin and is not affected. For a browser extension, add the origin that the server log names, such as chrome-extension://<id>. These wildcards allow every extension of one browser:

chrome-extension://* for Chrome, Edge and other Chromium browsers
moz-extension://* for Firefox
safari-web-extension://* for Safari, which changes an extension's ID at every start

A listed origin can use the tools of served assistants, which run on the Mac, so prefer one extension's own origin to a wildcard.

Model folders¶

Key	Default	Meaning
`model_dirs`	none	Folders searched in order for a relative `path`, `mmproj`, `asr`, `draft_gguf` or adapter. `gmlx pull` downloads into the first.
`hf_cache`	`false`	Run the Hugging Face libraries offline, so the server never downloads. `gmlx init --from-hf-cache` sets it.

Services¶

Each service adds an endpoint. true picks the default model. Speech, embeddings and rerank lists the models each one accepts.

Key	Default	Meaning
`stt`	A served model that hears audio, if any	Speech-to-text model: an alias such as `whisper-turbo`, a Hugging Face repo id, a folder, or the id of a served model that hears audio. `false` turns it off.
`tts`	none	Text-to-speech model: an alias such as `kokoro`, a Hugging Face repo id or a folder
`embeddings`	none	Embeddings model: a GGUF path, an `hf:` reference, an alias, or the repo id or folder of an MLX encoder checkpoint
`rerank`	none	Reranker: a Qwen3-Reranker GGUF path, an `hf:` reference or an alias

Memory and residency¶

The server keeps several models loaded up to budget_gb, and unloads a model after ttl_s idle seconds or when the budget needs room. Memory explains which models stay loaded and when two ids share one loaded copy.

Key	Default	Meaning
`budget_gb`	0.8 of the GPU working set	Memory that loaded models may use together, in GB
`max_models`	no limit	Most models loaded at once, checked after the budget
`cache_limit_gb`	4 to 12 GiB, set at start	Limit of the MLX buffer cache, in GiB. `0` turns the cache off, a negative value removes the limit.
`defaults.model`	the only model, else none	Model for requests that name none, and the first one loaded when no model is pinned
`defaults.ttl_s`	`900`	Seconds without a request before a model that is not pinned unloads. `null` or `0` turns off the timeout.
`defaults.preload`	none	Models and image models loaded at start, one at a time, after the pinned models. `all` loads every chat model.
`defaults.profile`	none	Profile for every model, below rules and the model's own profile
`defaults.image_model`	the only image model, else none	Image model for image requests that name none

The MLX buffer cache says when to change cache_limit_gb.

Scheduling¶

These keys apply to every model. Each one except token_queue_timeout_s also has a gmlx serve flag.

Key	Default	Meaning
`prefill_step_size`	`2048`, `4096` for a sparse MoE from the M5 on when memory allows, `8192` when streaming, `4096` for HY4 when streaming	Prompt chunk size in tokens. A lower value lowers a long prompt's memory peak and slows prefill.
`dtype`	`auto`	Type of activations, unquantized weights and KV cache: `bfloat16`, `float16`, or `auto` (`float16` on M1 and M2)
`decode_prefill_ratio`	`auto`	How a new prompt's prefill shares the GPU with generating requests. `auto` keeps generation above half its speed.
`prefill_tick_ms`	`500`	Longest prefill chunk, in ms, while other requests generate. A longer chunk is halved, except on a streamed model. `0` turns off the halving.
`token_queue_timeout_s`	`1800`	Seconds a request may wait for its next token before it fails. `0` waits forever.

decode_prefill_ratio also takes a number, such as 1.0, which makes each prefill chunk wait until the generating requests have had that multiple of its GPU time. 0 runs one chunk per generation step. Concurrent requests shows the effect.

Features¶

Key	Default	Meaning
`family_defaults`	`true`	Start each request from its family defaults and offer the intents. `false` turns off both.
`stochastic_mtp`	`true`	Accept more speculative tokens by rejection sampling. Sampled output is not token-identical. A reload does not apply it.
`gpu_keepwarm`	on	Keep the GPU clock up between tokens of a streamed model with a decode feeder. `false` turns it off.
`menubar`	`true`	Let a server started in the background open the menu bar app
`cache`	off	Prompt cache settings for every model

Stochastic acceptance describes stochastic_mtp.

Image requests¶

These keys set how the server answers the image routes for the models under image_models.

Key	Default	Meaning
`image.url_ttl_s`	`3600`	Seconds a file behind a `url` response stays available
`image.url_max_bytes`	`1G`	Most bytes of such files kept at once, as bytes or a size such as `2G`. The oldest go first.
`image.max_queue`	`4`	Image requests that may wait behind the one running. One more gets 429 with `Retry-After`.
`image.embed_metadata`	`false`	Write the prompt and settings into each PNG the routes return
`image.tensor_cache_mb`	`64`	MB each image model keeps of encoded prompts, and again of reference image latents. The step routes keep as much of uploaded tensors.
`image.adapter_cache_mb`	none	Optional cap in MB on the request LoRAs of each image or video model, again on their merged patches and on step route uploads. Unset, LoRAs use free memory.
`image.steps_api`	`true`	Serve the image step routes that the ComfyUI node pack and `gmlx image bench --via-steps` call.
`image.steps_remote`	`false`	Take step route requests from clients beyond loopback. The routes take uploads of any bytes, so they answer loopback clients alone by default.

The memory gmlx counts for an image model includes its tensor caches at their full size: two of image.tensor_cache_mb, and a third while the step routes are on. A video model with an upsampler keeps room past image.tensor_cache_mb for one encoded caption as long as the upsampler writes, about 120 MiB for MiniMax-H3, so the ComfyUI nodes' encode of a long caption is not refused. Request LoRAs are counted when their jobs run, as the LoRA refusal describes, and a LoRA over a set image.adapter_cache_mb is refused with a message that names the key. At the defaults, one model of any family still fits on a 16 GB Mac at its default quant and size.

Video jobs¶

These keys set the job store of POST /v1/videos for the models under video_models.

Key	Default	Meaning
`video.max_queue`	`2`	Video jobs that may wait behind the one running. One more gets 429 with `Retry-After`.
`video.job_ttl_s`	`86400`	Seconds a finished job and its MP4 stay available from when the job finishes. A job still queued that long after its creation never runs.
`video.store_max_bytes`	`8G`	Most bytes of MP4s and job inputs kept at once, as bytes or a size such as `20G`. The oldest MP4s go first.

Structured decisions¶

These keys set up POST /v1/systemone, which Structured decisions describes. canvas, constrained and the three think keys apply only to DiffusionGemma. A reload applies new values to the next request.

Key	Default	Meaning
`systemone.model`	none	Model id or alias, with an optional `@profile`, for a request that names no configured model
`systemone.canvas`	`64`	Most tokens one structured read fills, as a positive multiple of 16
`systemone.constrained`	`true`	Compute probabilities over the answer tokens only. `false` uses the full vocabulary.
`systemone.max_questions`	`64`	Most questions in one request
`systemone.max_samples`	`32`	Most `samples` and `auto_max` a request may ask for. A larger value is lowered to this one.
`systemone.think`	`0`	Thought budget in tokens, from 0 to 4096, for a request that sets none. `"auto"` thinks only when an answer is uncertain.
`systemone.think_threshold`	`0.8`	Under `"auto"`, think when the chosen label's probability is below this value
`systemone.think_budget`	`64`	Under `"auto"`, the thought budget in tokens

Served assistants¶

The server can run the assistant's tool loop itself and offer each assistant as a model. A client that names the assistant as its model gets tools without a loop of its own, as Assistant describes.

server:
  assistants:
    helper:
      model: qwen3.8-27b-ud-q6
      mcp: []

Key	Default	Meaning
`assistants.*.model`	required	The configured model that answers for the assistant
`assistants.*.memory`	`false`	Long-term memory, in one store that all the assistant's clients share
`assistants.*.mcp`	`null`	Tool servers, with the keys of `assistant.mcp`. `null` uses `assistant.mcp`, `[]` gives no tools.
`assistant_allow_remote`	`false`	Allow served assistants on an address other than loopback. Their tools run on the server's Mac.

Voice¶

The talk block sets up the voice client gmlx talk and the menu bar app's voice sessions. Most keys also have a gmlx talk flag, which wins over the file.

talk:
  model: qwen3.8-27b-ud-q6@instruct
  voice: af_heart
  mode: vad
  vad: {silence_ms: 450}

Key	Default	Meaning
`model`	the server's default	Model id or alias to talk to, with an optional `@profile`
`voice`	the server's default	A Kokoro preset or a Qwen3-TTS speaker name
`speed`	`1.0`	Speaking speed, from 0.25 to 4
`system`	a prompt for speakable text	System prompt for the spoken persona. `null` or `""` sends none.
`language`	detected	Language hint for speech recognition, such as `en`
`max_tokens`	no limit	Most tokens in a spoken reply
`mode`	`wake`	How listening starts: `wake` on the wake phrase, `vad` on any speech, `ptt` on Space, `text` from typed prompts
`wake_word`	`hey assistant`	The wake phrase, any English text. It needs no training.
`wake_threshold`	`0.3`	Confidence, from 0 to 1, that the wake phrase needs. A higher value gives fewer false wakes.
`push_to_talk_modifier`	`globe`	Modifier held with Space for the menu bar hotkey: `globe`, `right-command`, `right-option` or `control`
`input_device`	the system input	Microphone, by part of its name or by index. `/devices` in `gmlx talk` lists them.
`output_device`	the system output	Speaker, by part of its name or by index
`chime`	`true`	Play a sound on wake and at the end of a turn
`brain`	`chat`	What answers: `chat` is the plain model, `assistant` adds the assistant block's tools and memory
`vad.threshold`	`0.6`	Speech probability, from 0 to 1, above which audio counts as speech
`vad.silence_ms`	`550`	Pause in ms that ends an utterance. Lower answers sooner and cuts off more sentences.
`vad.min_speech_ms`	`300`	Utterances shorter than this many ms are dropped as noise
`vad.pre_roll_ms`	`400`	Audio in ms kept from before speech starts

Assistant¶

The assistant block gives the built-in assistant its tools and memory. gmlx chat --assistant, gmlx talk with brain: assistant, and served assistants use it. The coding agents that gmlx launch connects do not. Assistant explains the tool loop and memory.

assistant:
  max_tool_rounds: 8
  mcp:
    - name: files
      command: [npx, -y, "@modelcontextprotocol/server-filesystem", "/Users/me/notes"]
    - name: search
      url: http://127.0.0.1:8931/mcp
  memory:
    top_k: 6

Key	Default	Meaning
`max_tool_rounds`	`8`	Most rounds of tool calls in one turn before the model must answer
`tool_timeout_s`	`60`	Most seconds one tool call may take
`mcp`	none	MCP servers that provide the tools. Each entry has a `name` and one of `command` and `url`.
`mcp[].name`	required	A name for the server, unique in the list
`mcp[].command`	none	Command that starts a server over stdio, as a list of arguments or a shell-style string
`mcp[].url`	none	Address of a server over streamable HTTP
`mcp[].env`	none	Environment variables for a stdio server, such as the token it needs
`memory.enabled`	`true`	Remember facts across conversations. Needs `server.embeddings`.
`memory.path`	`~/.local/share/gmlx/assistant-memory.db`	Database file of the memory, under `$XDG_DATA_HOME` when that is set
`memory.top_k`	`4`	Remembered facts added to each turn
`memory.extract`	`true`	Store short facts that the model extracts. `false` stores each exchange as it is.
`memory.ttl_days`	forever	Facts older than this many days are forgotten at start
`memory.max_items`	`20000`	Most facts the store holds

A stdio server gets only HOME, PATH, SHELL, TERM, USER and LOGNAME from your environment, so put anything else it needs in env. Its log goes to ~/.cache/gmlx/mcp-<name>.log.

Launch¶

The launch block configures container mode and defines custom agents. launch reads it only from the config file in your home folder, ~/.config/gmlx/gmlx.yaml or ~/.gmlx.yaml, never from a file that --config names. The server ignores the block.

Keys under launch.container apply to every client. Each one also works under launch.container.clients.<client> for a single client, where the client's value wins and lists from both levels add up. Six keys exist only under a client. Agents take most of these keys, plus five of their own.

This block turns on container mode with more memory, gives Claude Code a volume and a seed, runs Open WebUI from its official image with the served assistant home, and defines the agent research-bot:

launch:
  container:
    enabled: true
    memory: 6G
    clients:
      claude-code:
        volumes: [claude-pg:/var/lib/postgresql:8G]
        seed: [~/.claude/CLAUDE.md]
      open-webui:
        image: ghcr.io/open-webui/open-webui:v0.11.4
        command: image
        assistants: [home]
  agents:
    research-bot:
      runtime: python
      command: [research-bot]

`launch.container`¶

Key	Default	Meaning
`enabled`	`false`	Run clients in a container. `--container` and `--no-container` override it.
`mount_cwd`	`true`, `false` for `open-webui` and `elia`	Share the current folder read-write at the same path, and start the client there. `--mount-cwd` and `--no-mount-cwd` override it.
`mounts`	none	More folders to share, each `PATH[:DST][:ro]`. `PATH` is a full path or starts with `~`. `--mount` adds entries.
`volumes`	none	Named volumes, each `NAME:/path[:SIZE]`, created with `SIZE` when missing. The default size is `32G`.
`forward`	none	Ports the container reaches on its own `127.0.0.1`, which lead to the same ports on the Mac's `127.0.0.1`
`network`	`default`	`default` reaches the internet and your local network. `none` reaches only the gmlx server and forwarded ports. `--network` overrides it.
`cpus`	`4`	CPUs the container gets
`memory`	`4G`	Memory the container gets, such as `6144M`. It counts against the memory the model server can use.
`ssh_agent`	`false`	`true` lets the client use the SSH agent in `SSH_AUTH_SOCK`. A socket path gives it that agent instead.
`env`	none	Variables for the container: `NAME` passes yours in, `NAME=VALUE` sets one. An entry can replace a client setting such as `OPENAI_API_KEY`.
`open_browser`	`true`	Open a browser app in the Mac's browser once it answers. `false` prints the address.
`paste_copy_max`	`1G`	Largest pasted file that `launch` copies from another disk into the private home. A file on the same disk is cloned, with no limit.

For a variable that is part of the client's settings, such as OPENAI_API_KEY or ANTHROPIC_BASE_URL, an env entry NAME keeps launch's value. Only NAME=VALUE replaces it.

A volume under a client is separate for each project, while a volume at the top level is shared by every project and client. With ssh_agent, the client can sign with every key in the agent. To use 1Password's agent, give its socket path, ~/Library/Group Containers/2BUA8C4S2C.com.1password/t/agent.sock.

`launch.container.clients`¶

Settings for single clients, keyed by claude-code, opencode, pi, omp, hermes, goose, aichat, elia, open-webui or dsh. Each client takes the keys above and these six:

Key	Default	Meaning
`image`	the image gmlx builds	Run this image, a local tag or a registry reference. Not with `build` or `packages`.
`build`	the image gmlx builds	Build the image from this Containerfile, or from a folder with a `Containerfile` or `Dockerfile`. A full path or one that starts with `~`.
`command`	the client's own	A list replaces the client's command. `image` runs the image's own ENTRYPOINT and CMD.
`packages`	none	Debian packages added to the image gmlx builds. With `build`, only when the Containerfile starts from the client's `:base`.
`seed`	none	Files and folders copied from your home folder into the private home, at the same path
`assistants`	none	Served assistants the client can use. The others stay hidden from it.

Container images covers image, build and command. Here a client gets extra packages and two seeds:

launch:
  container:
    clients:
      claude-code:
        packages: [make, python3, postgresql-client]
        seed: [~/.claude/CLAUDE.md, ~/.claude/commands]

Seeds says when a seed is copied. --seed-instructions also seeds the client's instruction and skill files, which Instructions and skills lists. A served assistant's tools run on the Mac, so list assistants only for chat apps where you write the messages, as What the client reaches on the server explains.

`launch.agents`¶

Each entry under launch.agents defines a custom agent, such as research-bot in the example above. A name starts with a lowercase letter and holds lowercase letters and digits, with single - or _ characters between them, as in research-bot or ci_fixer2. It has at most 32 characters, does not end in - or _, and is not a client's name or menubar.

An agent needs command and one of runtime, image or build. It also takes every key of launch.container except enabled and clients, and the client keys image, build, seed and assistants. The table lists command, which an agent always needs, and the five keys only agents take:

Key	Default	Meaning
`command`	required	A list that starts the agent, or `image` for the image's own ENTRYPOINT and CMD. With `runtime`, the list runs in the project's environment.
`runtime`	none	`python` installs the project's dependencies with uv before the command runs. Without it, the command runs as it is in the image.
`source`	the current folder	The project folder that uv installs, shared read-only. A full path or one that starts with `~`. Only with `runtime`.
`api`	`openai`	`openai` sets the `OPENAI_` variables, `anthropic` the `ANTHROPIC_` ones, and `none` neither
`model`	the server's default	The served model in `GMLX_MODEL`. `--model` overrides it.
`web_port`	none	Port of the agent's web app in the container, which makes it a browser app. Without it, the agent runs in the terminal.

Custom agents covers dependencies at run time, the source folder, what the agent gets and a browser interface.

Chat themes¶

The theme and themes keys set the colors of gmlx chat.

theme: my-black
themes:
  my-black:
    extends: dark
    heading: {bold: true, rgb: "#88c0d0"}

Key	Default	Meaning
`theme`	`dark`	Theme each chat starts with: a built-in name or one from `themes`. `--theme` and `/theme` change it.
`themes`	none	Your own themes, keyed by name. A theme with a built-in name replaces it.

Each theme sets styles for kinds of text, such as heading and thinking, and takes extends for the ones it leaves out. Themes lists them.

Changing the file¶

gmlx checks the whole file each time it reads it. An unknown key fails the load and names the key, so a typo such as pinned: for pin: is caught before the server starts. Under sampling, load and cache, an unknown key prints only a warning and the load goes on, so read the server log after you change those blocks.

A running server reads its file again on POST /v1/reload or SIGHUP, and the commands that edit the file send the reload for you:

curl -X POST http://127.0.0.1:8080/v1/reload   # with api_key set, add -H "Authorization: Bearer <key>"
gmlx restart                                   # applies everything, such as a new api_key

Loaded models stay loaded through a reload. A change to a model's load settings, such as mmproj or speculative, applies the next time it loads. A reload also turns each service on or off, or moves it to another model.

Command	What it changes
`gmlx sync-models`	Adds new GGUFs from `model_dirs` and removes entries whose file is gone
`gmlx pull`	Adds an entry for each GGUF it downloads
`gmlx rm`	Deletes a model's files and its entry

These commands keep your comments and formatting. Pass --no-reload to init, sync-models or rm to leave a running server alone, and --no-register to gmlx pull to leave the file alone.

Flags and environment variables¶

Some settings also have a gmlx serve flag or an environment variable. The flag wins, then the key in the file, then the variable. Two variables win over the file: GMLX_CACHE_LIMIT_GB over server.cache_limit_gb, and GMLX_MTP_WIDTH_CAP over each model's speculative_width_cap. The flags are in the CLI reference and the variables in Environment variables.

Complete example¶

This file uses the blocks most setups need:

server:
  model_dirs: [~/models]
  budget_gb: 96
  cache:
    enabled: true
    disk: {path: ~/.cache/gmlx/apc, max_gb: 200}
  defaults:
    model: qwen3.8-27b-ud-q6
    ttl_s: 900
profiles:
  review:
    extends: coding
    system: "You are a terse senior engineer."
    sampling: {temperature: 0.2, max_tokens: 4096}
  long-context:
    load: {kv_bits: 8}
rules:
  - {match: "*coder*", profile: review}
models:
  qwen3.8-27b-ud-q6:
    path: Qwen3.8-27B-UD-Q6_K.gguf
    speculative: true
    pin: true
    profiles:
      coding: {sampling: {min_p: 0.05}}         # changes @coding for this model
    overrides: {sampling: {max_tokens: 8192}}   # wins over every profile
  gemma-31b:
    path: google_gemma-4-31B-it-Q6_K_L.gguf
    draft_gguf: gemma-4-31B-it-assistant.Q8_0.gguf
  gemma-e4b:
    path: gemma-4-E4B-it-Q6_K.gguf
    mmproj: mmproj-gemma-4-E4B-it-bf16.gguf
    profile: long-context
aliases:
  coder: qwen3.8-27b-ud-q6@review
assistant:
  mcp:
    - {name: clock, command: [uvx, mcp-server-time]}
discover:
  - {dir: null, recursive: true}   # serves GGUFs in model_dirs that have no entry

The smallest useful file has one model with a path:

models:
  my-model:
    path: ~/models/my-model-Q4_K_M.gguf