Vision and audio¶

A multimodal GGUF model reads images or audio as well as text. It comes as two files that gmlx pairs, and it takes media from the command line, from chat and from the server. Most features of the language model still work when a turn carries media.

The two files¶

A multimodal model in GGUF is two files. The language model GGUF is quantized as usual, and a companion mmproj GGUF holds the encoder that turns images or audio into tokens for the language model. Hugging Face repositories ship the companion as an mmproj-*.gguf file beside the language model. Download both in one command, where the second name is a file in the same repository:

gmlx pull hf:unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q6_K.gguf mmproj-BF16.gguf

A model pulled by name, as in gmlx pull qwen3.8-27b, comes with its companion unless you pass --no-vision. A model that runs from the SSD takes its companion in run and serve, not in chat.

Pick the BF16 or F16 companion when the repository has one, because it reads media most accurately. A quantized companion, such as mmproj-Q8_0.gguf, takes about half the memory. Use it only when the model does not fit otherwise.

When both files land in a folder of server.model_dirs, pull registers the model with its companion and prints (vlm) after the id. For a model that is already registered, add the models.*.mmproj key to its entry, or run gmlx sync-models --add-companions. discover[].pair_mmproj pairs the files that a folder scan finds, including a companion at the root of a repository whose quants each sit in their own folder.

--mmproj pairs the two files for one command, and --no-mmproj loads a registered model without its companion. gmlx validate recognizes a companion and says to pair it with its language model. The language model alone still runs as a text model, and media support needs no optional package.

gmlx builds the image processor and the chat template from the two files' metadata. For a family whose files leave something out, --hf-source names a Hugging Face repository, and the whole processor, including the tokenizer and chat template, then comes from that repository.

Sending media¶

# One reply about an image file or URL.
gmlx run model.gguf --mmproj mmproj.gguf --image photo.jpg --prompt "What is this?"

# A chat, with /image, /audio, or a file dragged into the prompt.
gmlx chat model.gguf --mmproj mmproj.gguf

# A server for one model and its companion.
gmlx serve model.gguf --mmproj mmproj.gguf

run takes local files and URLs, and chat takes local files. Image files can be PNG, JPEG, GIF, WebP, BMP, TIFF or HEIC, and audio files can be WAV, MP3, FLAC, M4A, Ogg, Opus or AIFF. A request to the server carries images as Media in requests describes. The flags are listed under gmlx run.

Each family resizes images to its own limits before encoding. The size sets how many tokens an image becomes, and those tokens make up most of the time to read a prompt with images. --resize-shape, such as 448 or 672x448, shrinks images first, which helps when speed matters more than fine detail.

Audio works in the same way on a companion with an audio encoder, such as gemma-4 E-series, Qwen3-Omni and Muse Spark. --audio on run and /audio in chat attach a clip to the next turn.

Nemotron 3.5 Transcribe hears audio through its own speech encoder, which pairs with the model when the two files share a folder. Nemotron 3.5 Transcribe shows how to transcribe, translate and label speakers with it.

Supported families¶

The companion's metadata names its projector. When several families share a projector, the language model's architecture tells them apart. A pairing that gmlx does not support fails at load, and the error names the projector and architecture that it found. gmlx supports these families:

Family	Projector and architecture	Examples	Notes
LLaVA-1.5	`has_llava_projector`	llava-1.5-7B	It needs `--hf-source llava-hf/llava-1.5-7b-hf`, because the GGUF has no image processor.
Pixtral	`pixtral`	Mistral-Small-3.x, Pixtral-12B	Vision quality is poor, because the published companion files were converted with a defect.
Qwen3-VL	`qwen3vl_merger` with `qwen3vl`	Qwen3-VL-8B-Instruct	Each image becomes 64 to 4096 tokens. The vision encoder runs in float32.
Qwen3.5 and Qwen3.6	`qwen3vl_merger` with `qwen35` or `qwen35moe`	Qwen3.5-VL-9B, Qwen3.6-VL	`GMLX_QWEN35_VISION_FP32=1` runs the vision encoder in float32.
Qwen3.8-Flash-Next	`qwen3vl_merger` with `qwen4exp`	Qwen3.8-Flash-Next
Qwen3-Omni	`qwen3vl_merger` with `qwen3vlmoe`	Qwen3-Omni	It reads images and audio.
gemma-4 E-series	`gemma4v`, `gemma4a`	gemma-4-E2B, gemma-4-E4B	It reads images and audio.
gemma-4 unified	`gemma4uv`, `gemma4ua`	gemma-4-12B	It reads images, and audio when the companion has an audio encoder.
GLM-5.3-Flash	`glm5v` with `glm5-next`	GLM-5.3-Flash	The first uploads name both files `glm5next`, and those load too.
Muse Glimmer	`muse-glimmer`	Muse-Glimmer-30B
MiniCPM-V 4.6 and 4.7	`minicpmv4_6` or `minicpmv4_7` with `qwen35` or `qwen35moe`	MiniCPM-V-4.6, MiniCPM-V-4.7-35B-A3B, MiniCPM-V-4.7-1B	Video input is refused.
Nemotron 3.5 Bolt and Super VL	`nemotron_omni` with `nemotron_h_moe`	NVIDIA-Nemotron-3.5-Bolt-74B-A9B, NVIDIA-Nemotron-3.5-Super-VL-120B-A12B	Each image becomes 256 to 3328 tokens. Video input is refused.
MiniMax-M3.1-Flash	`minimax_m3` with `minimax-m3`	MiniMax-M3.1-Flash	Each image becomes 8 to 576 tokens. Video input is refused. It is larger than RAM on most Macs, so it needs `--stream-experts`.
Nemotron 3.5 Transcribe	`pee` with `nemotron_h_moe`	NVIDIA-Nemotron-3.5-Transcribe-30B-A4B	It reads audio only, up to six clips and an hour in all.
Muse Spark	`muse-glimmer` and `muse-spark-audio` with `muse-spark`	Muse-Spark-Open-816B-A42B	It reads images and audio. It is larger than RAM on most Macs, so it needs `--stream-experts`.
Kimi K2.5 and K2.7	`kimik25` with `deepseek2`	Kimi-K2.5, Kimi-K2.7-Code	It is larger than RAM on most Macs, so it needs `--stream-experts`.
DeepSeek-V4-Flash-Vision-Exp	`deepseek4v` with `deepseek4`	The unsloth UD builds
DeepSeek-V4.1-Flash-Vision	`deepseek4-vision` or `deepseek41v` with `deepseek41`	The antirez encoder GGUF, the unsloth DeepSeek-V4.1-Flash-1017 companion	unsloth puts the companion above the quant folders, and a folder scan pairs it with each quant. `--mmproj` works too.

Qwen2-VL and Qwen2.5-VL companions, with the projector qwen2vl_merger, are not supported, and the load fails with a message that names the family. On LLaVA, the loader reports two unfilled post_layernorm parameters, which is expected, because LLaVA does not use them.

MiniCPM-V repositories ship mmproj-model-f16.gguf, which pairs on its own. The 4.7 repositories also ship mmproj-model-f16-4x.gguf, which turns each image into four times as many tokens for finer detail. Pass it with --mmproj to use it. A 4.7 repository holds the model, the companion and a DSpark drafter. For the 35B-A3B, one pull fetches all three:

gmlx pull hf:openbmb/MiniCPM-V-4.7-35B-A3B-gguf \
  MiniCPM-V-4_7-35B-A3B-Q4_K_M.gguf mmproj-model-f16.gguf \
  dspark-MiniCPM-V-4_7-35B-A3B-F16.gguf

For the 1B, pull the model and the companion:

gmlx pull hf:openbmb/MiniCPM-V-4.7-1B-gguf \
  MiniCPM-V-4_7-1B-Q4_K_M.gguf mmproj-model-f16.gguf

The 1B's DSpark drafter decodes slower than the model alone on Apple silicon, so gmlx leaves it off, and --draft-gguf still loads it.

On DeepSeek-V4-Flash-Vision-Exp, image turns need a KV cache without quantization, so --kv-bits applies to text turns only, and the server runs image turns one at a time. Each image becomes a block of up to 384 tokens. The prompt cache recognizes these blocks, so a conversation that repeats its earlier image turns unchanged reuses the cache. On V4.1, each image becomes up to 1024 tokens, and image turns have no such limits.

Media with other features¶

Most features of the language model work with media. These combinations have limits:

Combination	Result
Speculative decoding	Text turns use speculation. Media turns use it on the server, and decode without it in `run` and `chat`.
`--stream-experts`	`run`, `chat` and `serve` stream the language model and keep the vision encoder on the GPU. Text turns keep speculation, except on a server.
`--stream-cpu`	The command refuses it, because it would move the vision encoder to the CPU too.
The lossy MoE settings, such as `--moe-experts`	Chat refuses them with `--mmproj`.
`--adapter`	The command refuses it, because a LoRA adapter applies only to text models.
`--stop`, `--xtc-probability`, `--xtc-threshold`	`run` ignores them with a warning.

Chat and the server keep each image with the user message that sent it, so a later question about an earlier image reads the right history. Once a conversation holds media, chat reads the whole conversation and encodes the media again on every turn. The server reuses the cached earlier turns when a turn adds an image, on Gemma-4 and on the qwen35 and qwen35moe architectures.