gmlx¶

gmlx runs GGUF models on Apple Silicon. It runs the K-quant and IQ-quant files that the community publishes, unchanged, on Metal kernels for Apple's MLX framework. One command, gmlx, chats with a model in the terminal, serves it over OpenAI and Anthropic compatible APIs, connects coding agents to it and talks to it by voice.

These pages are the starting points:

To run your first model, follow the Quickstart.
To serve models to apps and agents, read Configuration and Agents and chat apps.
To look up a flag, a key or an endpoint, use the CLI reference, Configuration or the HTTP API.
To work on gmlx itself, start with Internals.

Getting started¶

Installation: Homebrew, uv and pip, the optional features, upgrading and removal.
Quickstart: A first model, the server, a request and a connected client.
Migrating from other tools: What carries over from llama.cpp, Ollama and LM Studio.

Serving¶

Configuration: gmlx.yaml, its models, its profiles and how a request gets its settings.
Agents and chat apps: Claude Code, other coding agents and Open WebUI, set up by gmlx launch.
Menu bar app: Server status and controls in the macOS menu bar.
Speech, embeddings and rerank: The services a server can host beside chat models.
RAG pipelines: Retrieval with the embeddings and rerank services.
Structured decisions: A probability for each answer to a fixed set of questions.

Clients¶

Chat: The terminal chat client, its commands, sessions and themes.
Voice chat: Talking to a model with gmlx talk.
Assistant: Tools and long-term memory for chat, voice and served models.

Models¶

Supported architectures: The GGUF architectures gmlx loads, with their caveats.
Vision and audio: Multimodal models and their mmproj files.
Models larger than memory: Mixture-of-experts models that stream their experts from disk.
LoRA adapters: Training an adapter and serving several on one base model.
Distillation: Teaching a small model a document or a larger model's behavior.

Performance¶

Performance tuning: What makes a model fast, measuring, and choosing a quant.
Speculative decoding: Faster decoding with a drafter, with the same output.
Prompt cache: Skipping prefill for prompts the server has seen.
Concurrent requests: Batching, admission pacing and shared prompts.
Memory and the KV cache: How much memory a model and its context take.
KV cache quantization: Storing the context in fewer bits.
Benchmarks: gmlx against llama.cpp on the same files, with the method.

Help¶

Troubleshooting: gmlx doctor, common failures, and where gmlx keeps its files.
Glossary: The terms these pages use.

Reference¶

CLI reference: Every command and flag.
Configuration: Every key of gmlx.yaml.
Family defaults: The sampling defaults and intents of each model family.
HTTP API: The endpoints and request features.
Environment variables: The variables a user can set.
Python API: Using gmlx from Python.

Development¶

Internals: How gmlx works inside.
Adding a GGUF architecture: What supporting a new model family involves.
Contributing: Development setup, tests and commit style.
Changelog: What changed in each release.