Local inference on a Mac is straightforward until several requests overlap. Then time to first token, memory growth, and admission control become serving problems rather than model-execution problems. vllm-metal brings vLLM’s scheduler, paged KV cache, and OpenAI-compatible server to Apple Silicon, with MLX and Metal handling execution.

v0.4.0 adds batched MTP, broader model and workload support, and automatic prefill acceleration on M5. On SiliconBench’s agent split, vllm-metal keeps TTFT flatter as concurrency rises while serving from a fixed memory budget.

How vllm-metal fits into vLLM

vllm-metal is a plugin, not a fork. Upstream vLLM provides the V1 scheduler, paged KV block management, chunked prefill, sampling, and the OpenAI-compatible frontend with streaming and tool-call parsing, including its new Rust implementation. mlx_lm provides the model implementations and MLX executes them. The plugin connects the two, with most of its model-specific code concentrated in one layer.

Architecture overview: clients speak the OpenAI API to upstream vLLM's frontend and V1 scheduler, which hand the vllm-metal model runner a packed step plus block tables; the runner reuses mlx_lm's token-wise layers and adds custom Metal paths for paged varlen attention, MTP, and M5 NAX prefill, all executing through MLX and Metal on Apple Silicon unified memory

Start an OpenAI-compatible server

Install vllm-metal into its own virtual environment and activate it:

curl -fsSL https://raw.githubusercontent.com/vllm-project/vllm-metal/main/install.sh | bash
source ~/.venv-vllm-metal/bin/activate

The installer adds the plugin, vLLM core, and their dependencies to ~/.venv-vllm-metal.

Then launch a model:

# --gpu-memory-utilization caps vLLM's share of unified memory; see below.
vllm serve Qwen/Qwen3.5-0.8B --gpu-memory-utilization 0.5

# 64 GB Macs: the 27B hybrid
# vllm serve mlx-community/Qwen3.8-27B-8bit --gpu-memory-utilization 0.7

# Speculative decoding: Gemma 4 with its MTP assistant
# vllm serve google/gemma-4-E4B-it --gpu-memory-utilization 0.5 \
#   --max-model-len 16384 --no-async-scheduling \
#   --speculative-config '{"method":"mtp","model":"mlx-community/gemma-4-E4B-it-assistant-bf16","num_speculative_tokens":1}'

More models: model matrix. Speculative options: speculative decoding guide.

The server speaks the OpenAI API:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3.5-0.8B",
       "messages": [{"role": "user", "content": "Say hi"}]}'

Anything that takes an OpenAI-compatible base URL can point at http://localhost:8000/v1, coding agents included; the vLLM docs cover Claude Code and Codex setup.

Set a predictable memory budget

vllm-metal reserves its KV cache at startup and serves every request from that fixed pool. If you are used to runtimes whose memory footprint moves with load, this is the main mental-model change. The key setting is --gpu-memory-utilization: it sets the share of the Mac’s GPU memory budget used for serving.

On Apple Silicon there is no separate VRAM: GPU allocations come from the same unified memory used by macOS, your browser, and your editor. Treat the setting as a serving budget, not a hard cap on the process. As a starting point, 0.5 keeps a laptop usable while it serves; a dedicated machine can go higher. The same reasoning applies to any unified-memory system vLLM runs on, including DGX Spark.

The reservation buys admission control. The scheduler knows exactly how many KV pages exist, packs requests against that budget, and queues what does not fit, so a burst of requests changes queue depth rather than memory footprint.

At startup, vllm-metal accounts for model weights and temporary GPU buffers before sizing the KV pool, then limits how much temporary memory MLX can retain (PR #268). This keeps serving memory predictable even as batch shapes change.

Concurrent serving under agent load

Agent workloads are the concurrency case. A coding agent fans out tool calls, each one a request with a few thousand tokens of context and a short reply, several in flight at once, and the latency the user feels is time to first token, paid on every round trip.

Qwen3.8-27B

We measured this shape with the agent split of SiliconBench, our benchmark harness for LLM inference engines on Apple Silicon: 100 requests averaging 4.6K input and 70 output tokens, run closed-loop at concurrency 1, 2, and 4 against Qwen3.8-27B in 8-bit on an M5 Pro with 64 GB. oMLX appears twice because its default mode spills KV cache to SSD without eviction, which no other engine here does; we report it both with that offload and in its bounded in-memory mode.

SiliconBench agent split on Qwen3.8-27B: TTFT, output token throughput, and inter-token latency versus concurrency for llama.cpp, vllm-metal, and oMLX with and without SSD offload

  • At concurrency 1, the three bounded engines sit within 0.7 s on roughly 10-second TTFTs; oMLX’s SSD-offload mode is fastest at 7.8 s.
  • As requests overlap, vllm-metal’s TTFT rises from 10.1 s to 14.7 s while completing all 100 requests at every level. llama.cpp also completes every request, about one second behind at concurrency 4. oMLX’s SSD mode reaches roughly 27 s, while bounded oMLX rejects 37 of 100 requests.
  • The tradeoff is inter-token latency: 534 ms for vllm-metal versus 293 ms for oMLX’s SSD-offload arm at concurrency 4. Chunked prefill favors starting new requests by sharing each engine step with active decodes, which suits agent traffic with short outputs and repeated TTFT costs.

Gemma 4 E4B

Gemma 4 E4B is small enough that the same machine sweeps to concurrency 16, on the same agent split.

SiliconBench agent split on Gemma 4 E4B: TTFT, output token throughput, and inter-token latency versus concurrency for llama.cpp, vllm-metal with and without the MTP drafter, and oMLX in its default SSD-offload mode

At concurrency 16, vllm-metal averages 1.9 s TTFT, against 14.7 s for oMLX and 28.3 s for llama.cpp, and generates 63.6 output tokens per second, against 50.4 and 40.2. The dashed MTP arm raises vllm-metal to 71.6 output tokens per second and is analyzed below.

llama.cpp uses its default --parallel 4 configuration; --parallel 16 is not uniformly better (sensitivity results). That default also explains its flat inter-token latency: at concurrency 16 it decodes four streams and queues the other twelve, the same choice that puts its TTFT at 28.3 s. Stats cover completed requests: vllm-metal fails the same 7 of 100 prompts at every concurrency level, and oMLX the same 6. Those failures track the prompts rather than the stack. Upstream vLLM on a DGX Spark, running CUDA kernels against the same weights, returns empty completions for the same prompts, 6 of 100 there, and every one of them is a very long Hermes trace that the model answers with an immediate stop token. llama.cpp is the one engine here that completes all 100. oMLX ran only its default configuration, with SSD KV offload on; there is no bounded-memory arm for this model.

Replacing attention, reusing the rest

At the model level, vllm-metal reuses mlx_lm’s weight loading, RMSNorm, linear, MoE, and MLP layers unchanged. Those layers are token-wise: they process each token independently and do not care how tokens are grouped into sequences, so they run as happily on a packed token axis as on a padded one. Attention is the only layer that needs sequence boundaries, and it is the one we replaced with a custom paged varlen flash-attention Metal kernel.

mlx_lm’s stock attention runs on a contiguous cache of shape [B, H, T, D], with every sequence padded to the length of the longest and prefill and decode handled as separate phases. MLX’s scaled_dot_product_attention accepts no varlen argument.

vllm-metal instead flattens each step onto a single token axis with cu_seqlens marking request boundaries, the packed-query layout vLLM’s unified Triton kernel consumes on NVIDIA. KV remains in fixed-size pages located by per-request block tables, so admitted requests can grow without reshaping a padded cache. One kernel launch covers whatever the V1 scheduler packed into the step: prefill chunks, decode tokens, and speculative-decoding verify windows.

The swap flattens the whole forward pass, not only attention, so the scheduler’s packing reaches every layer. MLP and MoE blocks compute only real tokens, while a padded engine pushes padding rows through the network.

A padded rectangle versus vllm-metal's packed varlen step for a 30,000 + 5,000 + 10 token batch, with KV read from paged storage

Among the serving stacks we audited on Apple Silicon, this pairing of a packed query axis with paged KV storage is what distinguishes vllm-metal:

Engine Encoding Query axes KV
mlx_lm padding [B,T_max] contiguous
oMLX padding [B,T_max] contiguous
llama.cpp mask [total_q] fixed cells
vllm-metal cu_seqlens [total_q] paged

A padded rectangle spends memory and attention time on the longest sequence in the batch whether or not the others need it. A concurrent mix that fits comfortably on a large machine can push a smaller one into macOS memory compression and slow down without an explicit error. Sizing one paged pool up front removes that failure mode.

The Metal kernel is a port of vLLM’s unified Triton kernel, down to the binary search each threadgroup runs over cu_seqlens to find which request owns its query token.

What v0.4.0 adds

Batched MTP under concurrent load

The same packed batch structure also supports batched speculative decoding. vllm-metal’s Gemma 4 MTP assistant drafts one token per request per step, and the draft pass itself is batched: every request’s seed runs in one grouped varlen dispatch against the target’s shared KV cache. Verification runs in the next packed target batch. Speculation never leaves the continuous-batching path, so it can be measured on the same agent split as everything above; the dashed blue line in the Gemma 4 figure is this arm. Change against the vllm-metal baseline:

c Wall Output tok/s ITL avg TTFT avg Acceptance
1 +1% −3% −13% +5% 73.4%
8 −21% +23% −36% +15% 73.5%
16 −12% +13% −25% +16% 73.3%

Single-stream, MTP is a wash on this model: the drafter’s cost roughly cancels the accepted tokens. The gains appear where speculation is usually turned off: at concurrency 8 it cuts inter-token latency from 152 ms to 97 ms and total wall time by a fifth, and acceptance holds at 73% from one stream to sixteen. Today the Metal MTP path is limited to Gemma 4 and requires --no-async-scheduling; the quickstart above includes both. In this comparison, the cost is TTFT, up 15% under load. MTP is opt-in through --speculative-config, so prefill-dominated deployments leave it off.

Faster prefill on M5

On M5 Macs, vllm-metal automatically uses the NAX kernel for compatible prefill batches; no extra configuration is required. Pre-M5 Macs keep using the existing path.

NAX prefill kernel A/B benchmark: TTFT, throughput, and TPOT with the tensor units on and off

NAX cuts mean TTFT by 41% on the prefill-heavy split and 26% on the standard split, while total throughput rises 33% and 8%. It also lowers TPOT by 25% and 7% because faster chunked prefill returns time to active decode streams.

Prefix caching on hybrid models

vLLM 0.28 turns prefix caching on by default for hybrid models, which were held out while the feature matured. That covers the Qwen3.5 family, whose layers alternate standard attention with gated-delta-net linear attention. Linear attention carries recurrent state that a block table cannot address the way it addresses KV, so reusing a prefix means checkpointing that state at block boundaries. Upstream calls this align mode.

On Metal that default would have arrived as a regression. Under align mode on Qwen3.5-0.8B, eighteen linear layers share six state pools, each about 1.6 GiB once roughly 1,650 blocks are cached, and every MLX assignment path copies the whole pool before writing a single row. v0.4.0 writes those rows in place through a custom MLX primitive instead (PR #634). Shared-prefix traffic moves from 1.03x to 1.26x faster, and traffic with nothing in common moves from 2.24x slower back to neutral, which is the behavior a dense model already had.

The feature is still experimental on Metal. It needs the paged attention path, which is the default, and it does not yet coexist with speculative decoding, because rolling draft state back across mamba state blocks is not implemented. In either case the plugin turns prefix caching back off and logs the reason rather than refusing to start.

Models, formats, and deployment modes

v0.4.0 also adds:

  • GGUF checkpoints, including Hugging Face config sources for local GGUF weights.
  • Hybrid-attention models: the Qwen3.5, 3.6, and 3.8 family and Qwen3-Next alternate standard attention with gated-delta-net linear attention; v0.4.0 serves mlx-community/Qwen3.8-27B-8bit on a single Mac.
  • Pipeline parallelism across multiple Macs over the MLX ring backend.
  • Experimental vision-language models, text embeddings and reranking, and speech-to-text.

The supported-model matrix and feature guides are in the vllm-metal documentation.

The same stack on DGX Spark

Apple Silicon is one unified-memory target, not the only one vLLM serves from. NVIDIA’s DGX Spark shares memory between CPU and GPU the same way, takes the same --gpu-memory-utilization budget, and runs the same V1 scheduler, chunked prefill, and paged KV blocks. What changes underneath is CUDA kernels rather than MLX and Metal, and upstream vLLM rather than the plugin.

The two are less lopsided than they look:

  Apple M5 Pro DGX Spark (GB10)
Unified memory 64 GB LPDDR5X-9600 128 GB LPDDR5X-8533
Memory bandwidth 307 GB/s 273 GB/s
GPU shader lanes 20 cores × 128 = 2,560 48 SMs × 128 = 6,144
Serving stack vllm-metal (MLX + Metal) upstream vLLM (CUDA)

The memory subsystems are the same design: LPDDR5X on a 256-bit bus, with both bandwidth figures falling straight out of the data rate. The laptop’s memory is clocked higher, so the laptop has more bandwidth. The shader blocks are the same width as well: an Apple GPU core and a Blackwell SM are both 128 lanes. So the counts compare directly, and the Spark has 2.4 times as many of them, plus twice the memory to put a model in.

We are deliberately not quoting peak TFLOPS for either machine. Published figures are not reachable on either side, and independent measurements of GB10’s dense BF16 throughput vary by roughly an order of magnitude depending on thermal and power state. The benchmark below is the compute comparison.

That split predicts where each should win, because the two halves of serving are bound by different things. Decode is bandwidth-bound: every token reads the whole weight set, so per-stream token rate tracks GB/s. Prefill is compute-bound: a 4.6K-token prompt is one large matmul. The agent split is prefill-heavy by construction, so wall time and throughput should follow compute to the Spark, while inter-token latency is where bandwidth answers back.

SiliconBench agent split across three models, vllm-metal on an M5 Pro against upstream vLLM on a DGX Spark: TTFT, output token throughput, and inter-token latency versus concurrency

That is what happens on the throughput side. At Qwen3.5-0.8B the two machines sit within 8% on a single stream, 76 against 82 output tokens per second, and separate to 297 against 685 by concurrency 16. Gemma 4 E4B goes 64 against 225 at that same point. Qwen3.8-27B, where the Mac serves an 8-bit MLX conversion and the Spark serves Qwen’s own FP8 checkpoint, goes 5.3 against 24.0 at concurrency 4. Time to first token separates furthest, since prefill is where the compute gap lives: on the 27B it is 14.7 s against 0.6 s.

Inter-token latency is the column that runs the other way, and it does so only on a single stream. Decoding one sequence reads the entire weight set for every token, which is the bandwidth case the table predicts, and the Mac takes it on Gemma 4 E4B, 43.0 ms against 56.5, and on the 27B, 108 against 139. That advantage does not survive batching. A batched step reads those same weights once on behalf of every sequence in flight, so decode stops tracking bandwidth and starts tracking compute: by concurrency 2 on the 27B the Mac is at 287 ms against the Spark’s 142, and by concurrency 4 it is 534 against 151. Across its whole sweep the Spark moves from 139 ms to 151. The Mac’s rises fivefold over the same three points.

None of that is surprising for a dedicated box with twice the memory, measured against a laptop that was also running an editor. The useful part is what does not change. The scheduler makes the same admission decisions on both, --gpu-memory-utilization means the same thing on both, and a request that fits the KV budget on one fits it on the other. The workload you develop against a Mac is the workload that runs on the Spark, and the ceiling you hit is the hardware’s rather than the serving layer’s.

Reproducing the benchmarks

The cross-engine serving benchmarks use the SiliconBench agent split: 100 prompts averaging 4.6K input and 70 output tokens, run closed loop at fixed concurrency on an Apple M5 Pro with 64 GB running macOS 26.6. The NAX A/B instead uses the two Sonnet configurations described above: 100 prompts at request rate 10 and concurrency 32.

The DGX Spark comparison reuses that split unchanged, on a GB10 box with 121 GB of usable unified memory running the same harness and the same client.

Stats cover completed requests; an empty response counts as failed. The harness and per-engine configurations live in the SiliconBench repo. The MTP arms ran vllm-metal 0.3.0.dev20260821152549 with the serve command from the quickstart.

Serving benchmark reproduction settings
  • llama.cpp: -ngl 99 --parallel 4 -c 49152. The context is divided across slots, giving 12,288 tokens per slot; the agent split’s longest prompt is 8.7K tokens.
  • vllm-metal (27B): VLLM_METAL_MEMORY_FRACTION=0.7 with --max-model-len 16384. Prefix caching was enabled explicitly with --enable-prefix-caching, which these runs predate needing; vLLM 0.28 turns it on for hybrid models by default. At the default fraction of 0.5, the available KV cache held 36,408 tokens and preemption began at concurrency 4.
  • oMLX: --paged-ssd-cache-dir <fresh-empty-dir> and --hot-cache-max-size 0, with a restart and a new directory before each concurrency level. Its default 100 GB prefix cache persists in ~/.omlx/cache, while CLI values persist in ~/.omlx/settings.json. The model directory contained only the target checkpoint because oMLX auto-discovers every entry and returns them in ASCII order. At concurrency 4, its bounded-memory admission guard rejected 37 of 100 requests, so the main chart reports a failure count rather than survivor-only latency.
  • vllm-metal MTP: --no-async-scheduling with "num_speculative_tokens":1. Without the scheduling flag, server health and /v1/models succeed, but inference returns HTTP 500. Values above 1 are ignored rather than rejected, so they do not test a wider speculation window.
  • DGX Spark: upstream vLLM 0.27.2rc1.dev568+gf25c580af built from source against torch 2.13.0+cu130 and flashinfer 0.6.17, on GB10 with driver 580.159.03, CUDA 13.0, Ubuntu 24.04 and kernel 6.17. Same harness, same agent split, same 100 prompts, --max-model-len 16384 with prefix caching on by default. vLLM selects its FLASH_ATTN backend on this GPU, not FlashInfer. The 27B arm serves Qwen/Qwen3.8-27B-FP8, which is not the checkpoint the Mac serves.
  • --gpu-memory-utilization on the Spark: the 27B needs an explicit 0.7, matching the fraction the Mac uses for the same model. The knob behaves exactly as the memory section above describes, which is the problem: GB10 shares one 121 GB pool between the GPU and the operating system, so vLLM’s default 0.9 reserves around 109 GB of it. Small models absorb that (Qwen3.5-0.8B took a 102 GiB KV cache and served fine), but 28.5 GiB of 27B weights on top of it exhausted the machine, the driver returned NV_ERR_NO_MEMORY, and the desktop session died. At 0.7 the KV pool holds 619,557 tokens, 37x the 16K budget of a single request, so nothing is preempted at concurrency 4.
llama.cpp server-slot sensitivity

llama.cpp defaults to four server slots, which is the configuration in the main figures. A 16-slot sensitivity run improves output throughput at concurrency 16 but regresses at concurrency 8:

Split Concurrency --parallel 4 --parallel 16 Change
Chat 1 22.4 tok/s 24.2 tok/s +8%
Chat 8 77.0 tok/s 42.7 tok/s −45%
Chat 16 81.5 tok/s 104.7 tok/s +28%
Agent 1 18.3 tok/s 19.0 tok/s +4%
Agent 8 44.2 tok/s 26.7 tok/s −40%
Agent 16 40.2 tok/s 49.7 tok/s +24%

Acknowledgments

vllm-metal builds on MLX and mlx_lm from Apple’s MLX team, on mlx-vlm for the vision-language paths, and on the vLLM engine and its hardware-plugin interface. Thanks to the upstream vLLM maintainers for review and support along the way, and to everyone who filed issues and shared benchmarks against the v0.2 and v0.3 releases.