MLX-Serve 26.9.5 is out, and this one is a pretty serious update for anyone running local AI on Apple Silicon.
The Desktop edition now adds support for Qwen-Image-2.1, so image generation is becoming part of the same local serving stack instead of needing a completely separate setup.
It also ships with Prism Bonsai 2 support.
That is the 27B Qwen3.8-based model compressed down to roughly 6GB, which is already interesting for local inference. The MLX implementation is being positioned as one of the fastest ways to run it, although the actual speed will depend on the Mac, context size and workload.
The other change I’m watching more closely is concurrent inference.
MLX-Serve has been pushing beyond the “one request at a time” local-server experience, and this release improves how multiple streams are handled. That matters a lot if you’re using a Mac as an actual AI server rather than just chatting with a model.
And finally, there are now charts and evaluation results.
That’s a welcome addition.
Raw tok/s numbers are useful, but seeing the benchmark methodology, scaling behaviour and model comparisons makes it much easier to understand what the runtime is actually doing.
So this release is less about adding one flashy model and more about turning MLX-Serve into a broader local AI stack:
Qwen-Image-2.1 for vision generation.
Bonsai 2 for lightweight 27B inference.
Better concurrent streams for serving multiple requests.
Charts and evals to show the performance instead of just claiming it.
If you’re running Apple Silicon and experimenting with local models, MLX-Serve is definitely worth watching.
And yes, apparently the GitHub link is the kind you should click before it mysteriously disappears.
A 12GB RTX 3060 is now running a 9B reasoning model with the full 262K context window.
The model is Xiaomi’s MiMo-V2.6-Distill-Qwen-9B, running at Q5_K_M on a single RTX 3060.
The numbers from this setup:
12GB VRAM
262K context
~47 tok/s decode
~1,600 tok/s prefill
Q5_K_M quantization
Flash Attention enabled
Q8 KV cache
Reasoning enabled
And the setup is surprisingly straightforward with llama.cpp.
llama-server \
-m MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf \
-ngl 99 -c 262144 -fa on --jinja \
-np 1 \
--cache-type-k q8_0 --cache-type-v q8_0 \
--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 \
--reasoning on
The model itself is interesting too.
MiMo-V2.6-Distill-Qwen-9B is based on Qwen3.5-9B and fine-tuned using outputs from MiMo-V2.6.
So you’re essentially taking a relatively small Qwen model and distilling reasoning behavior from a much larger teacher into it.
Xiaomi reports SWE-Pro improving from 32.0 to 44.6 compared with the base model.
That’s a sizable jump on their reported benchmark, although the more interesting test for me is what happens when you put it inside an actual coding agent.
Because static benchmarks only tell you part of the story.
An agent has to read files, reason about the codebase, make edits, run commands, inspect failures, recover from mistakes and eventually finish the task.
That’s where small local reasoning models can get interesting.
If this 9B model can maintain useful coding performance while running at roughly 47 tokens/sec on a $300-ish class GPU and holding 262K context, that’s a very practical local setup.
And 262K context on a 12GB card is probably the headline I wouldn’t overlook.
You don’t need a 70B model or a pile of GPUs to experiment with serious long-context agentic workflows anymore.
I’m running my own agentic coding benchmark on it next.
That’s the test I’m actually curious about.