Frontier on-device AI lab. Models, runtime & infrastructure to make on-device AI interactive, ambient & continuous.
San Francisco
- Tweets 100
- Following 33
- Followers 1,280
- Likes 343
Pinned Tweet
Today we are releasing our speculative decoding implementation in our inference engine uzu.
Initially for Qwen3.6 27B, with support for Qwen3.8 27B and Muse Glimmer coming soon.
On Apple M5-series chips, we outperform MTPLX (MLX + speculative decoding) by almost 2x, and llama.cpp by over 3x at comparable quantization levels, with the strongest gains achieved on mathematical reasoning and coding tasks.
Run the model:
Mirai-M: trymirai.com/local-models/al…
Mirai-L: trymirai.com/local-models/al…
Explore the benchmarks:
trymirai.com/metrics
Learn more about our speculative decoding implementation:
trymirai.com/blog/speculativ…
Our draft model, quantized checkpoint format, verification algorithm, and GPU kernels are co-designed from the ground up around the latest Apple M5 chips to take maximum advantage of GPU Neural Accelerators.
Unlike popular speculative decoding architectures such as model-native MTP, which produce small draft chains of 3-4 tokens at a time, we use extremely aggressive speculative budgets of 16-32 tokens. This enables us to use Neural Accelerator-backed GEMM kernels, achieving maximum utilization of hardware arithmetic throughput.
29
61
458
219,582
Mirai Labs retweeted
We may have cooked a 2-bit version of Qwen3.8 27B which *actually works*, is not deep fried to benchmaxx, and fits in 8.5GB
We have not yet finished evaluating it, and it'll take some time to properly ship it, but if someone wants to try and share feedback please dm me
13
12
159
18,361
Mirai Labs retweeted
Mirai Labs (@trymirai) compressed Qwen3.8-27B to 2.4 bits. I ported it to llama.cpp and now it runs on a 12 GB card with 128K context.
With q4 KV cache the 128K window takes 11.3 GB of VRAM, with q8 KV you get 74K in 11.1 GB. Decode is 40 tok/s, still 34-35 tok/s at 62K deep, and prefill around 1000 tok/s. I measured all of it on 1 RTX 3090 at 300 W, capped to what fits in 12 GB.
No requant anywhere. The GGUF carries Mirai's compressed codes bit for bit, and greedy output matches their own vLLM plugin.
Everything in the video was built by this same model as an agent in pi, 11 builds from 1 prompt each, a few needed a follow up and 1 is a blooper. That run was vLLM on my 3090s. Today the same weights run in llama.cpp.
Big thread, all links and recipes 👇
13
13
143
9,054
Our team loves to catch unique bugs
Our GPU whisperer @ychnzhn4 wrote a small blogpost about the most annoying bug I have ever seen, which was an absolute nightmare to catch
trymirai.com/blog/tackling-r…
4
749
📣 We've just published our latest research on state-of-the-art speculative decoding in interactivity.
Introducing a new approach that achieves a 4.37× speedup over autoregressive decoding in small batch setting and outperforms the strongest tuned DFlash baseline by 24.7%.
arXiv: arxiv.org/abs/2607.06763
github: github.com/trymirai/sglang
huggingface: huggingface.co/trymirai/weav…
4
21
4,409
Mirai Labs retweeted
We are releasing our first quantized checkpoints for the Qwen3.5 series of models, co-designed jointly with our inference engine to achieve maximum possible performance on Apple hardware
Starting from 0.8B, 2B and 4B models
trymirai.com/blog/quantizati…
15
54
438
69,331
Google DeepMind released Gemma 4.
Our engineer @norpadon analyzed the architecture
Gemma 4 architecture analysis thread
Just as Gemma3n, this thing has a galaxybrained architecture, very much not a standard transformer
1
15
5,111
If you've implemented speculative decoding, you've run into this: your speculator predicts the distribution correctly and still gets rejected.
Let’s take an example:
"The random number from 1 to 10 inclusive is"
Ideal LLM: uniform 1/10 across all numbers.
Good speculator: same distribution.
In that situation, the naive scheme takes a random number from the speculator, then takes another random number from an LLM and can only accept the bonus token in 1/10 cases when they happen to match.
We should be able to always take its guess without the loss of generation quality.
The fix is straightforward.
Sample from both the speculator and the LLM using the same seed via the Gumbel-max trick.
Shared randomness means shared outcome when distributions match.
100% acceptance rate in this case. Near-optimal in practice.
We shipped this in github.com/trymirai/uzu
Full kernel implementation: trymirai.com/blog/how-to-imp…
1
21
2,337
Day 0 on-device support of the latest and the smallest @liquidai model
LFM 2.5 350M is now available on @trymirai
Day 0 support across the stack:
> Hardware: @AMD, @Intel, @Qualcomm
> On-device: @lmstudio , @Cactuscompute, @RunAnywhereAI , @zeticai_ , @trymirai
> Customization: @distil_labs
2
18
2,034
LFM2.5-350M is now available on Mirai.
@liquidai smallest model outperforms Qwen3.5-0.8B on reasoning and agentic tool use.
Running on Mirai in full precision, it exceeds 70 tokens/second on iPhone.
1
5
23
2,297
LLM activations have outliers. A few channels spike 100x past the rest, every token. Standard INT8 wastes almost all its precision covering them.
The fix: rotate the weight space so outliers disappear before quantization. It's why QuaRot and TurboQuant work.
Here's how we implemented it:
trymirai.com/blog/why-activa…
2
16
1,304
Morton codes for GEMM
Apple just released its programming guide for Metal Performance Primitives, and they suggest using Morton codes for tiled GEMM, but why?
In computer graphics, you use such space-filling curves all of the time
It makes objects that are close in space to be close in memory
There are several reasons, but one of them is that you get better cache locality, meaning less expensive reads from the device memory
This is exactly why it’s appealing for GEMM too - you have a lot of overlapping memory reads between the tiles
Morton schedules tiles in compact square patches, minimizing the working set that fits in last-level cache simultaneously, so nearby threadgroups are more likely to reuse the data they share
8
942
Considering quantized activations
(1/n) I recently joined @trymirai, where we are working on LLM inference targeting Apple Silicon. Lately I've been digging into quantization.
LLM inference is mostly memory-bound. The byte/FLOP ratio is high enough that a lot of the machine's time goes to moving data around instead of doing compute. Quantization helps with that in general, but on Apple Silicon there's an extra payoff: the GPU has a fast W8A8 path. If both weights and activations are INT8, you can use that path for prefill and speculative-decoding verification.
Weights are easy since they're static and can be quantized offline. Activations are where the real pain starts.
7
910
Efficient quantization is coming soon for on-device inference
We are doing really cool hard tech at @trymirai, but until recently our social media feeds were full of linkedinish cringe. We decided to fix it and share more technial content
I am currently working on our quantization pipeline, so here is a thread about LLM quantization
1
17
1,591
How to improve acceptance rate in speculative decoding?
Inference of LLMs requires reading large amounts of data from memory while doing relatively little compute with that data, which means that compute is significantly underutilized.
github.com/trymirai/uzu can turn that underutilized compute into higher decoding throughput by trying to cheaply guess the next token, computing the next token for both the current sequence and current sequence + guessed next token, and then, if the next token matches our guess, we can take both at once, doubling the throughput. This is the simplest form of speculative decoding.
1
5
19
1,941
The simplest way to improve upon this is by realizing that the LLM is often unsure about what the next token should be. Let's take an example: "The random number from 1 to 10 inclusive is", an idealized LLM outputs uniform 1/10 probability for every number 1–10. A good speculator would output the same. In that situation, the naive scheme takes a random number from the speculator, then takes another random number from an LLM and can only accept the bonus token in 1/10 cases when they happen to match. But here the speculator exactly predicted the LLM's distribution. We should be able to always take its guess without the loss of generation quality.
1
2
461
We achieve this by sampling from both the speculator and the LLM via the Gumbel-max trick, sharing the same seed. This will achieve a 100% acceptance rate in the toy example above, and a near-optimal acceptance rate in more complicated real-world cases, while being much simpler than the mainstream rejection-sampling-like algorithm that most other inference engines use.
github.com/trymirai/uzu/blob…
1
329
Why Muon performs exceptionally well on quantized models
Why does Muon beat Adam for training quantized networks?
It comes down to what each optimizer treats as "distance" in weight space.
Adam treats a weight matrix as a flat vector of numbers. Muon treats it as a linear map — and measures change by how much the input-output mapping moved.
gradient G has SVD G = U Sigma V^T. Muon's update is just U V^T. keep the directions, throw away the magnitudes
1
10
167
21,270