Glad I can finally share another project we have been cooking! We built Weaver to make speculative decoding practical( and SOTA😉) for Qwen3.5-27B… we’re ready to share the paper, draft model, and SGLang reference implementation. It will be in our own inference engine soon but in the meantime test and play with it on your own! Paper: arxiv.org/abs/2607.06763 Model: huggingface.co/trymirai/weav… Code: github.com/trymirai/sglang

Qwen3.8-27B running at just 8.45 GB and hitting 60 tok/s on a single RTX 3090 is the kind of local AI result that makes you rethink what a 27B model actually needs. I tested the new 2.4-bit Qwen3.8-27B quant from trymirai on a real coding task. The prompt was basically: Build a voxel pagoda garden. No pile of separate files. No huge framework. Just a single HTML file using Three.js. The agent setup was: • Pi Agent • Medium reasoning • 10 turns • 32.5K output tokens • About 12 minutes • One prompt to start the project And it produced the pagoda. That matters more to me than a model fitting into a certain file size. A heavily compressed model is only useful if it can still maintain context, reason through changes and actually finish a task. The performance on my setup was also surprisingly strong. 1× RTX 3090 60 tok/s with MTP 44 tok/s without MTP ~1,190 tok/s prefill on a 7K-token prompt 159K context The GPU was running around 290W. Then I moved to two 3090s with pipeline parallelism: 2× RTX 3090 66 tok/s decode ~1,890 tok/s prefill Full 262K context So the trade-off here is pretty interesting. The model is slightly larger than some of the more aggressively compressed Qwen3.8-27B releases, but the goal isn’t simply to win a file-size contest. You want the compressed model to remain useful. And in this test, the 2.4-bit model was able to handle a multi-turn Three.js build, generate 32.5K tokens and maintain the project through ten turns. That’s the real promise of aggressive quantization. Take a model that normally requires much more memory, compress it heavily, put it on hardware that people already own, then see whether it can still perform actual work. 27B parameters. 8.45 GB. 60 tok/s on one 3090. 262K context with two. That’s a pretty serious local setup for a model this size.

qwen on 16gb mac mini is crazy

Mirai Labs (@trymirai) compressed Qwen3.8-27B to 2.4 bits. I ported it to llama.cpp and now it runs on a 12 GB card with 128K context. With q4 KV cache the 128K window takes 11.3 GB of VRAM, with q8 KV you get 74K in 11.1 GB. Decode is 40 tok/s, still 34-35 tok/s at 62K deep, and prefill around 1000 tok/s. I measured all of it on 1 RTX 3090 at 300 W, capped to what fits in 12 GB. No requant anywhere. The GGUF carries Mirai's compressed codes bit for bit, and greedy output matches their own vLLM plugin. Everything in the video was built by this same model as an agent in pi, 11 builds from 1 prompt each, a few needed a follow up and 1 is a blooper. That run was vLLM on my 3090s. Today the same weights run in llama.cpp. Big thread, all links and recipes 👇

27B model in 8.45 GB. 60 tok/s on a single 3090. And it built this voxel pagoda from a single prompt. Qwen3.8-27B in 2.4 bits from @trymirai is on a completely different level compared to the mess from @PrismML. Yeah, it's slightly larger in size, but these guys don't claim 98% accuracy either, unlike some others. Pagoda: 1 prompt, medium reasoning, pi agent, 10 turns, 32.5K output tokens, 12 minutes. A single HTML file on three.js. Speed on my rig, GPU at 290 W, vLLM: 1x 3090: 60 tok/s with MTP, 44 without, prefill ~1190 tok/s on a 7K prompt, 159K context 2x 3090 (pipeline parallel): 66 tok/s, prefill ~1890 tok/s, full 262K context Huge thanks to @norpadon and @gapDEEPry for the opportunity to test this model before release and help the team with testing.

We made the preview publicly available: huggingface.co/trymirai/Qwen… Qwen3.8 27B that fits in 8.5Gb Works with an experimental build of uzu (our inference engine) on Apple Silicon, and in vllm on Nvidia See the model card for instructions

We may have cooked a 2-bit version of Qwen3.8 27B which *actually works*, is not deep fried to benchmaxx, and fits in 8.5GB We have not yet finished evaluating it, and it'll take some time to properly ship it, but if someone wants to try and share feedback please dm me

The latest version got significantly improved performance, especially on prefill

Today we are releasing our speculative decoding implementation in our inference engine uzu. Initially for Qwen3.6 27B, with support for Qwen3.8 27B and Muse Glimmer coming soon. On Apple M5-series chips, we outperform MTPLX (MLX + speculative decoding) by almost 2x, and llama.cpp by over 3x at comparable quantization levels, with the strongest gains achieved on mathematical reasoning and coding tasks. Run the model: Mirai-M: trymirai.com/local-models/al… Mirai-L: trymirai.com/local-models/al… Explore the benchmarks: trymirai.com/metrics Learn more about our speculative decoding implementation: trymirai.com/blog/speculativ… Our draft model, quantized checkpoint format, verification algorithm, and GPU kernels are co-designed from the ground up around the latest Apple M5 chips to take maximum advantage of GPU Neural Accelerators. Unlike popular speculative decoding architectures such as model-native MTP, which produce small draft chains of 3-4 tokens at a time, we use extremely aggressive speculative budgets of 16-32 tokens. This enables us to use Neural Accelerator-backed GEMM kernels, achieving maximum utilization of hardware arithmetic throughput.

Today we are releasing our speculative decoding implementation in Uzu. Biggest release since inception of our lab. Initially, for Qwen3.6 27B, with support for Qwen3.8 27B and Muse Glimmer coming soon.

what if they used this style for their iPod launch? with contentyard.io, you don't have to imagine.

wow looks cool

It's cool!

We are releasing our speculative decoding approach, together with the draft model for Qwen3.5-27B and the reference implementation in SGLang. Paper: arxiv.org/abs/2607.06763 Model: huggingface.co/trymirai/weav… SGLang implementation: github.com/trymirai/sglang

When we invested in Mirai Labs (@trymirai) earlier this year, they had a long-ish term plan to offer full-stack on-device inference infrastructure and the underlying models. I thought it would take them a year... In less than four months, their co-designed quant+inference stack for Apple Silicon shows 40–60% more tokens/sec vs llama.cpp and MLX at the same quality level. This week they published technical posts on quantization (trymirai.com/blog/quantizati…) and sparse buffers (trymirai.com/blog/sparse-buf…) which I encourage app builders and inference consumers to read. Awesome stuff from @Darkolorin, @dimashvets, and the Mirai team.

Really proud of this release from the team :) We just shipped our first quantized Qwen3.5 checkpoints, the quantization format was co-designed with our inference engine from scratch instead of trying to make an off the shelf format fast. Our 8-bit models decode faster than llama.cpp 4-bit on Apple silicon, and they're virtually identical to the original!