Introducing 1.58bit DeepSeek-R1 GGUFs! 🐋
DeepSeek-R1 can now run in 1.58-bit, while being fully functional. We shrank the 671B parameter model from 720GB to just 131GB - a 80% size reduction.
Naively quantizing all layers breaks the model entirely, causing endless loops & gibberish outputs. Our dynamic quants solve this.
The 1.58-bit quant fits in 160GB VRAM (2x H100 80GB) for fast inference at ~140 tokens/sec.
By studying DeepSeek-R1’s architecture, we selectively quantized certain layers to higher bits (like 4-bit), and leave most MoE layers to 1.5-bit.
Benchmarks + Blog:
unsloth.ai/blog/deepseekr1-d…
Dynamic GGUFs (131GB–212GB) on Hugging Face:
huggingface.co/unsloth/DeepS…