This really matters for Local AI! We keep shrinking the model weights. Now researchers are going after the model's working memory. A new technique called MILO cuts KV-cache memory by up to 👉 50% 👈 and reports up to 1.8× higher throughput on Qwen2.5 experiments. 🎯 This is really signficant because KV cache grows with context. So once the quantized model fits into VRAM/RAM, context is the next memory hog. As an example ... 🧠 8GB KV cache @ 128K context ↓ 50% compression 🧠 ~4GB for the same cached context So that saved memory could mean ... 📚 ~2× the context for the same KV-memory budget 🤖 more simultaneous agents/sessions 💾 less KV spilling into slower RAM 🧠 more room for model weights MILO breaks the KV cache into blocks, finds redundancy using low-rank compression, and gives information-dense blocks more precision while compressing redundant blocks harder. ⚠️ MILO currently targets many-shot long-context inference and was tested on Qwen2.5 3B/7B with an Nvidia L4. Obviously this is research and not a llama.cpp feature you can turn on tonight. 🔗 Paper in ALT