KTransformers v0.7.1 is here 🚀 Now supporting: • Qwen VLM multimodal LoRA fine-tuning in BF16 • Native RAWINT4 LoRA for Kimi K2.5 / K2.6 Fine-tune massive VLM and MoE models with a more memory-efficient training path.

🚀 Over one trillion tokens per day, powered by Mooncake and SGLang. @ApproachingAI shares the system architecture and engineering lessons behind its token factory, running leading trillion-parameter models with SGLang HiCache + Mooncake Store. A deep dive into: 🏗️ Production architecture and system design trade-offs ⚡ Performance optimizations for trillion-token-scale workloads 🛡️ Failure isolation, fault tolerance, and recovery Real production lessons on scaling KV cache reuse while meeting strict inference SLOs. Read more: kvcache.ai/blog/mooncake-in-…

🚀 KTransformers v0.7.0 is here! This release brings major upgrades for ultra-large MoE fine-tuning: • Native FP8 Expert weights for LoRA, reducing host memory usage for DeepSeek-V3.1 from ~1.4 TB to ~800 GB • Support AVX-512 backend for compatible AMD and x86 CPUs • Support full fine-tuning • Checkpoint saving and resuming • A complete KTransformers × LLaMA-Factory fine-tuning Cookbook 📷 Blog: kvcache.ai/blog/ktransformer… #KTransformers #LLM #FineTuning #DeepSeek

📍 vLLM Community Meetup — May 23 Approaching.AI joined the vLLM @vllm_project community meetup themed "How will inference acceleration frameworks reshape LLM deployment in the Agentic AI era?" — alongside Red Hat, Inferact, Alibaba, NVIDIA, Moonshot AI, and the open-source community. Lu Jiahao (Approaching.AI tech expert, Mooncake core contributor) co-presented "llm-d + Mooncake: Large-Scale Agentic AI Inference in Practice" with Greg Pereira, core maintainer of llm-d. ━━━━━━━━━━ 🔹 The Agentic AI bottleneck As context lengths grow, traditional compute paradigms face new cost and performance pressure. But multi-turn interactions, tool calls and long-context reuse give Agentic workloads very high KV cache hit rates — a window for "storage-for-compute" trade-offs. 🔹 Mooncake: store more, transfer fast Transfer Engine (TENT) — evolves from a passive communication library into an active orchestrator. Declarative orchestrationreplaces imperative APIs, delivering low-latency, high-throughput transfer across heterogeneous interconnects. Mooncake Store — a high-capacity, low-cost SSD tier joins the cache hierarchy. TCO drops, performance holds, and the KV cache poolscales beyond single-machine limits for cross-instance, cross-node context reuse. 🔹 llm-d × Mooncake roadmap Phase 1 — MooncakeConnector: PD disaggregation, P2P transfer, Messages API Phase 2 — MooncakeStoreConnector: tiered cache offload, Responses API ━━━━━━━━━━ Approaching.AI will keep contributing to Mooncake and partner with the community to push inference systems toward a new balance of scale, performance and cost for the Agentic AI era.

Handy for capacity planning and config sizing — give it a try 👍

At AMD Developer Day @AIatAMD 🚀 Tsinghua University Assoc. Prof. and Approaching.AI co-initiator Mingxing Zhang @james0zan gave a talk: “How Do We Reduce Token Costs in the Agent Era?” --- The core shift in AI workloads: We're moving from simple chatbots to complex agent applications. Each agent task requires dozens of model calls and heavy context I/O, pushing token consumption from 1x to 1000x. Not all tokens share the same SLO, and compute infrastructure does not equal efficient token production. Cache, transfer, and token reuse matter as much as the silicon underneath. --- This thinking shapes Mooncake github.com/kvcache-ai The open-source KV-cache-centric serving substrate for modern LLMs. Originally focused on KV cache — store more, transfer fast, easy to use — Mooncake has grown into a common substrate for disaggregated LLM serving: ✅ PD / EPD / RL disaggregation ✅ Large-scale elastic EP ✅ Efficient model weight loading --- Latest Mooncake highlights from the talk: ⚡ Checkpoint Engine — 1T parameters synced in seconds for distributed RL ⚡ Elastic EP in SGLang — partial failure tolerance for DeepSeek MoE deployments ⚡ vLLM native integration — scaling agentic workloads in production ⚡ Omni Models ⚡ TorchSpec — decoupled speculative decoding framework, streaming hidden states from inference to training ⚡ RL toolchains — TransferQueue / Slime (WIP) / Roll (WIP) for cross-device async non-blocking scheduling ⚡ Production-proven — large-scale deployment validated by RBG + SGLang + Mooncake --- Thanks @AIatAMD for hosting. We look forward to deepening collaboration with the AMD developer community on the infrastructure layer of the agent economy.

Proud to be a launch partner for Kimi K2.6! 🎉 we shipped full support on Day 0： ✅ Token service live； ✅ KTransformers adapted — full-precision inference & fine-tuning with just 48GB VRAM + sufficient RAM Try it now 👉 github.com/kvcache-ai/ktrans…

Mooncake × SGLang introduce Elastic Expert Parallelism (Elastic EP) for large-scale MoE inference. With partial failure tolerance, Elastic EP enables recovery from GPU/node failures in seconds — without interrupting serving. lmsys.org/blog/2026-03-25-ee…

Big update from Approaching.ai: Dr. Jenny Wu joins as President, bringing experience from Baidu Capital, Trip.com, Morgan Stanley, and New Hope. She’ll lead strategy, finance, and global ops as we scale efficient AI inference. More to come. #Inference