Your cache is everything when it comes to inference, but how do you make sure you're keeping it around with local models? Typically when you start a server, it's a fresh slate. Then the KV cache grows, evolves, and your cache hits keep growing. But then you shut down the server, swap to a new fancy toy (model), and your cache is destroyed. Nothing we can do there, since caches are model-specific. But let's say then you want to go *back* to deploying the original model. Now *it's* KV cache was also destroyed! @sgl_project supports directly setting up hicache through some configuration arguments. The important part: * L1: GPU KV cache * L2: RAM KV Cache (usually larger, but in the world of local LLM's sometimes not!) * L3: Disk/storage-based KV Cache If you have the storage, keeping/enabling L3 allows for you to plug and play as many models as you'd like, and it'll still keep the KV Cache of your old deployment! Now, on a reboot say you have a hit on L3. Because you got a hit, it'll then keep that "warm" in L2 and L1-based caching so you don't face that terrible TTFT again. In my own learnings, you have --hicache-storage-prefetch-policy. This tells SGLang how to check and when to check if something exists in hicache in say L3. The two options I looked at were: * wait_complete * best_effort `best_effort` will check if L3 has that cached prefix, start retrieving it, but don’t block inference waiting for the full cache to arrive, reusing what’s ready and recompute anything that isn’t. Or: it'll promote whatever it can retrieve from L3 to L2/1 without blocking the request. `wait_complete` means that if L3 has that cached prefix, finish retrieving it rather than racing storage retrieval against recomputation. (These are Codex definitions) Translation: `best_effort` could have a better TTFT (probably) since it checks if we have it and then recomputes, while `wait_complete` will just fully reuse the cache and you'll wait a bit longer since it's going from cold -> hot. Don't quote me here, just where my intuition. There's a third one called `timeout`, which acts as a middle ground, still need to try it out. Last few little important parts: * SGLang will store the L3 wherever SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR is set as the env variable. * When using L3, it's based on the exact configuration and script you're currently serving with (usually with model/config information & topology, as well as tokenizer and such). I haven't tested if changing a deployment let's you reuse the L3, but my hunch is not without being careful * (Just a hunch, again still needs testing) You should probably use m.2 drives here and not HDD's if doing `wait_complete`, and possibly `best_effort` for HDDs. Need to toy with this more. But since `best_effort` recomputes, we likely can just grab it off the HDD and not worry about the transient speed.

🚀 More reliable agents with DeepSeek V4.1! XGrammar brings strict tool calling to SGLang & vLLM through Structural Tags, enforcing tool argument schemas in DeepSeek’s native format. See how the Structural Tag works 👇 blog.mlc.ai/2026/05/04/xgram… Check out XGrammar 👇 github.com/mlc-ai/xgrammar

Halo is not limited to supervised fine-tuning. It supports async RL and training with external environments. We are excited to partner with the @sgl_project team to make it the primary engine for rollouts.