Fully agree with Zhihao. As the cost of implementation and verification asymptotically approaches 0, there is little argument for broad generalization.
This is very much in the spirit of Unikernels (one of my all-time favorite projects, shoutout to
@avsm). The hope here is that one doesn't need to write a declarative spec themselves as agents become more capable.
I think this comes down to a classic systems tradeoff: generality vs. specialization. vLLM/SGLang cover a huge space of models × hardware × workloads. It’s hard for a single system to be best at every combination, and focusing on a narrower set gives you more room to optimize.
Historically, the challenge was engineering cost: optimizations for one configuration often need substantial rework for another. Coding agents are lowering that cost, which I suspect is one reason we’re seeing more specialized inference engines.
Fragmentation is worth discussing, but so is the value of specialization. Many of these engines can complement vLLM/SGLang, especially when users need more performance from a specific setup. My 2 cents.