I think people don't understand the structure of the gap. There are multiple components, and "benchmaxing" in the naive sense, direct training on test or its narrow paraphrases or reconstructions in case of private evals, is trivial. The interesting gap on most of these metrics now comes from the coverage of problems by RL, both in terms of effective compute (so, for example, DeepSeek with V4.1 can afford like 16-50x more trajectories per GPU-hour compared to Kimi with K3.1) and, even more importantly, in the sense of quality of environments (tasks, grading, guards against reward hacking). Data is upstream of the function gain here.
A lab with a high-quality internal research program and sufficient resources may still get set back by the eval refresh, because the base models are weak, because the tasks get intrinsically harder, because they haven't yet collected enough data for tasks in this regime, because they just haven't dedicated enough RL to this new distribution of tasks. But such a lab doesn't need a lot of effort to upgrade, they're already doing it, they will catch up quickly, and their current model will likely decline relatively less. A lab whose internal program is less advanced and more reactive will fall off harder, and will struggle more to catch up.
This allows us to make some interesting comparisons.
As we see here, V4.1F-0910 and 1017 score identically on FrontierCode 1.1 (≈44%), and diverge heavily on FrontierCode 2.0 (19.3% vs 28.5%).
This is a gain over 37 days.
K3 did 43.6% on 1.1 and 17.1% on 2.0.
Kimi K3.1 scores 47.1% on FrontierCode 1.1, and 25.5% on 2.0.
This is a gain over ≈97 days.
(though adjust for noise, Cognition != AA)
Nominally, 44 is not far from 47, and 25 is not far from 28. But I think the direction of delta, as well as the timeline, matter. We can assume both labs target this broad capability basin to a similar extent. FrontierCode 1.1 was released on July 7, ≈10 days before Kimi K3's debut and months before V4.1. By October, we can expect that both labs have hill-climbed it about as much as their resources and model allowed. But Kimi's climb has been narrower, so they got *higher* on 1.1 and fell off *harder* on 2.0. 1017 doesn't gain on 0910 whatsoever on 1.1 – which suggests that DeepSeek weren't even going after it with this release, they dedicated extra resources to new aspects of quality (here, product-grade mergeability), were expanding their RL pipeline, and perhaps have better understanding of the direction in which the frontier propagates.
The MiMo story and GLM story would benefit from more comparison points, but the logic is similar to Kimi's.
I recommend reading the post below to understand my perspective better.
x.com/teortaxesTex/status/20…