Additional inference-time compute doesn’t seem to help frontier models much with our conceptual reasoning benchmark, LMCA. This is despite larger models continuing to perform better on LMCA. It’s also in contrast to what we see with LLMs solving coding and math problems where inference-time compute seems very helpful. Some speculation for what might be going on: Use of inference-time compute is trained during post-training, and post training is largely focused on coding, math, and other domains with verifiable rewards. So plausibly models aren’t very effectively trained to use inference time on “fuzzy” tasks like conceptual reasoning. (By contrast, more effective use of pretraining data presumably does improve conceptual reasoning, because some of the pretraining data is conceptual.) See the below inference-time scaling curves from us comparing the performance of models using different effort levels. You can tell that the models in fact think for longer with higher effort (cost goes up), it just doesn’t result in better performance. (For better readability, see separate plots for the different model families in thread, also includes a plot for Gemini models.) (The black line shows the cost-performance pareto frontier: the best performance achievable at a given average cost, taking into account the possibility of randomising between different models.)

Separate plots for the different model families. The lines for Claude and Gemini models are strikingly flat. Some weaker GPT models might benefit from some increases in effort level.

New CRI results are in! Highlights: - Fable 5.1 leads but Astra 6 is really close. - The gap between Anthropic and OpenAI models and the rest of the field is huge. The best non-Anthropic, non-OpenAI model ranks 11th overall. - Meta is now the third-best lab on our metrics, pulling ahead of Google DeepMind. - Gemini 3.8 flash still looks better on the CRI compared to Artificial Analysis. (Full results on conceptualreasoning.ai)