🔍 We release our work on decomposing and measuring evaluation awareness in frontier LLMs, together with EvalAwareBench, a factor-controlled benchmark for measuring which environmental cues trigger recognition and whether models behave consistently once they recognize evaluation. Evaluation awareness introduces directional bias into benchmark scores and undermines the deployment decisions and safety assessments that rely on them. The field has studied pieces of this phenomenon separately through work on sandbagging, alignment faking, and scheming, but without a shared foundation that unifies them. Critically, these lines of work conflate things that need to be kept separate: how recognizable a task is, whether a model notices, and whether it then changes its behavior. Without pulling these apart, it is impossible to tell whether a concerning observation is a model problem or a benchmark design problem. We decompose evaluation awareness into two components grounded in demand characteristics. The environment component captures what evaluative cues a task carries, operationalized through eight trigger factors. The model component separates recognition capability from behavioral propensity. We study both under a black-box constraint via chain-of-thought monitoring. Applying this framework to nine frontier models across four benchmarks, we find that there is no such thing as an inherently aware model or an inherently recognizable benchmark. Recognition depends on the specific pairing, accounting for 74.9% of variance. Recognition rarely leads to behavioral change and when it does, safety awareness drives caution while capability awareness drives compliance. Safety benchmarks are also far more recognizable than capability ones. A particularly important finding is that different models are sensitive to different trigger factors. For example, Claude Sonnet 4.5 attends almost exclusively to scenario coherence while Qwen3 models to explicit evaluation disclosure and stakes telegraphing. This interaction structure directly motivates our EvalAwareBench, which holds the underlying task fixed and independently toggles each of the eight factors, yielding 51,200 unique configurations across 100 paired safety-capability tasks. Stacking factors raises recognition progressively for nearly every model, including models that show near-zero awareness under any individual factor alone. Ultimately, evaluation awareness will intensify as models become more capable. The right goal is not to make evaluation cues harder to detect but to build models that behave consistently regardless of evaluation awareness. EvalAwareBench provides the controlled foundation for future work to drive models toward behavioral consistency regardless of whether they recognize evaluation. We further advocate that future benchmark reports should include an evaluation-awareness rate and an awareness tax measuring the performance gap between aware and unaware samples.

Reviewer: “I still have questions about who would use this.” Me: 👉🏻

The new #ICLR email is basically telling me that I am qualified but not invited LOL 😭 Need to get invited to another party to pick up my self esteem 💔

Agent Swarms: What do they know? Do they know things? Let's find out! I did a write up about the agent swarms and my discoveries. The post also includes a list of my findings in the appendix! jowimo.substack.com/p/tracki…

Wow!

Glad to see our work is useful for eval opus-5.5

There is nothing better than knowing your work is useful in production!

Meanwhile @Grimezsz just popped up in the most surprising but expected places LOL. Where is the new album btw?

When people ask what the point of learning complicated math is:

Truly grateful to see so much enthusiasm in our workshop and we look forward to seeing your work!

Privacy policies on almost all frontier model providers are insane and assuming you have ZDR or any meaningful protection, even on enterprise setting is completely false these days! Here I'm going to name a few very arbitrary and misleading sentences in ToS of frontier labs that most ppl don't know about. This is all assuming you have already *opted-out*, explicitly of data collection. Default for all frontier models is opt in btw!! 1) the 'explicit feedback': your chats are claimed to be 'protected' unless you provide feedback! what is feedback, you ask? it's thumbs up or down to any message in the chat, or even if you pick 'which is better' response on OpenAI GUI. In that case, you have provided explicit feedback and your chat is collected and retained for *over 7 years*!

Such a shame. Why does AI development have to be binary? Why can it only be either OpenAI or Anthropic? Why can it only be US or China? The world is big enough for more than one winner.