We spent 9 weeks testing agent reliability during software tool use. Nearly 60% of claims accepted as correct could not be backed by evidence from recorded tool calls. The answer looked right. The evidence wasn't. 🧵

We tested 15 models across 180 runs and 570 claims whilst using @scale_ai s MCP Atlas. In 161 claims, the final answer and the agent's recorded tool calls told different stories. 109 claims appeared in the answer without evidence. 52 had evidence that the agent failed to use.

That is 109 out of 183 claims accepted as correct. Nearly 60%. We cannot say whether the model guessed, remembered the answer or found it elsewhere. We can only say its own recorded work did not support the answer.

The reverse matters just as much. For 52 claims, the agent had already retrieved the supporting evidence but failed to use or present it correctly. The retrieval work was done. A potentially correct answer was left on the table.