Let's talk about evals... AI without evals is like having junior colleagues and never checking their work. Ever. Just hoping they get it right every time. Obviously not a great strategy!
Running evals occasionally isn't much better. That's the equivalent of a 1:1 every few months vs actually looking at what someone produces day to day.
Humans make mistakes. Without a framework to catch them, the business pays for it eventually. Agents are no different... except you can test agents constantly, 24/7, and they don't even mind, even when you give them 'feedback' π
@HeyHelixAI runs over 40 agents. Many of them have a narrow remit with one job. Narrow jobs are testable jobs. Every agent ships with its own golden dataset before it ships; we run those with the help of
@pydantic 's Logfire (which we love for a bunch of reasons, evals being one of them, go check it out!).
When do we run evals?
1. When a system prompt changes. Ours are assembled dynamically, so one edit can mean dozens of permutations to check.
2. When we're evaluating a model. New ones land every few weeks, and the cost/quality trade-off shifts every time (sometimes less thinking / reasoning can increase accuracy AND reduce cost!)
3. On every conversation. Not "did the LLM think it went well" but did the booking actually happen, did the search actually return the specific results, did we tell the user when it didn't.
4. In Continuous Integration (CI). GitHub Actions, run conditionally depending on what actually changed.
5. When production catches a failure or edge case from real users. That conversation becomes a new eval case so the same mistake can't creep back in.
6. When we add a new agent. Obviously! Agentic engineering works best with test-driven development.
The holy grail is closing the loop properly: a failing eval raises a GitHub issue, an agent picks up the issue and proposes a fix, another agent makes the necessary change(s), the evals run again, and a human reviews the diff before anything ships. Same process we'd use for a colleague's PR. Which is sort of the whole point. I'm always happy to wake up in the morning & see that the robots have been self-improving! π€
What triggers evals for you? Do you have a continuous loop in place?