1/ Most LLM teams have evals. Almost none have a gate. The difference: evals tell you something got worse. A gate stops you from shipping it. Here's how to build one in GitHub Actions. 🧵

3/ Wire it into CI like this: - eval script runs as a CLI step in your PR workflow - it exits with code 1 if any metric falls below threshold - GitHub Actions fails the check - branch protection blocks the merge The whole thing fits in one YAML file. No new tooling required.

I modeled our dependence on LLMs as an epidemic — the same SIR machinery used for real infections. Compartments: unexposed -> casual -> dependent. The equations returned something uncomfortable. Thread 🧵

And the fix is asymmetric. Fighting the spread costs ~a 50% cut in transmission (expensive). Making dependence easy to walk back — exportable workflows, retained skills — moves the escape boundary almost linearly (cheap).

Hot take: most LLM benchmark results are meaningless. Not because the models are bad. Because nobody puts error bars on the scores. You're comparing 84.2 vs 81.9 without knowing if either number is real. Is this a problem on your team? 👇 (Replying with a link to a free fix)

1/ You ran the same LLM eval twice and got different scores. Is that a regression — or noise? Most teams can't tell. Here's the method that answers it every time. 🧵

3/ The fix: bootstrapping. Here's the idea in plain English: • Take your N eval results • Resample with replacement 1,000 times • Compute your metric on each resample • That spread IS your error bar No distribution assumptions. Works on accuracy, cost, latency — anything.