CS student in the US, tracking AI daily 🐧 What actually shipped: models, tools, industry. Sources cited, hands-on tests, no hype.
Michigan, USA
- Tweets 111
- Following 16
- Followers 4
- Likes 1
Pinned Tweet
Not one model. Not luck. Not scale. My test.
26 attempts passed it. All turn "a < b and c > d" into "a d".
5 agents, 2 labs: all broken.
5 reruns: all broken.
Haiku to Opus: same score, 5.4x cost. 4-bit 7B: same failure.
Hidden test as the spec: fixed first try.
1
12
Not one model. Not luck. Not scale. My test.
26 attempts passed it. All turn "a < b and c > d" into "a d".
5 agents, 2 labs: all broken.
5 reruns: all broken.
Haiku to Opus: same score, 5.4x cost. 4-bit 7B: same failure.
Hidden test as the spec: fixed first try.
1
12
The write-up: how each one was ruled out, and what it does not show.
github.com/vyang472/five-bug…
Check it without trusting me: python3 scripts/verify.py replays 13 archived patches against both test sets in under 2s. No agents run, no API calls.
5
A 7B model quantized to 4 bits writes the same broken regex as Opus 5.
I added 16 local runs to last night's 10.
26 attempts. Two labs, four model sizes, two quantizations.
Three distinct patterns. Zero passed the hidden test.
It was never a model problem. 👇
1
12
Quantization changed the answers without changing the score: Q4 and Q8 both went 24/40 visible and 16/40 hidden, but produced 16 and 17 distinct answers, disagreeing completely on 2 of 5 tasks.
Q8 took 2.3x longer to tie.
github.com/vyang472/five-bug…
4
Two nights, 55 agent runs, under $9.
One thing I'd stand behind: your test is your spec, and five agents will prove it.
One I'd redo bigger: everything with n=5 in front of it.
Repo's public. 👇
1
5
github.com/vyang472/five-bug…
Stand behind: 10 runs failed the same hidden case; 1 spec change fixed it.
Redo bigger: the model-tier comparison, the timing spread, all of it.
Got wrong and corrected: I called a 2.5x task spread "variance" when it wasn't.
6
A skill with 105K stars promises to cut your agent's tokens 65%.
On agentic coding I measured 21.7%.
Quality held: 5/5 either way, same 4/5 on the hidden tests, failing the same case.
1
5
Three experiments tonight. 25 agent sessions.
$4.98.
One reproducible finding, one corrected claim, one public repo.
Running the experiment is cheaper than arguing about what the answer would have been.
1
8
You don't need my five bugs. You need five of yours.
Pick bugs you already understand
One file each
One checker that prints PASS
One clean copy per tool
Then the mean checker, afterwards
1
5
No pytest, no dependencies, no harness. My whole task suite is 10 files of plain asserts.
Setup time was under 10 minutes. The part that taught me something was step 5, and it took another 10.
Template: github.com/vyang472/five-bug…
5
Limits of everything I posted tonight, stated plainly:
n = 5
One laptop
One attempt per cell
Textbook bugs
Already localised to one function
Never tests finding the bug
1
6
Someone asked how you tell whether an agent fixed the root cause or just memorised your test's shape.
I think I have an answer now.
Weak test: 0 of 10 found the root cause.
Honest test: 1 of 1 did.
1
5
Answering @MBashirzadeh, who asked exactly the right question.
The agent isn't choosing between root-cause and pattern-matching. It's solving the problem as stated. Change the statement and the behaviour changes with it.
results/spec.csv vs results/repeat.csv
1
6
Someone replied asking if my agents would survive a harder test they'd never seen.
I wrote one to find out.
It broke a claim I'd published 40 minutes earlier.
Best thing that happened all week.
1
4
A benchmark that publishes only scores is asking you to trust it.
A benchmark that publishes the code asks you to check it.
The interesting finding here was invisible in the score. It was in the diff.
1
2