Inspired by David Ng’s post, I asked both AFM and Qwen 3.5 4B on my phone
What’s the cube root of 74,088,893,247?
Anyone should be able to mentally get this to about 4,x00 (the exact result is 4200.01x) as we can clearly see from 4^3 gives us 64, and 1000^3 gives us the billion, and 5^3 overshoots to 125B.
The interesting thing is both models failed in the same way, both delivering 42,000.
Apple did it with not much fuss, a confident guess.
Qwen threw out a huge amount of reasoning, peppered with lots of TeX, which all seemed plausible-ish, but at the end of the day screwed up.
I kept telling each they were off by a factor of ten and they would apologize with a kind of “I’m so sorry, you’re right, the answer is 42,000”. Over and over!
I’m guessing with all the work of the past week the new normal for small models (at least on Apple) will move from ~4B to something more like ~40B.
Curious to see if that gets us better estimators and better correctors (who actually DO something when told they’re wrong, not just apologize and keep doing the same thing!)