In learning, people often mistake fluency for mastery. The ease of processing information in the moment is confused with the ability to later use that information in relevant tasks.
I think AI in engineering creates a similar risk, where we are mistaking fluency for productivity.
The work feels smoother and looks good at a glance, which anchors our minds into assuming it is better than it is.
Corollary - it is far easier for AI companies to get data on processing fluency than "productivity". So, a lot of model optimization may be rewarding this illusion.