This messes with my mind ALL THE TIME
When I do stuff with AI I am an *expert* in (eg coding on my project I know well, or doing research in an area I am very familiar with) I spot problems/errors/the model BS'ing me immediately. No big deal, I have it correct or start again
BUT when I'm not an expert in an area, I find that I trust it, and then later learn it does stupid stuff. Like a case when I asked 3x AI models to give feedback on a legal case (that I also paid a lawyer for, but lawyer was slow). All 3x models confidently told me X, lawyer then told me Y, and lawyer was right
Claude Opus 5.5 is great, and most of the time, the solution is similar to how I'd implement it myself.
However, every now and then, I find terrible design decisions that would fail in production, and that's when I realize how important software engineering is, and will be, years from now.