In fact THE main AI doomer fear has long been of sudden huge jumps in AI ability. Absent that, we can adjust to problems as they appear. But we just haven't seen such huge jumps in AI, and the history of tech gains suggests the odds are greatly against.
Suppose labs stumble on a new scaling axis with a multiplicative effect on existing compute, allowing us to go from the equivalent of GPT-6 to GPT-60 ~overnight. This risk also passes through automated AI R&D, but is a bit different from imagining GPT-N training GPT-N+1 using current recipes. It also seems strikingly plausible given the evidence of substantial algorithmic headroom, and fits the pattern of recent speed-ups like the 2-4x efficiency gained from inference-time compute, and perhaps similar gains from recurrent-depth. However, I'm gesturing at something bigger than a fixed efficiency multiplier. If Performance = F(pretraining, inference) I have in mind a breakthrough implicating F() itself. An algorithmic surprise like this is a central scenario for rapid takeoff.