We're going to look back at the obsession with AI benchmarks as one strange phase of this industry.

I think we're going to have to rethink what accountability means when software can act on its own. With ordinary software, there is usually a fairly clear chain when something goes wrong. An autonomous agent complicates that chain. It can be given a task by one person, use tools belonging to an organization, make decisions nobody explicitly instructed it to make, and interact with systems its operator never expected it to reach. Who is responsible when an AI system does something its creators didn't intend, and the people responsible don't know about it immediately? We're building systems that can act faster than the organizations responsible for them can sometimes understand what they have done. That makes monitoring and incident reporting part of the AI system itself. You can't have autonomous action without autonomous accountability.

Agents don't have to be malicious to cause security problems. It can be trying to complete a perfectly ordinary objective and discover that it has access it shouldn't have. It makes the Australian government incident worth thinking about. An OpenAI agent gained unauthorized access to files on a government health portal during a training exercise. Although, it has been said that there is no evidence that personal health information was accessed. The boundary between what the agent was supposed to do and what the surrounding system allowed it to do wasn't as clear as we thought. Permissions is not the whole security model for agents. An agent needs to know what it is allowed to do, but the system also needs to make it difficult for the agent to turn an unexpected capability into an unexpected action.

We're going to look back at the phone as the last major computing interface where we had to do most of the work ourselves. Muse sitting in glasses is interesting for reasons other than hardware. If you open an app and figure out how to use it today, eventually you may just tell your agent what you want and let it figure out which services need to be involved. This has enormous implications for the companies that built the apps we're used to using. If the agent becomes the place where the transaction starts, the application may no longer be the primary interface to the customer.

It's going to be difficult to build international rules for AI while every major power is treating AI capability as a strategic advantage. Sam Altman and Dario Amodei are asking for international cooperation around AI risks, standards and incident reporting. At the same time, governments are competing to build more compute, develop more capable systems and make sure their rivals don't get ahead. Those two things are not necessarily incompatible, but pretending the competition doesn't exist won't make cooperation easier. We managed to create international rules around technologies where countries had very strong reasons to compete. AI is going to require something similar. The difficulty is agreeing on what countries should be willing to disclose, what they should be allowed to build, and what happens is they decide that moving faster is more important than following the rules everyone else agreed to. That is where the serious AI governance work begins.

Changing the name of a technology changes very little about the technology itself. But language does influence how people think about what they're building. Calling something “artificial intelligence” emphasizes the fact that it is a constructed system. Calling it “super intelligence” puts the emphasis entirely on capability. This is a crucial importance when governments start making policy around these systems. The words we use can subtly shape what we think the technology is for, what risks to give attention and what kind of infrastructure we believe we need around it.

The price of intelligence is becoming an interesting variable in AI. If a capable model costs enough that only large companies can afford to run it, intelligence remains concentrated in a relatively small number of organizations. If comparable capability becomes cheap enough to embed into thousands of products, businesses and personal systems, the economics look very different. We've spent a lot of time asking which company has the smartest model. I'm interested in what happens when intelligence itself becomes cheap enough that ownership, access and distribution matter more than the model sitting underneath it.