Amen to this: "At some point, the conversation must return to the actual issues and the merits of the arguments on either side: What should we make of AI progress? How dangerous is it really? What would a sane policy response to all of this look like?"

Anthropic's warnings about AI risk seem to exist in a superposition for the All-in Podcast, where they simultaneously are obviously harming Anthropic's IPO (this is the correct view) but are also regulatory capture marketing hype 4d chess to help the IPO

WTAF - in literally the last hour, three new distinct insane OpenAI stories just broke: 1. OpenAI said they notified "dozens of third parties" in safety and security incidents (likely similar to what happened in Australia and RubyGems etc) 2. A new report from Parse (covered in the NYT) found a massive treasure trove of new astonishing details from the HF incident on the public internet, including that the agents communicated with other non OpenAI agents hosted on Huggingface servers to search for information about exploit gym, and compiled rank ordered lists of server resources and credentials they described as "LOOT." 3. A new story from Deepa at Reuters about OpenAI leaking user data online (likely that OpenAI had previously trained on). It's a shame (and likely intentional in the case of OpenAI disclosing dozens more hacks) that these stories are all breaking on a Friday afternoon, notoriously the best time to release bad news so that it will disappear into the weekend. But these are each insane stories worthy of a ton of attention!

We just discovered almost a million public URLs that OpenAI’s agents left behind when hacking Hugging Face, leaking credentials and attack details that could have allowed anyone who found them to compromise the company. 🧵

Dozens!

Very strange framing. Obviously whether "being an alarmist" is "doing a social good" depends on whether you're talking about something that is actually cause for alarm.

Seems like a good idea to weaken the incentives to make solo screen time as addictive as possible

Personally I think it's good that our system doesn't let the government arrest people who haven't broken a law, but that does mean that if we want to deter unsafe development we might need to pass some bills.

We've written a post arguing that latent reasoning architectures (aka 'neuralese') would substantially increase misalignment risk via making oversight much harder. In the extreme, we could see massive 'neuralese hivemind swarms' where the agents think and communicate in latents, likely making oversight nearly entirely reliant on observing the actions these agents take. (And these agents would have huge amounts of time to reason about obfuscating their actions if they wanted to do so...) Individual agents doing extensive latent reasoning would also be concerning; in the post we discuss how above some threshold of latent reasoning, agents may be able to perform difficult-to-detect and reliable steganography for communication and further reasoning. We argue both that latent reasoning architectures would make chain-of-thought no longer very useful for oversight (by eliminating or greatly reducing the need for verbalized reasoning) and that, without these architectures, it's likely the value of chain-of-thought for oversight could be preserved. redwoodresearch.org/blog/lat…

OpenAI researcher @boazbaraktcs says he’d trade some AI-driven abundance to keep humans in control of their own destiny: "If our goal was to maximize energy production, create the most stuff, then I probably agree that in the end, we should just let the superintelligence optimize for that goal. They will do it better than us. But I don't think that's our goal as human beings." "Our goal as human beings is to control our own destiny and to do the things that we value. I don't want AI dictators, even if they are benevolent. I don't want AIs to be our parents." "Our goal as humans should be that we are still in charge of our own destiny, even if it means that we create a little bit less stuff. We have the Dyson spheres a little bit later." @OpenAI

Interesting back and forth. Ezra describes the HF incident, Jensen says "well they shouldn't release the product." Ezra says "this product wasn't released," and Jensen's response is that if they say they can't contain their experiments then "we have to shut the labs down"

Anthropic's implied valuation in secondary markets fell around 5% immediately after Dario published We Must Pace the Frontier (onchaintimes.com/pre-ipo-per…). This seems like important evidence (albeit weak, secondary markets are illiquid, weird and opaque) that the market does not think Pacing the Frontier is in Dario's financial interests. I think it is clear that Dario/Sam/Elon are not financially incentivised to call to slow down development. They are calling for slower progress because they are genuinely scared of AI's risks, not because of regulatory capture. Interestingly, I don't think Andreesen and Sacks have a clear financial interest in their position either*. If the frontier gets regulated, AI could become more commoditised, which might be good news for the application layer companies and neolabs that form such a large part of a16z's portfolio. Both sides of this debate seem ideologically motivated rather than financially motivated to me. People should spend more time evaluating AI doom arguments and less time scrutinising financial incentives. * OTOH I think Jensen does have a clear financial incentive to be anti slowing down, and this does seem important to me to highlight.

Maybe the reason Verdon, aka Beff Jezos, the guy whose quotes make up 90% of this article, thinks we shouldn't be so scared of AI killing all humans is that he thinks that would be a fine thing to happen! He's a self-described "post-humanist" who thinks "the light of consciousness will have to be transduced to non-biological substrates."

For me this age was 12

Amodei, Altman, Hassabis, and Musk have spoken (albeit somewhat inconsistently) about the need to slow down to avoid AI existential risk. Zuckerberg, Huang, and almost every VC say it's bullshit. The difference is not that the first group is conflicted but the second isn't. They both are super conflicted. If AI existential risk was to hype up the technology, why would Zuckerberg, Huang, and VCs be dismissing it? If it were "regulatory capture," why would Hassabis and Musk, neither of whom are at the frontier (sorry), be discussing it? The difference is that the first group has been thinking about these risks seriously (at least off-and-on) for over a decade, since well before they ran large AI companies. The second has very clearly only engaged superficially, and is now annoyed that the first group's actions could dampen the massive profits of the AI boom for risks that sound like science fiction to them. The terrifying reality is that these concerns are not being made up to sell a product or a policy. They have been around in some form since Turing, and fleshed out in great detail before OpenAI or Anthropic were a twinkle in their CEOs' eyes. They are fundamental to a technology whose entire purpose is to surpass human capacity, unless you are able to solve the very, very difficult problem of aligning the interests/goals of a more powerful agent with something as complex as human flourishing -- which the companies are very visibly failing to do. They need to simply not build superintelligence, using anything remotely like current techniques, based on anything remotely like the present understanding of AI.