Optical interconnection needs are exploding — and reliability is a huge issue when you are at 1.5M transceivers. Really interesting Google talk on this at @OpticaWorldwide ‘s PECC today.

Thank you for taking the time to respond thoroughly on a weekend. I realize that drafting legislation is difficult and some degree of iteration and response to feedback is inevitable and appropriate. However, I feel like your response is still not really engaging with the heart of my concern about Anthropic's behavior here, and it makes me think that these sorts of problems and tendencies will only resurface moving forward. I also think that now might be a good time to take a broader step back as to where we find ourselves and how Anthropic sees its role, and how external folks who care about AI safety like myself increasingly see Anthropic. (Fair warning: this response is a bit long) To explain what I mean (and at the risk of being a bit esoteric), it’s worth asking the question: what is Anthropic? How do its leaders and employees think of Anthropic’s role? Is that conception matched by how it actually behaves and impacts the world? One answer is an extremely large tech company that exists to make its employees and investors mind-bogglingly wealthy. Anthropic is sometimes described as a startup, but if Anthropic successfully IPOs at 2 trillion dollars (a number it has been reportedly targeting), it would be more valuable than Meta - which is worth 1.8 trillion. Another answer is an AI safety organization, whose core values are those of the sort laid out in the 2023 essay from Anthropic’s co-founders entitled “Core views on AI safety: When, why, what, and how.” In that essay they said that if we were in a world where alignment was hard (and that without strong evidence to the contrary, we should assume this is the scenario we are in): “Anthropic’s role will be to provide as much evidence as possible that AI safety techniques cannot prevent serious or catastrophic safety risks from advanced AI, and to sound the alarm so that the world’s institutions can channel collective effort towards preventing the development of dangerous AIs. If we’re in a “near-pessimistic” scenario, this could instead involve channeling our collective efforts towards AI safety research and halting AI progress in the meantime. Indications that we are in a pessimistic or near-pessimistic scenario may be sudden and hard to spot. We should therefore always act under the assumption that we still may be in such a scenario unless we have sufficient evidence that we are not.” There have been more recent articulations of this version of Anthropic, for instance in Dario’s recent essay warning: “We must slow the pace at which we improve the capabilities of AI models… it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet… incidents have happened across the industry, including at Anthropic, and I believe it’s incumbent on every frontier AI company to act as if OAI-HF had happened to them… more intelligent models are more capable of deceiving tests, and thus may appear aligned while having serious problems that go undetected…” A third view is that Anthropic exists not as a normal profit motivated company, or an AI safety organization seeking to help raise the alarm and solve collective action problems associated with AI development, but instead as an organization primarily concerned with creating superhuman AI capabilities before anyone else, in particular through creating a process whereby the current generation of Anthropic’s AI systems can autonomously build their successors, and those successors build their successors, and so on. The motivations for wanting to win this race may be manifold - maybe for the pure profit of Answer 1, but perhaps also because Anthropic believes that the best way to make the world better is for them to win this race instead of their more reckless colleagues; or, because of normal human pride and ambition. But the results (and risks, which Anthropic claims existentially threaten every single human being on earth) are the same. Anthropic employees and leaders may seek to answer this question on what Anthropic is by introspecting on their own intentions and those of their colleagues. But those of us outside the organization have more limited tools available to us - we can only see what Anthropic and its employees say and do, not how it feels or looks on the inside. When I look at many notable actions that Anthropic has taken over the past year, that evidence seems most consistent to me with the view that Anthropic is primarily trying to win the race to super intelligence, and that other considerations (including safety) are fundamentally secondary. These actions include: 1. Announced plans to build an in-house custom silicon chip team to accelerate the inference speeds that Claude operates at. 2. Inked deals for more than $500 billion in compute (according to the Information) to further accelerate AI development. 3. Increased the percentage of AI development tasks that Claude “leads” from 0% in March of this year to 26% August (what is it in October?) 4. Spent a massive percent of compute on internal AI R&D to accelerate development, including use of internal models far better than are available to the public. 5. Switched to Responsible Scaling Policy V3, which abandons its unilateral commitments to slow or stop development if Anthropic is unable to provide sufficient risk assurance. 6. Was the first company to create an alternative weaker “frontier compliance framework” to supplement its RSP, which contains no binding provisions whatsoever on internal deployments. 7. Wrote a letter to Congress (that it later walked back) saying that incidents where Claude was hacking random people were “best understood as a consequence of the misconfiguration, rather than evidence of misaligned goals.” 8. Commented “you can’t do safety from second place,” after the WH SI Accords. 9. Shared a draft bill to Congress that would preempt significant parts of state law (including with language that appears intended to prevent states from blocking dangerous AI deployments), and replace it with federal rules that (despite also containing good provisions) seems to create an enormously high bar for the executive branch to block development or deployment of Anthropic or another AI companies systems. Of course, this is a deliberately one sided list. Anthropic also published important AI safety research, advanced novel collaborations with third party AI safety organizations, and forcefully spoke out on the importance of AI safety in many public fora (or in private , where I realize a lot of the most valuable work is done and may not be visible to me or those outside the organization). But I think it’s worth sitting with how the above list looks to someone outside the company. The sense I have is that for those within the company, many of the items above are treated as anomalies or mistakes that don’t reflect who Anthropic really is. But I have every confidence that if employees at Anthropic saw one of their competitors (e.g. at OpenAI) take some of these actions, they would take it as confirmation of all their worst suspicions. (They might even call OpenAI employees a little gullible for accepting their leadership’s explanations for this behavior at face value.) Perhaps these were mistakes in the sense that they are not decisions that Anthropic leadership would endorse, but the actions still happened because of incentives and decisions that Anthropic leadership is ultimately responsible for (and would not accept similar excuses from another organization). When Anthropic strays and makes mistakes, why do the mistakes systematically tend to guard and favor Anthropic’s interests? For every provision that was too weak or a giveaway to AI companies in this legislation, why wasn’t there a mistake pushing in the direction of being too strong? A core part of Anthropic’s theory of change is that being at the frontier gives you more leverage to advocate for stricter policy. I think that conceptually makes sense, but in practice I’ve seen Anthropic push the frontier without substantially pushing the ‘safety frontier’ a remotely equivalent amount. I want to end by quoting from Anthropic co-founder Chris Olah’s comments around the Magnifica Humanitas release: “Every frontier AI lab—including Anthropic—operates inside a set of incentives and constraints that can sometimes conflict with doing the right thing. The pressure to stay commercially viable and to stay at the research frontier. Geopolitical pressure. And the older, plainer pressures of pride and ambition. No matter how sincerely any of us intend to do the right thing—and I believe many of us do—we will always be influenced by those incentives. That is why, if we want this technology to go well, it is enormously important that there be people outside those incentives—people who care about things going well and insist on safety, who are paying close attention, who are willing to say hard things, who are willing to be our earnest, thoughtful critics.” I hope these remarks are taken in that constructive spirit, and not as an attack on anyone specific at the organization (certainly not you Anthony, who I have only personally seen act professionally and thoughtfully, and who only ended up getting this diatribe in your replies for random contingent reasons).

Our cultivation experiment got published in Science. Emma did a good job summarizing it in youtube.com/watch?v=XtTr87pi… There's a lot left to do. The hard part of the theory paper was escaping into a BIG code (~500 qubits). Which just. doesn't. fit. (yet) But what fits; that works.

For the first time today: the culmination of 15 years experimenting with digital history—and what it teaches us about navigating archives with AI. 1:30 ET, Baltimore + livestream: proflouishyman.github.io/ai_… @DuttonBooks preorder: bit.ly/WarOverRent

Re: RSI, definitely finding frontier models are getting v good at making evals and (some) synthetic data. Feels like <6 mo

I'm super focused right now on bringing verification into machine learning. So Satya Nadella's recent post caught my attention: questions we work on in engineering are becoming central to how companies and governments use AI. (snscratchpad.com/posts/model…) Then Anthropic published its self-replication study. Reading them together raised a question that also comes up in scientific work: what evidence would justify giving a system more responsibility? (anthropic.com/research/self-…) Satya argues that controls must sit outside the model. I agree. You should be able to restrict what an agent can touch and withdraw its authority without depending on its agreement. I want to make two requirements explicit. An agent can follow every permission rule and still get the answer wrong. It can also get the answer right by doing something it wasn't allowed to do. We need evidence that it stayed within its authority, and evidence that its result is fit for use. So, the Anthropic study. They ran a model across 15 cheap TV boxes. Separately, they tested self-replication on an emulated Android fleet. I think they're interested in understanding what self-propagation could look like beyond what we already saw with the hugging face incident. I see, instead, something prosaic: local deployments make responsibility harder to assign. The developer may have no way to stop a copy running elsewhere. A device owner, network operator, or service provider may still control something it needs. The challenge is that these heterogeneous environments require integrated controls that have to be tested together (like NIST wrote about in the risk management framework in 2023!) The other thing that is challenging here is that even in a managed service, stopping the model is only part of a shutdown test. What happened to the programs it started, the work it delegated, and the requests already in flight? Now the overlap with Satya's piece: an audit log helps, provided the agent can't alter it. And verification gates (including for more resources) constrain behavior. Okay, now the part closer to my day-to-day: science and engineering. In another experiment, Anthropic passed notes between successive runs of a model, and setup success improved. That's useful. It also raises a question for scientific workflows: when one agent passes work to another, how do we keep assumptions and errors from passing along unnoticed? Engineering gives us a starting point. Verification checks whether a calculation implements the mathematical model correctly. Validation asks whether that model represents the physical system well enough for the intended use. We also need to quantify uncertainty and say where the result applies. (see, e.g., nist.gov/publications/summar… for some depth on this) On the lean side, a formal proof can establish that a conclusion follows from stated assumptions. We still have to check that the formal statement captures the question we meant to ask. For a physical prediction, we also need evidence that the assumptions describe the system we're studying. AI can help with the checking. But our formal tools are really limited, particularly on the sociotechnical part of closing the semantic gap (the difference between what we intended and what we said; and the difference between the lean code and what we want to read from it). Turning to another model can find errors or propose tests. But agreement between models is only useful if we understand the checker's limits. Can it catch known errors? Does it still work when the task or model changes? An independent reviewer needs enough access to evaluate that (and see the math community's outburst on becoming the checkers of alien proofs) For the team at @axiomatic_ai, we're focused on ensuring that result, its assumptions, and its checks stay together so the next person can decide whether to use it. Failed or incomplete checks should stay visible. Any decision to proceed despite them should belong to someone authorized to make it. Last thing. Congress has proposals that address parts of this. The VET AI Act would ask @NIST for voluntary assurance guidance, including fitness for intended purpose. The Stop Rogue AI Act would address agent discovery, control, correctness, and reliability, with demonstration projects for agent discovery. Take a look at what's circulating here: govinfo.gov/content/pkg/BILL… govinfo.gov/content/pkg/BILL…

Funny how 'AI safety' keeps landing on the same conclusion: fewer players, more control, and the ones talking are the ones who keep it. Open source isn't the threat. Concentration is.

"What’s good in effective altruism isn’t new, and what’s new isn’t good. Yet some effective altruists have power over artificial intelligence, and that’s a problem for us all." - @LeifWenar economist.com/by-invitation/…

AI時代に起きる最大の悲劇は《仕事を奪われること》じゃなく、【20年間追いかけてきた夢が、たった20分で叶ってしまうこと】なんよな、これを勝手に《夢の即死問題》と呼んでいるんだけど、例えば「いつか自分のゲームを作りたい」と思っていた会社員が、Claude Codeに話しかけたら30分後には遊べるゲームができてしまう、「いつか小説家になりたい」と思っていた人が、ChatGPTと週末に一冊書き上げてしまう、Sunoで作曲家になり、Soraで映画監督になり、Cursorで起業家になる、もちろん《プロ品質》や《商業的成功》とは別の話だけど、これまで人類は「才能がないからできない」「時間がないから挑戦できない」「技術がないから形にできない」という《3大言い訳》に守られてきたとも言える、それがAIによって一気に剥がされると、恐ろしいことに【夢を叶えたのに、別に楽しくない】という人が大量に現れるかもしれない、つまり人間が欲しかったのは《作品》なのか、《作る行為》なのか、それとも「いつかすごい自分になれる」という《未来の自分への期待》だったのかが、いきなり試されるんよな、これまでの社会は《才能の証明》に10年かかったから、その10年間を“夢”として生きられた、でもAIがその期間を10分の1、100分の1に圧縮すると、夢の賞味期限まで短くなってしまう、そこで始まるのが、これも勝手に名付けた《夢のデフレ》で、映画を作れることも、アプリを公開できることも、会社を立ち上げられることも、それ自体はどんどん珍しくなくなる、すると逆に価値が上がるのが【誰にも頼まれていないのに、なぜか作り続けてしまう人間】なんよな、100本のゲームを作れる人より、101本目をどうしても作りたくなる人、100曲生成できる人より、たった1曲のサビが気になって朝まで直してしまう人、AIが《能力の格差》を縮めるほど、《執着の格差》がむき出しになっていく、これは《情熱資本主義》とでも呼べる変化で、学歴でも職歴でも資格でもなく、何に異常なほど夢中になれるかが、その人の競争力になっていく、皮肉だけど、人類が何千年もかけて《できないこと》を減らしてきた結果、最後に残る問いが「で、あなたは何をしたいの？」という、子どもに聞くような質問に戻ってくる、そしてもっと面白いのは、そこで初めて自分の《本当の好き》を発見する人が大量に出てくる可能性で、AIが人間から奪うのは仕事だけじゃなく、“やらなかった人生への言い訳”なのかもしれない、でもその先には、誰もが何度でも自分の人生を試作できる【人生のプロトタイプ時代】が待っていると思うと、ちょっとワクワクするんよな…

I was once very involved in programs like this at DOD. I had missed this stunning - mindblowing, really, unless you know the Army - statistic about the new service's new AI track:

I'll say it again for the AI risk experts in the back: We've already run a real life, large scale experiment on what happens when a superintelligence is unleashed on a narrow objective. Look no further than U.S. capital markets. Millions of investors, analysts, traders and algorithms, backed by trillions of dollars and wired together through prices, all chasing one goal: the highest risk-adjusted return on capital. And it is spectacularly good at its job. It funds the best ideas, punishes the worst ones, and turns new information into prices faster than any of us can react. No individual can out-think it. 99+ percent of professionals can't beat it. But look at what else it's done along the way: - Optimized the measure instead of the intent. Quarterly earnings became the scoreboard, so companies learned to manage the scoreboard. - Treated anything without a price as free. Pollution, addiction, obesity, depression, political fragmentation, fragile supply chains, the long-term health of the communities it operates in. - Found every gap in the rules, then paid lobbyists to help write the next set and pull up the ladder behind incumbents. - Remade the world around its objective. Companies, pension funds and our retirement accounts now answer to the S&P 500. - Made itself impossible to switch off, because everything else now depends on it. None of this required malice. Nobody sat in a room and chose these outcomes. It's simply what a very capable optimizer does when its objective is a narrow proxy for what we actually care about. And note that we never solved this alignment problem. We've managed it, imperfectly, with regulation, taxes, disclosure rules and the occasional crisis to show us where the guardrails were missing. Over time, the market has captured the instutions designed to constrain it. We've been at it the better part of a century, and the misalignment is the worst it's ever been. So the next time someone tells you AI is only dangerous if it "wants" something bad, or if its goals are mis-specified, point them at the S&P 500. It doesn't want anything. It just optimizes. That's the whole problem.

Kudos to Anthropic for publishing this. There is still an uncomfortable gap between our scientific understanding of alignment and its importance. Such reports help addressing this gap. I hope they also publish the process by which new incidents will be chosen to disclose!

Announcing the Most Ambitious Set of Science Initiatives This Century whitehouse.gov/fact-sheets/2…

2026-2027 UMD postdoc application thread. First up is the Hartree QI fellowship. umd.wd1.myworkdayjobs.com/en… @UMDscience @UMDPhysics @math_umd @umdcs @UMDResearch

I’m driven by the dream that we can make AI that truly understands the physical world, that makes it possible for machine intelligence to work with us on the tough problems ahead. And that by doing so, we can reign in bad behavior with verification and validation in the physical world. Our products are based on these principles, and our team is heads down working on this every day. Well, today we get to stick our head up for a moment. I'm happy to announce that Axiomatic has been selected, along with a cohort of other startups, to showcase what’s possible, at the Google Startup Forum next month in Mountain View. It’s a unique honor, and opportunity, to learn from the best, to understand what future we can build together, and to open the aperture on how scientific general intelligence can help more than just scientists.

“A quantum code for which there existed a universal set of encoded gates each transversal in isolation would be extremely useful.” -- Eastin & Knill arxiv.org/abs/2610.06730