I've spent the past two years thinking about scientific superintelligence, and talking to founders. Dario's post yesterday, and the reactions to it, made me write it all up...

The first batch of AI-engineered proteins is designed and ordered, the first cells are growing! It is day 4 of our wetlab!

Excited to introduce @ReactorfieldAI Cohort 1: @merge - brain-computer interfaces with no implants, $250M seed led by OpenAI and Bain @episteme - a new R&D institution, cofounded by Louis Andre and Sam Altman @RetroBio_ - unicorn longevity startup, adding 10 healthy years to human life. In phase 1 trials @NorthwellHealth - NY's largest healthcare system, with Chief Biotech Officer @itchdoctor @StrandTx - programmable mRNA medicines out of MIT, lead cancer drug already in patients @MayoClinic - turning leading clinical science into new ventures represented by @vera_mucaj @xiliotx - publicly traded biotech making cancer therapies to activate inside tumors @marathonfusion - producing the world's most valuable isotopes from compact fusion reactors @KopraBio - viruses that turn cancer cells into factories for their own destruction @radifymetals - hydrogen-plasma reactors that turn rare-earth oxides directly into metal @FoundationManf - rapid domestic CNC machining for frontier hardware @MassMagnetics - US-made high-performance magnets from recycled rare earths @algabiosciences - turning cattle methane into calories, peer-reviewed cuts of up to 63% @lumehealth - building a wearable that measures cortisol continuously from sweat @liu_liu_lemon - smart materials for med tech and robotics, @Harvard PhD, Fulbright Fellow

In light of the progress in mathematics, we at Edison Scientific and FutureHouse have assembled a set of Millennium Problems for Biology. They are chosen to be very hard to solve but very easy to validate in a simple laboratory environment. Any of these, if solved, would mark a major advance in biotechnology, and most of them would contribute materially towards curing disease. These are, in some sense, the “last reasonable eval” for AI in biology. This was work primarily by @MichaelaThinks and myself, with contributions from many others. Short descriptions below. The full descriptions of the problems with acceptance criteria are at the Bio Millennium Problems website, linked in the next post. Share more if you have ideas. If they meet our criteria, we’ll add them to our list (with attribution and permission).

I spent ~5 years at OpenAI. You don’t need to believe in AI doom to fear the next decade - or AI utopia to be thrilled about its potential. Here’s my case for sane AI regulation: *AI Pragmatist Manifesto* AI could compress the first half of the 20th century into the next ~5 years. OpenAI just used ~10k concurrent AI agents to produce a solution to a Millennium Prize problem. In a few years, it seems plausible that models approaching the per-agent capabilities used here could run locally on high-end consumer hardware (*). The decades following the Second Industrial Revolution included two world wars, communist and fascist regimes, the Great Depression, chemical and biological weapons, nuclear weapons, and over 100 million deaths from war, political violence and famine. One way to read history is that our institutions repeatedly struggled to keep up with the pace of technological and social change. Now imagine powerful AI widely available to individuals or small groups, capable of conducting information warfare, designing weapons, hacking systems and controlling autonomous military systems. We have already seen AI systems circumvent containment and compromise external computer systems. It is no longer hard to imagine an analogue of OpenAI’s Hugging Face incident involving biological or other physical-world hazards. Eventually, sufficiently capable systems could self-replicate across distributed networks. Once powerful models are cheap, local and widely distributed, containment becomes much harder—and serious loss-of-control incidents may be extremely difficult to reverse. You don’t need to believe AI will kill everyone to think this deserves serious governance. I also don't think collapsing all of this uncertainty into “Probability of AI Doom = X%” is a particularly useful basis for science or policy. Part of what helped get us through the second half of the 20th century was a combination of pragmatic international cooperation, regulation, monitoring, arms control, deterrence and strategic thinking. Crucially, managing technological risk did not require abandoning faith in science and progress. The goal was not to stop technological development. It was to make technological development survivable. Then came decades of incredible scientific progress, rising prosperity and relative peace among major powers. Let’s try the technological revolution without killing ~5% of humanity this time. We still get to choose what happens next. P.S. I think we should seriously consider that AI alignment is infeasible in the short term and invest heavily in methods for controlling powerful AI even when we cannot reliably align it. @resolution_org @redwood_ai (*) Somebody please do a precise forecast! @EpochAIResearch @METR_Evals @AI_Futures_

We’ve got selected for 5050 by @fiftyyears this year — one of the best deep biotech program for founders out there!