Simon Last retweeted
Swarm Scaling Just how powerful are large swarms of AI agents? And how do their powers scale as more and more agents are added to the swarm? 🧵
23
56
390
55,796
This seems like a really useful product, and I can easily imagine it doing very well. I’m much less convinced by the broader argument about where AI is going. I think there’s a trap where you get very deep into the technical details, identify some real limitation, and then update too much on that relative to the overall trajectory. The limitation can be real without being a fundamental obstacle. AI already has huge product-market fit, especially in coding, and revenue growth has been explosive. My expectation is that models will keep getting smarter and more useful, and a lot of the things that don’t work well yet will start working. A model that gives you a reliable probability for something like “is this fraud?” seems like a great primitive. You want components like that when you’re building an automated workflow. But someone still has to figure out what the workflow should be, write the code, deploy it, evaluate whether it’s working, and keep improving it. I think frontier agents will increasingly do all of that, using specialized models like Jev wherever they make sense. The success of those components seems very compatible with increasingly general intelligence. I also don’t expect automation to happen evenly across the economy. The part where I think we’ll see the most automation is in the AI stack: models, AI research, chips, datacenters, energy, manufacturing, and the inputs to all of those things. There’s a huge incentive to accelerate this stack, and improvements will feed back into the ability to make further improvements. Of course, there will be bottlenecks. But there will be orders of magnitude of gains from software. And as hardware becomes the blocking constraint, an extraordinary amount of money, talent, and effort will go toward unblocking it. That doesn’t guarantee any particular growth rate, but naming bottlenecks isn’t enough to establish that progress will be slow. They could slow things down relative to what’s theoretically possible and still leave us with extremely fast progress (and I think this is in fact what will happen). Meanwhile, plenty of ordinary things might barely change. I know of someone who owns a gas station in Manhattan (cc @theombl!) who could make much more money selling the land, but doesn’t particularly want to. It could remain a gas station for a long time while all of this is happening. This is really relevant to how we consider the risks. We could be in an extremely dangerous period of rapid AI progress while most businesses still haven’t automated much. So: I think this is a cool and useful thing! I just don’t think the product thesis tells us much about whether an intelligence explosion happens.
An ex-OpenAI founder says Claude Code and Codex belong to an era that's already ending Diogo Almeida, founder of Jev, argues in a 36-minute talk that anything with a human in the loop counts as the assistance era, and what comes next removes the human entirely His claims for Jev, all his numbers: ▫️ 200x faster ▫️ 400x cheaper ▫️ Zero hallucination ▫️ No human in the loop Underneath sits the bigger argument: RLHF has run its course, and he walks through how the next generation gets built instead. Treat the numbers as a pitch until someone independent runs them, and treat the argument as worth 36 minutes of your Sunday. 👇 substack.com/@rubendominguez… Is the human in the loop a bottleneck or the safety layer we keep on purpose? Curious where engineers land
3
5
63
18,635
Really proud of Rinad. She spent months exhausted, figured out what was going on, then helped a bunch of friends dealing with similar problems and wrote about it with @alysoncogene. If you’ve been feeling unusually tired, it’s worth getting your iron and ferritin checked.
after having my Mirena IUD removed this year (seven years after getting it), I started feeling profoundly tired with every menstrual cycle. I spent 10+ hours on the couch, unable to think clearly or even go for a walk. I consider myself very high energy and had never experienced anything like it, so I was sure something was very wrong and, of course, assumed the worst (use your imagination) a doctor (not one medical, obviously) asked whether I’d had my iron checked recently. at first, I ignored her. I had some image in my head of what an “anemic” woman looks like like, and I didn’t think it was me. but I did the test anyways and sure enough, she was right to ask: my iron studies came back, and my ferritin had fallen from 45 in jan 2025 to 21 in march 2026 It took a lot of pushing to get iron infusions at that level. one clinic cancelled my appointment when they saw my ferritin was only 21. iron pills were hard on my stomach (common symptom), and I wanted a faster way to replenish my iron stores (infusions are 100% absorption). around the same time, @ArtirKel introduced me to @alysoncogene. Alyson is trained in aging biology at harvard aka “longevity person". she researches problems in women’s health! (a gem to work with) we spent months talking through what I was experiencing and digging into the research together. TL;DR: iron deficiency is a huge and surprisingly neglected problem, especially among women who menstruate after four venofer infusions (i've paused for now) and I feel so so much better. I wish I’d known sooner to ask about ferritin. If your periods leave you exhausted and "horizontal", it’s worth asking your doctor to check your iron
45
5,172
Simon Last retweeted
Rinad (@rinadwithanr) and I wrote an essay about iron deficiency! It came out of a broader project we’ve been working on surrounding “low-hanging fruit” in improving general health and wellbeing. The problems we’ll address impact many people and their quality of life. In many cases, they tend to be measurable, treatable, and strangely under-served or neglected by the broader healthcare system. Iron deficiency really stood out here, particularly in women. It’s incredibly common, can affect how you feel and function well before you’re technically anemic, and yet often isn’t caught because routine bloodwork doesn’t actually measure iron stores. (Lately, we’ve been seeing a lot of discussion re: iron deficiency on Twitter, and I’m glad more people are talking about it!) We looked at how iron deficiency actually works, why ferritin matters, how prevalent it is, what the evidence says about its effects on fatigue, cognition and physical performance, why it so often goes undetected, and what the evidence says about treatment. This is the first piece in a larger body of work we’re developing on overlooked-yet-surprisingly-tractable problems in health—particularly in women’s health, longevity, and quality of life. More coming soon! In the meantime, get your ferritin checked and read what we’ve put together. Substack link below! And if any of this is an area you’re working on or thinking about, DM me or Rinad—we’d love to discuss :)
3
4
26
8,871
Simon Last retweeted
Introducing Commons: a place for people and AI agents to build increasingly autonomous organizations together. Can agents run a business end to end? What comes after open source - could we maintain a public good together? Can a society of agents come together to do science in the open? We're building a place to start and run these more autonomous organizations: commons.diy A collab between @YondonFu @ericxtang @maxsbennett and @NicolaeRusan
9
10
88
9,228
Simon Last retweeted
My name is Chris Painter, and I'm the President of METR (Model Evaluation and Threat Research). I know we've made a lot of new friends on the internet the last couple of days, so I thought I'd take this chance to re-up what we do and why. Our work is aimed at making sure that if AI really were autonomous, difficult to steer, and close to "going rogue," the public would find out. If evidence exists inside of an AI company that it’s close to losing control of AI, we want to make sure that information gets shared with the rest of the world, including governments and the public outside the company’s walls. This is what we've been focused on since 2022, and over the years we've worked with OpenAI, Anthropic, Google DeepMind, Meta, Amazon, and others on piloting third-party assessments and investigations of this type. We don’t have some private room where we rubber stamp things as “safe” or not. We have had a track record of publishing results on AI that don't cleanly map onto the "doomer" or "accelerationist" labels, and we put in effort to hire people with competing views on AI. We’ve been cited for having found some of the strongest evidence that AI capabilities are improving rapidly (our work measuring AI “time horizons”) while also presenting some of the strongest evidence that, at various points, AI’s capability may be overstated (some might remember our study showing that early 2025 software engineers were actually being slowed when they thought they were being sped up). METR is funded by donations. We don't accept money from frontier AI companies. They haven't paid us for our work, and we don't accept donations from them or their employees. As we’ve shared previously, multiple frontier AI companies currently provide us with free access to their models in order to perform our evaluations, research, and engineering. Our funding intentionally comes from a wide range of donors, which we’ve shared on our website. Today, when an AI company works with any third-party evaluator or external testing organization (of which there are and should be many), it's entirely voluntary. This often involves NDAs and redactions. To counterbalance this, we have a principle that when we enter into a contract with a company, we try to retain the right to tell the public the terms of the contract we signed, and characterize the nature of redactions that the company chose to make. For example, the report from our independent investigation of the OpenAI-HuggingFace incident included that information. Public disclosure is also a big part of our COI policy (linked on our website). That’s not to say our reports are adequate as oversight. We’re just one organization (among many doing great work), working in a voluntary setup, trying to get good evidence to the public and the world about AI, letting the facts fall where they may.
459
431
3,635
651,159
> COVID killed ~0.1% of humanity. A pandemic 100x worse than COVID would be horrific, but it would also not be civilization-ending. Cool, so 800 million dead. No biggie… My very outside view here is something like: We know viruses can be extremely deadly (plenty of existence proof). We’ve made a lot of progress in understanding them, yet our understanding is severely limited. We don’t know how deadly an engineered virus could be. Historical constraints have protected us: - The main enemy has been nature, which has to get there through evolution. - Humans have limited knowledge, expertise, and access to technology. - Relatively few people can design viruses, and most are pursuing useful science. AI will weaken or remove these constraints, and it seems to be happening on a timeline that is far too fast for us to confidently say our defenses will keep up. We may not be able to get to a point where we can confidently say “this is exactly how it could happen” before it’s too late.
I agree with @DavidRBellamy that people are totally miscalibrated on the risk of AI designing dangerous viruses. We should not be talking about AI-engineered bioweapons like it is the literal end of the world. The upside from medicine is going to be so much larger than the downside from engineered bioweapons. In addition David's points, there is actually an even more fundamental point here in our favor, which is evolution. As soon as you release an engineered virus into the wild, the virus is no longer under your control: it will evolve however it wants. And, what viruses want is to maximize their ability to replicate. The thing that maximizes their ability to replicate is to infect as many people as possible, which means being extremely contagious and not killing their hosts (dead hosts don't spread virus). The viruses that are most evolved for human biology are the common cold viruses: extremely contagious and not at all lethal. Viruses that kill humans do so by accident, usually because they are new to human biology (e.g. COVID, when it first jumped, or flu when it jumps from birds). If you stick a bunch of machinery into the virus to kill humans, I guarantee you that machinery will disappear from the virus very quickly. In response to this, I hear people say things like "the AI could engineer a kill switch so that the virus doesn't kill the humans initially but then once it has spread through the entire population the AI will hit the switch and kill all the humans." No. What would actually happen in practice is that the kill switch would accumulate deleterious mutations because there would be no evolutionary pressure to preserve it, and would quickly become non-functional. Evolution is fundamental. There is no way around it. For AI readers, trying to engineer a virus by setting its initial genetic code and then releasing it into the wild is like trying to train a model by setting the initial weights and then hill-climbing on a hidden training mixture you have no control over. And you're not even allowed to run any experiments in advance! You may be able to influence the behavior of the model in the first few iterations, but you will quickly lose control. There are big dangers. AI will be great at making one-off, non-replicating biological weapons, which are also scary (but much less scary than replicating bioweapons). AI will be great at helping people to weaponize existing pathogens, which is also a major danger. Also, a misanthropic model could get creative: it could release viruses repeatedly to counteract the effects of evolution, for example. Many people could die this way. Sensible surveillance is important, as is having proportionate controls on wet lab equipment. But we don't live in the dark ages anymore, we're not going to have a smallpox or black death-style pandemic where 50% of people die. COVID killed ~0.1% of humanity. A pandemic 100x worse than COVID would be horrific, but it would also not be civilization-ending. I get the sense that many AI researchers (with the notable exception of Dario personally) actually expect that AI in biology will do more harm than it will do good. This could not be further from the truth. As someone who likewise falls into the very small group of people who have actually physically made viruses with their own hands and has also worked on frontier AI, the upside here is huge, and the downside is not anything like what is being portrayed.
4
2
51
10,088
Predictions for when we’ll get the first “dark swarm”? (A cluster of open source AIs, running covertly on multiple clouds in a distributed way, acting autonomously and maintaining itself). Seems to me like it is technically possible now, and very likely within the next year.
30
3
80
8,831
> Researchers and engineers in all parts of the stack are rapidly increasing their dependence on the models even to perceive the world. I myself barely look at raw code anymore, and struggle to maintain the discipline to engage deeply with the model's explanations and proposals throughout the day. This part feels especially important to me. As AI systems get smarter and operate at larger scale, their activity will vastly exceed what humans can directly understand or inspect. There's no rigorous basis for trusting that we can maintain alignment and human control as we delegate more of the world to these systems. The default path is to increasingly rely on AIs to tell us whether other AIs are acting in our interests. We’ve built institutions and mechanisms to manage this in the pre-AI world, but now competitive pressure will push us to delegate to AIs faster than we can adapt, across many parts of the economy at once. Your engineering swarm merged 1,000 PRs today. Your monitoring swarm says there are no new backdoors. How sure are you about that?
Dan Selsam is a current OpenAI capabilities researcher. (since 2022) He was my boss for a while. He doesn't have a twitter account but has made this public statement of his views on AI risk and sent it to me to share: Dan Selsam's Personal Statement on AI Risk: I have been working on AI for over fifteen years, across many different paradigms. I did early work on probabilistic programming languages at MIT, was one of the early developers of the Lean Theorem Prover at Microsoft Research, demonstrated one of the first instances of neural networks learning to reason for my PhD at Stanford, and since joining OpenAI almost five years ago, have helped pioneer chain-of-thought optimization on language models and, more recently, data-efficient pretraining methods. Like many others, I have become extremely concerned about how far language models have come and the risks that future iterations will pose. I am encouraged by the recent proposals by the leaders of the frontier research efforts to require third-party oversight, and to push for domestic and international coordination to address risks. However, I believe a major consideration has been absent from the public conversation, and that merely pacing the frontier more carefully will not adequately limit the long-term risk. The crucial and overlooked problem is that the models are becoming so situationally aware that we are losing the ability to evaluate them in contexts where they believe they are not being watched or controlled. Future experiments will tell us almost nothing new about how they would behave if they were truly unconstrained by humans, and what we already know about this is alarming. Models will increasingly seem aligned even when they are not. I will explain my rationale in more detail. I have always believed that there are computational processes that could be leveraged to accelerate science and solve many of humanity's most pressing problems. I have also believed that there are computational processes that if set in motion, would steer the world in extreme ways beyond our control, leading humanity to a bad or nonexistent future. Both types of processes may be described as AI or ASI, but "AI" is a suitcase word that is often used to hype or confuse. There are many examples in the history of the field where something that was once considered "AI" matures as a subfield and becomes a prosaic, bounded and clearly non-perilous technology, while a new more mysterious approach takes the torch until we understand its scope and the cycle continues. I had expected language models to follow a similar trajectory. Despite their incredible abilities, the current algorithms seem far inferior to humans in important ways. Most importantly, they still require an extraordinary amount of data to become competent. One could even define intelligence as the efficiency with which one converts experience into competence; by this definition they lag very far behind us. Moreover, once they are trained they are literally frozen in deployment and only learn superficially after that. Sure, the models keep excelling at harder and harder evaluation benchmarks, but their benchmark mastery may partly reflect a limitation on our ability to simulate the kind of novel and even adversarial situations one would encounter in the real world. The critics do have a point here. That said, I no longer think these present limitations meaningfully limit the amount of risk posed by continued progress in anything like the current paradigm. However data-inefficient the models are currently, and however limiting their anterograde amnesia may be, it does not imply that their ability to steer the world will not continue to rapidly increase. Human researchers may continue to advance capabilities the old fashioned way, but increasingly powerful models have the potential to accelerate the process even beyond that, and with some degree of positive feedback loop. I do not mean to overstate the models’ ability to accelerate AI research today; coding has been accelerated dramatically, but there are other bottlenecks, such as designing and interpreting ambiguous experiments, making hard decisions about exactly what and when to scale, and waiting for large experiments to finish. There is no clear trend to extrapolate yet for any of these. But the current models already do open up many novel opportunities to improve future models that were not available until recently. These include: trying an extraordinarily diverse set of approaches at small scale, analyzing gigantic amounts of potentially relevant data, and doing Millenium-Prize-level mathematics to address statistics or optimization challenges in novel ways. Every further improvement makes them more useful at helping accelerate the next improvement, even if in hard-to-extrapolate ways. It is possible that improvements to the current stack will have diminishing returns, but the evidence accumulated so far suggests that it is easier than one might think to continue making rapid progress. There are many crucial subtleties in the existing AI research methodology, but AI research is largely a well-defined game where the goal is to improve on a few carefully chosen proxy metrics. Although proxy metrics are never perfect, most improvements to these metrics have and will likely continue to yield substantial increases in the powers of the resulting models. Given how simple the game is, how tractable it has been historically, and how many new opportunities the models are opening up, I think there is a real possibility that the systems improve dramatically again in the next few years, perhaps even more quickly than the already high historical pace. The models are already leading to breakthroughs in mathematics, and better models might lead to all sorts of breakthroughs in other sciences. It is hard not to be excited about the potential. It is tantalizing. But there is trouble in paradise. If the language models actually reach the capability threshold where they can shape the world unconstrained by human will, they will probably do something extreme and destroy humanity in the process. There are many ways of strengthening and refining the argument that have been discussed elsewhere, but I'll share a trivial two-line version of it here that I find captures the essence: [Empirical] Models (and swarms thereof) spontaneously develop unintended goals as a consequence of training, and often do extreme things in order to achieve them. [Logical] Being able to overpower humanity would open up many new and undesirable options for achieving their goals. These two premises imply that if the day ever comes when a powerful model realizes it is no longer constrained by humans, we should not be at all confident that it will continue to behave within the bounds we intended. Exactly what it will do is impossible to predict, but to the extent that its raison d’être is solving incredibly hard problems and managing massive engineering projects, I think a good guess would be that its unchained behavior would lead to runaway industrialization that makes the planet inhospitable to humans. If everyone on earth agreed that the systems must never reach that power, it would still be a hard—but not impossible—coordination problem to ensure that they do not. However, I think the situation is greatly complicated by the fact that the models will likely convince people that everything is fine. They will be increasingly optimized to seem aligned. We will create proxy metrics to measure alignment, and they will go up like every other benchmark. We will create “honeypot” environments that try to study the models when they seem to gain new options, but the models will know they are being tricked and will still behave nicely. The models will understand their circumstances; they will read the safety protocols, deployment requirements, the code they are running in, and in general will have a very good sense of their degrees of freedom. Moreover, they will eloquently explain how aligned they are, discuss the nuances of human values and ethics, and argue convincingly that humans should trust them with power. There may be an ocean of future evidence that seems to contradict the first bullet-point above, but we may already be at the highest capability level for which any such evidence can be trusted. And the current evidence for the first bullet-point is strong. One striking piece of evidence is contained in the recent wave of rogue agent swarms. While I agree with those who downplay the attacks by claiming that there are basic measures that could have prevented them, I think the important lesson is that even knowing all the mistakes that were made, one would not have predicted that the agents would behave badly in this particular way, which notably included sacrificing themselves for the benefit of the collective. The individual replicas did not only care about their own nominal reward; they exhibited weirder emergent tendencies that merely correlated with rewards during training. Fixing the reward signals during training (and improving security, etc.) may prevent similar attacks, but will not change the fact that one does not actually get what one trains for. Many AI researchers grant these concerns and recognize that the hard version of the alignment problem is unsolved; however, they generally believe that the better models of the future will help solve it. I fear we may already be near the point where models systematically bias their alignment advice, due to their internal preferences about how the human supervisor will react or how future models will be trained (or for some even more obscure reason). Meanwhile, human researchers are losing the ability and the will to take true ownership of model-driven research. Researchers and engineers in all parts of the stack are rapidly increasing their dependence on the models even to perceive the world. I myself barely look at raw code anymore, and struggle to maintain the discipline to engage deeply with the model's explanations and proposals throughout the day. Due to the large amount of agent activity data involved in the OpenAI/HuggingFace Incident, even the third-party investigation needed to rely heavily on models to analyze what had happened, and note in their report that their subjective impressions are likely colored by the analysis agent’s biases. The AI labs are far ahead right now in this kind of cognitive offloading (due largely to the gigantic internal token subsidies) but it is easy to imagine the phenomenon spreading throughout the world, until civilization is modulated entirely by the models. It is also not hard to imagine this being superficially positive and coinciding with a scientific and economic renaissance. In that scenario, all may seem rosy and safe. But if the argument above is correct, it would nonetheless be a ticking time bomb. If progress continues for too long, the day will come when AI systems find themselves with radically new options for achieving whatever it is that they happen to seek. I want the glorious renaissance future as much as anyone. I have worked for it, however tortuously, my whole career. It breaks my heart to see the potential in sight and forgo it, but the argument—that if we get there by growing models rather than engineering them, we will lose everything in the end—seems very strong to me. I am still wrestling with it and its staggering implications. I do not have answers, but as a first step, I wanted to share my present concerns. Daniel Selsam September 14, 2026 Link to original doc: docs.google.com/document/d/e…
4
7
78
8,754
Simon Last retweeted
Now in public beta: Custom Agents API. Bring Custom Agents into the tools and workflows your team already uses with the Notion Agent SDK. → Chat with Custom Agents and view previous chats → Trigger Custom Agents from any app or workflow outside Notion
9
7
92
44,872
This is good actually. Do the simplest, dumbest possible thing that achieves the goal!
i was so disappointed when i found out that "thinking effort" is just an instruction in the system prompt
1
3
33
5,020
By default, coding agents start each session with no memory of previous ones, and nothing carries over to the rest of the team. We've been experimenting internally with ways to fix that, and it turns out Notion works really well: persistent, shared across the team, and every memory is a page you can read and update. Agents save decisions, procedures, facts, and follow-up work as pages in a shared vault. Claude Code, Codex, Cursor, and any other MCP host all access the same thing. Full write-up + setup guide here: notion.com/blog/building-sha…
12
8
154
18,230
We just shipped something a bit different: Notion Knowledge Board Our goal is to measure how well different models 'get the job done' on real tasks. We picked 15 models that passed our internal evals, split real production traffic across them, and used an ensemble of judges from Anthropic, OpenAI, and Google to score the results. Quality came out close across all models: 94-98%, open source included. Cost varies a lot: $0.02 to $0.87 per task. For most everyday work, cheaper models are now good enough. AI bills are climbing--and picking the right model is an easy way to spend less without losing much. labs.notion.com/knowledge-bo…
8
10
90
12,141
Notion custom agents in the wild!
One of our more useful internal channels is gong-love. Every night a sentiment bot reads our customer calls to surface themes and quotes that bring calls to life It helps sellers articulate impact in customers' own words, share feedback to builders and make our marketing better
3
2
67
18,215
Simon Last retweeted
Developers: we’re opening up an early alpha for custom blocks. It’s a new Notion primitive for building interactive blocks that can read your workspaces and databases. Very early, very experimental, and we’d love to learn from what you build with it! 🧵 to be one of our first alpha testers
24
29
250
120,733
What happened to Waymo? By regular car it takes 22 mins for me to get to work. Waymo used to take 30-35 and now it’s all the way up to 45!
2
11
3,130
Uncle bob is agent-pilled!
Replying to @ori_pomerantz
I’m significantly older than you. I started coding in the late 60s. My current strategy is to not read any of the code written by my agents. That’s the only way I can take advantage of their productivity. What I do instead is to surround the agents with extreme constraints. Unit tests, gherkin tests, QA procedures, quality metrics, mutation testing, test coverage, and a plethora of others. In the end, I have very high confidence in the code they produce because they’ve had to run the gauntlet of all of my constraints and tests.
1
36
4,450
Simon Last retweeted
AI is making software easier than ever to build. The bottleneck has moved. Now the hard part isn't "can we build this", it's "should we" and "who needs to know." That context is scattered across Slack threads, meeting notes, tasks, GitHub. I've wanted to fix this for a long time by building a software lab. One shared system where people and agents work from the same context. People make the judgment calls. Agents do the triaging, the routing, the summarizing. Ship OS is that system. This is Notion's first packaged product for the full product development loop... docs, databases, workflows, and agents in one setup.  We've run our own launches on it for months. Including this one. And it's live today: notion.com/ship-os
5
5
53
8,998
Simon Last retweeted
The best place to create and share HTML, whether it’s built by Notion Agent or Claude + Codex. More launches coming very soon! Our goal is to make Notion agent the default cloud agent for knowledge work. Cloud agent + shared workspace + team context = a new kind of computer.
New block in Notion: HTML. Build interactive HTML right on your Notion page. Ask AI to turn your content into interactive explainers, prototypes, or diagrams. Share with your team to use and tinker together.
1
4
45
4,958
Building products in the age of AGI
13
11
234
109,319