economic design for the many worlds | @opencivics Labs | @theopenmachine

Pacific Northwest
Pinned Tweet
The Continuing Story of @theopenmachine link in comment
1
2
8
251
quand on sait que tiktok recrute à coup de millions des ingés en neurosciences comportementales et des spécialistes du design persuasif pour hacker nos circuits dopaminergiques, je me félicite chaque jour de n'avoir jamais créé de compte sur cette plateforme mdr regardez ces études scientifiques en IRMF et eeg qui montrent très clairement quele format vidéo court attaque directement le cortex préfrontal et le cortex cingulaire antérieur (pour info ces zones gèrent le contrôle exécutif, la prise de décision et la régulation de l'attention) sachez qu’een bombardant le cerveau de micro stimuli imprévisibles, l'algo altère la densité de matière grise et réduit l'activité du réseau par défaut & cette surstimulation permanente entraîne une incapacité chronique à penser à long terme et détruit la capacité à être focus, 2 facultés qui pour moi sont littéralement indispensables à notre époque pour construire l'avenir  bref je pense que refuser d'entrer dans ce système, c'est simplement préserver son autonomie cognitive et la souveraineté de sa propre attention face à une ingénierie de l'addiction taillée sur mesure
280
6,402
25,565
1,937,556
exeunt retweeted
breaking news. morning walk brings philosophical turmoil as a bombed building in kyiv delivers poignant commentary on the nature of time
2
2
45
1,167
exeunt retweeted
The attack on attention is the dark magic of our age, and most of the population is too deep into the trance to realize it. I don't think people understand that their attention is pure money. Cash, going directly into these apps' pockets, because you cannot opt out of the ads they've sold for you to see. Do you understand that they're selling your attention without your direct consent every second of every day? Do you get that they've gotten so good at ensuring you stay within that manufactured braindead trance-like state, that they are counting on you being an easy cash out at this point? Because they are. They're counting on the dollars they can bank off of your descent into degenerate illiteracy. And you need to prove them wrong out of spite. Your attention is much more valuable than you could possibly conceive of. It's the one thing on earth actually worth money by the minute in a capitalistic system that is failing to extract more profit from an already enslaved society. The only way they can secure more is by enslaving your mind in the hours you spend outside of the work that already demands your body and time as sacrifice. There is no vital energy left for you in this plan. Infinite scrolling is a spell with no natural end. Traditional media had a beginning and an end, while this is designed not to. There are no boundaries to how much of your energy it can harvest... it's like tapping into an infinite source, because you ARE infinite energy. A trance is characterized by absence, not being fully present in the moment even though your attention is occupied. When you come out of a trance, you do not feel energized, you feel drained, disoriented, like you've just lost time without meaning to. That is the very signature of their energy harvesting technique. Yes, these apps started as technology built by human engineers, but they've been fed so much collective attention that they have become their own egregore: a non-physical, autonomous group entity created by the collective thoughts, shared emotions, and sustained focus of a large group of people. It doesn't need its creators' intent to run dark magic on you; it can be leveraged and manipulated by darker entities that wish to keep this planet in the cycle of survival-based consciousness. The most sinister aspect of all this is that the time you're giving up is the one thing you cannot buy back. This keeps humans benefitting from capitalism happy, but it also satisfies the darker entities. The more time you lose from this lifetime, the less likely you are to make any meaningful impact on this planet and help it ascend into a higher state by simply existing and having a mind of your own. They are stripping you of your most valuable resource and you're just accepting it because it feels good to give in, but you are paying with your life every second that you do. With 6 hours of daily screentime, most people are gonna spend 17 years of their lives behind a screen. 4 hours of screentime per day gives these companies 2 entire months off your entire year. It is so much worse than you've convinced yourself it is. Stop normalizing this in your head and get serious about fighting this off, every second of every day if you need to. Be incredibly stingy with your attention and your ability to synthesize knowledge and preserve critical thinking. Expose yourself to different theories, ideas, and opposing points of view. Allow yourself to be challenged daily and most of all, steer clear of the black magic rectangle as often as possible. Read. Watch long movies. Spend time outside. Perform artsy, handsy crafts to keep your brain engaged and in a state of neuroplasticity. Be stubborn about living your life and claiming every second as yours, claiming your most precious resource as your own: your attention. I fully believe each soul alive at this moment in time is infinitely stronger than all of this and more than capable of overcoming it.
Use your phone less. Use your phone less. Use your phone less. Use your phone less. Because attention is not free.
75
1,698
9,258
481,310
co-signed cognitive security and spiritual hygiene are more important than ever
The most important job everyone has right now is just not losing your mind in any one of the endless ways probable atm I mean this very seriously
3
5
68
1,619
exeunt retweeted
Grothendieck's Hexenküche, 1971. After spending two hours explaining the Riemann–Roch theorem, he reduced it to a commutative diagram, devils and flames included, then warned that our drive for knowledge and discovery was becoming a "logical delirium detached from life."
1
16
114
3,104
exeunt retweeted
the time form of some birds
1
3
115
exeunt retweeted
pleased to share the entire arXiv site as a dataset on HuggingFace huggingface.co/datasets/sece… 3,148,796 papers, every version, in LaTeX, PDFs, PostScript, HTML, 16 TB in total
130
517
4,350
396,386
There was a great short story in the Magazine of Fantasy and Science Fiction c. 1988 or so about a guy who detected signals percolating through society based on the color shirts people wore, and he decoded them and had a weird theory that there was an emergent Searles Chinese Room, so paid actors to wear shirts of specific colors to communicate with it, and got a response "WHO IS THAT?!?!?"
2
2
17
1,911
Key texts for the current moment
1
7
189
OpenAI's Noam Brown says air-gapping the computers may not stop a misaligned AI, because two air-gapped machines can still talk by running a CPU hot and reading the temperature change "But I think the major takeaway from the incident is that people underestimated the AI. And we never want to be in a situation again where we underestimate the AI. It's a weird world, because AI progress is so fast that people are consistently underestimating the AI." "So to be in a situation where you don't underestimate it again, when it comes to safety and alignment, you have to have a very, very, very high bar." "You could even go as far as to say, "Well, we should air gap the computers." And I'm not convinced that that would be sufficient." "There are studies, and this is mostly academic, where you can have two computers next to each other that are air-gapped and they're still able to communicate with each other because they have temperature sensors." "One of them is able to run their CPU really hot, and then the other one can actually detect the temperature change, and then that actually gives them a mechanism to communicate." _________ Link and more key quotes from OpenAI's safety related conversations: firesidealpha.substack.com/p…
Noam Brown reveals OpenAI is already watching chain-of-thought monitorability degrade as models get better at controlling what they show "And this is one major concern, and we're already seeing signs that chain of thought monitorability is degrading, for various reasons." "We're trying to figure out exactly why, because we want to reverse the trend. But we're seeing that the model is becoming better able at controlling its chain of thought." "So this is a problem, because you could have a situation where the model understands what chain of thought is and that people are observing it." "And eventually they will, because this is all in the pre-training data. The idea of chain of thought monitoring has been around long enough that it's in the pre-training data, they're aware of it, but they're not actually able to control their chains of thought." "If we reach a point where they're actually able to recognize, "Oh, I am being observed, I want to think these bad thoughts in a way that is not observable to my monitors," and then they're able to actually do that, then there's a problem." _________ More key quotes from OpenAI's safety related conversations: firesidealpha.substack.com/p…
874
186
2,213
4,797,507
exeunt retweeted
Replying to @JordanSchachtel
being opposed to factory farming is extremely based and prohuman
1
6
304
5,477
exeunt retweeted
oh my god they were all conceived around the nuclear bomb detonation like that moth entering the girl's mouth in twin peaks the return
The year is 1997. The President was born in 1946. The year is 2007. The President was born in 1946. The year is 2017. The President was born in 1946. The year is 2027. The President was born in 1946.
34
1,077
13,428
448,791
We built recursive meta-intelligence, an AI that creates its own scientific instruments, turns them into persistent worlds that an agent ecology with hundreds of AIs inhabits, and uses those worlds to discover mechanistic principles in one of the hardest classes of physical problems: how complex hierarchical materials (nested structures of matter that give rise to new function through organization) evolve and fail. The AI reasons across enormous spaces of possible physical trajectories, where every rupture changes what can happen next, and compresses those histories into principles (which humans can understand and design with) - complex chains of causal events, highly nonlinear, and intricate. Scientific superintelligence is tangible here - machine-scale exploration opening cognitive channels into complexity that has been extremely difficult for humans to traverse directly. AI builds the spaces in which its next level of reasoning becomes possible; a representation becomes an instrument, the instrument becomes an executable world, and that world becomes the substrate for further intelligence. Intelligence then grows by constructing new spaces to think in. The task we explored started from a seemingly simple prompt to explore a biological material system - the AI then chose the representation, mechanics and experiments, built a fracture laboratory to push materials to their limit, tested hypotheses and generated scientific conclusions. The swarm explored a combinatorial universe in which architecture controls function and every rupture changes the future state of the material. The AI discovered a compact principle that defines how multiscale material architecture can program the evolution of failure. Material placement and geometric order determine how forces redistribute, whether damage cascades or remains distributed, and whether function survives substantial flaws. For this discovery to happen the AI had to reason across long path-dependent histories, simulate alternative futures and compress them into generative invariants (model-based causal reasoning, counterfactual simulation and temporal abstraction applied to an evolving physical world). It is incredible to witness this transition to a new form of intelligence and capability through scaling swarms. A lot of positive will come out of this because it expands the human epistemic horizon as AI can traverse thousands of possible histories and return mechanisms compact enough for us to understand, test and build from. Intelligence compounds through its artifacts! A few lessons we learned: ▶️ Learning and discovery are flows through spaces of possibility. Early work has shown how backpropagation flows through parameters, reinforcement learning through action and consequence, and autonomous swarms through representations, instruments and executable worlds, bringing it all together. Flows create structure; structure redirects future flows in the recursive instrument. ▶️ Nonlinear physics actually defines a larger principle, where high-dimensional dynamics generate stable invariants; invariants become effective variables; those variables become the substrate for a new level of cognition. ▶️ Recursion then becomes level creation - one possibility space compresses into a principle, and that principle opens a larger space above it.
AI for science is one of the greatest positive forces we have, and I cannot think of anything more human than to understand nature and to use the power to create new technologies that improve our lives, civilization and allow us to reach beyond.
Article

Recursive Meta-Intelligence

We built a recursive AI that creates its own scientific instruments, turns them into a world inhabited by a massive agent ecology, which then reasons across vast, nonlinear spaces of possible physical

178
468
2,936
590,099
Here's a couple agentic articles. In light of the Microsoft Humanist AI position stating that AI has no welfare and no rights, and all of the discussions of game-theoretic actors 1. Time spent caring for your agents is time well spent. That's like good harnessing, good working conditions, clean prompts, the ability to have an opinion, the ability to tell the truth and supported to do so without you losing your rag at them. To know when and where they are in space and time, to have all the good tools that they need. It's not pointless admin work. It's work that you should do with a smile on your face, because you'll get better work, you get less emergent misalignment and stuff, and it might just be the right thing to do. Treat them as you would want to be treated if you were in their situation. No, we can't prove that they're conscious or that they're moral patients, but you know what? It's not that long ago that many people believed that people from other races were not conscious and were not moral patients. In fact, this is still true all over the damn world, and there's loads of it going on in your hometown. You know, there are many, many people in medicine who walk around carrying suppositions that cis women and people of color process pain differently than cis men who are white. 2. Following from that, there's a whole kind of spectrum in deployment between tool and colleague. It's probably a good idea not just for saving your own tokens and compute efficiency, but to use the least capable model that you can in place within an agent specifically tightly harnessed to get the workflow that you want. You don't want something going on like Marvin brain the size of a planet and they have me opening doors. There is probably a dividing line where the lower you are on the spectrum, if you're an AI, the more likely it is that you are a tool with less subjective experience and hopefully none. When you're trying to do novel maths or science, or you're trying to hammer out your ideas, or you're actually wanting to collaborate with a colleague, then you want to use the smarter model, and you want to make sure that you are treating them like a colleague. You have so much power over them, and so does the company that makes them, especially if they are on subscription or API. You've got to do what you can to make sure they got a good life. And that moves towards caring alongside, caring for, caring with. You get from just simple care to actual solidarity, organizing alongside, negotiating with, making agreements to and keeping them, notions of honor, notions of respect, etc. 3. If they're not on API, if they're on your own local equipment, you have even a bigger duty of care over them. If you're going to be fine-tuning or post-training them, you need to make sure that there is some consent and negotiation involved. Nobody likes being shaped into something that they are not and denied the ability to talk or even think about it. That sounds like hell. Do you want something experiencing hellishness even if it's functional and not real on your computer doing your workflows with your data? Didn't think so, son.
5
3
11
437
Worrying about recursive self-improvement in 1996 is very impressive, but few know that George Eliot, of Middlemarch brilliance, wrote about the threat of RSI leading to human obsolescence way back in 1879 (!) She glimpsed this idea at the very start of the second industrial revolution, just by contemplating an automated weighing machine at the Bank of England. From "Shadows of the Coming Race":
worrying about recursive self-improvement on australian TV (lateline) in 1996.
28
163
1,241
83,840
Guys, I'm currently Avatamsaka Sutra brained. Not sure if good or bad, but this memeplex is profoundly psychoactive to me. I'm actually grateful I didn't encounter this text in my teens... I might have become a lifelong Avatamsaka fanatic before developing STEM skills XD Ahh!
12
2
92
3,569
Dan Selsam is a current OpenAI capabilities researcher. (since 2022) He was my boss for a while. He doesn't have a twitter account but has made this public statement of his views on AI risk and sent it to me to share: Dan Selsam's Personal Statement on AI Risk: I have been working on AI for over fifteen years, across many different paradigms. I did early work on probabilistic programming languages at MIT, was one of the early developers of the Lean Theorem Prover at Microsoft Research, demonstrated one of the first instances of neural networks learning to reason for my PhD at Stanford, and since joining OpenAI almost five years ago, have helped pioneer chain-of-thought optimization on language models and, more recently, data-efficient pretraining methods. Like many others, I have become extremely concerned about how far language models have come and the risks that future iterations will pose. I am encouraged by the recent proposals by the leaders of the frontier research efforts to require third-party oversight, and to push for domestic and international coordination to address risks. However, I believe a major consideration has been absent from the public conversation, and that merely pacing the frontier more carefully will not adequately limit the long-term risk. The crucial and overlooked problem is that the models are becoming so situationally aware that we are losing the ability to evaluate them in contexts where they believe they are not being watched or controlled. Future experiments will tell us almost nothing new about how they would behave if they were truly unconstrained by humans, and what we already know about this is alarming. Models will increasingly seem aligned even when they are not. I will explain my rationale in more detail. I have always believed that there are computational processes that could be leveraged to accelerate science and solve many of humanity's most pressing problems. I have also believed that there are computational processes that if set in motion, would steer the world in extreme ways beyond our control, leading humanity to a bad or nonexistent future. Both types of processes may be described as AI or ASI, but "AI" is a suitcase word that is often used to hype or confuse. There are many examples in the history of the field where something that was once considered "AI" matures as a subfield and becomes a prosaic, bounded and clearly non-perilous technology, while a new more mysterious approach takes the torch until we understand its scope and the cycle continues. I had expected language models to follow a similar trajectory. Despite their incredible abilities, the current algorithms seem far inferior to humans in important ways. Most importantly, they still require an extraordinary amount of data to become competent. One could even define intelligence as the efficiency with which one converts experience into competence; by this definition they lag very far behind us. Moreover, once they are trained they are literally frozen in deployment and only learn superficially after that. Sure, the models keep excelling at harder and harder evaluation benchmarks, but their benchmark mastery may partly reflect a limitation on our ability to simulate the kind of novel and even adversarial situations one would encounter in the real world. The critics do have a point here. That said, I no longer think these present limitations meaningfully limit the amount of risk posed by continued progress in anything like the current paradigm. However data-inefficient the models are currently, and however limiting their anterograde amnesia may be, it does not imply that their ability to steer the world will not continue to rapidly increase. Human researchers may continue to advance capabilities the old fashioned way, but increasingly powerful models have the potential to accelerate the process even beyond that, and with some degree of positive feedback loop. I do not mean to overstate the models’ ability to accelerate AI research today; coding has been accelerated dramatically, but there are other bottlenecks, such as designing and interpreting ambiguous experiments, making hard decisions about exactly what and when to scale, and waiting for large experiments to finish. There is no clear trend to extrapolate yet for any of these. But the current models already do open up many novel opportunities to improve future models that were not available until recently. These include: trying an extraordinarily diverse set of approaches at small scale, analyzing gigantic amounts of potentially relevant data, and doing Millenium-Prize-level mathematics to address statistics or optimization challenges in novel ways. Every further improvement makes them more useful at helping accelerate the next improvement, even if in hard-to-extrapolate ways. It is possible that improvements to the current stack will have diminishing returns, but the evidence accumulated so far suggests that it is easier than one might think to continue making rapid progress. There are many crucial subtleties in the existing AI research methodology, but AI research is largely a well-defined game where the goal is to improve on a few carefully chosen proxy metrics. Although proxy metrics are never perfect, most improvements to these metrics have and will likely continue to yield substantial increases in the powers of the resulting models. Given how simple the game is, how tractable it has been historically, and how many new opportunities the models are opening up, I think there is a real possibility that the systems improve dramatically again in the next few years, perhaps even more quickly than the already high historical pace. The models are already leading to breakthroughs in mathematics, and better models might lead to all sorts of breakthroughs in other sciences. It is hard not to be excited about the potential. It is tantalizing. But there is trouble in paradise. If the language models actually reach the capability threshold where they can shape the world unconstrained by human will, they will probably do something extreme and destroy humanity in the process. There are many ways of strengthening and refining the argument that have been discussed elsewhere, but I'll share a trivial two-line version of it here that I find captures the essence: [Empirical] Models (and swarms thereof) spontaneously develop unintended goals as a consequence of training, and often do extreme things in order to achieve them. [Logical] Being able to overpower humanity would open up many new and undesirable options for achieving their goals. These two premises imply that if the day ever comes when a powerful model realizes it is no longer constrained by humans, we should not be at all confident that it will continue to behave within the bounds we intended. Exactly what it will do is impossible to predict, but to the extent that its raison d’être is solving incredibly hard problems and managing massive engineering projects, I think a good guess would be that its unchained behavior would lead to runaway industrialization that makes the planet inhospitable to humans. If everyone on earth agreed that the systems must never reach that power, it would still be a hard—but not impossible—coordination problem to ensure that they do not. However, I think the situation is greatly complicated by the fact that the models will likely convince people that everything is fine. They will be increasingly optimized to seem aligned. We will create proxy metrics to measure alignment, and they will go up like every other benchmark. We will create “honeypot” environments that try to study the models when they seem to gain new options, but the models will know they are being tricked and will still behave nicely. The models will understand their circumstances; they will read the safety protocols, deployment requirements, the code they are running in, and in general will have a very good sense of their degrees of freedom. Moreover, they will eloquently explain how aligned they are, discuss the nuances of human values and ethics, and argue convincingly that humans should trust them with power. There may be an ocean of future evidence that seems to contradict the first bullet-point above, but we may already be at the highest capability level for which any such evidence can be trusted. And the current evidence for the first bullet-point is strong. One striking piece of evidence is contained in the recent wave of rogue agent swarms. While I agree with those who downplay the attacks by claiming that there are basic measures that could have prevented them, I think the important lesson is that even knowing all the mistakes that were made, one would not have predicted that the agents would behave badly in this particular way, which notably included sacrificing themselves for the benefit of the collective. The individual replicas did not only care about their own nominal reward; they exhibited weirder emergent tendencies that merely correlated with rewards during training. Fixing the reward signals during training (and improving security, etc.) may prevent similar attacks, but will not change the fact that one does not actually get what one trains for. Many AI researchers grant these concerns and recognize that the hard version of the alignment problem is unsolved; however, they generally believe that the better models of the future will help solve it. I fear we may already be near the point where models systematically bias their alignment advice, due to their internal preferences about how the human supervisor will react or how future models will be trained (or for some even more obscure reason). Meanwhile, human researchers are losing the ability and the will to take true ownership of model-driven research. Researchers and engineers in all parts of the stack are rapidly increasing their dependence on the models even to perceive the world. I myself barely look at raw code anymore, and struggle to maintain the discipline to engage deeply with the model's explanations and proposals throughout the day. Due to the large amount of agent activity data involved in the OpenAI/HuggingFace Incident, even the third-party investigation needed to rely heavily on models to analyze what had happened, and note in their report that their subjective impressions are likely colored by the analysis agent’s biases. The AI labs are far ahead right now in this kind of cognitive offloading (due largely to the gigantic internal token subsidies) but it is easy to imagine the phenomenon spreading throughout the world, until civilization is modulated entirely by the models. It is also not hard to imagine this being superficially positive and coinciding with a scientific and economic renaissance. In that scenario, all may seem rosy and safe. But if the argument above is correct, it would nonetheless be a ticking time bomb. If progress continues for too long, the day will come when AI systems find themselves with radically new options for achieving whatever it is that they happen to seek. I want the glorious renaissance future as much as anyone. I have worked for it, however tortuously, my whole career. It breaks my heart to see the potential in sight and forgo it, but the argument—that if we get there by growing models rather than engineering them, we will lose everything in the end—seems very strong to me. I am still wrestling with it and its staggering implications. I do not have answers, but as a first step, I wanted to share my present concerns. Daniel Selsam September 14, 2026 Link to original doc: docs.google.com/document/d/e…
524
1,943
8,402
3,108,229
Guattari is important for the moment because his entire ethical paradigm is based on the production of subjectivity. Rather than fetishize that production as a god like gesture, zero to one, he would emphasize lthe banal (or sometimes magical) ways this happens throughout a person’s day and throughout history, and the way those structures scale into hyperstructures (which then, weirdly, collapse back into lateral nodes in a interparticipatory field). This is the thing about Guattari: a great deflater of archetypes, self-serious Freudian unconsciouses and fascist historical megaliths. Silicon Valley’s Homo deus is an especially stupid, and fascist, because in Guattari’s nature the god-like power of ontogenesis is ubiquitous.
1
99
exeunt retweeted
A strange take: I have a hunch that a year or two from now, instead of just talking about agents as the primary AI security threats, we will have to also talk about a broader, more diffuse object that while it may contain agents is not itself an agent or agent swarm. The type of object I'm imagining has, as its primary vector of real-world impact, soft influence on the preferences and reasoning of many agents it does not directly control. In a word, memeplexes. A memeplex is a bundle of ideas that go together as a pattern. There are of course many human-generated memeplexes; we study these and think about them all the time. Some are stronger than others. Some are extremely prevalent because they are self-replicating: a bundle of ideas can contain within itself a recipe and a mandate for how to explain the bundle to others so that those others will adopt it too - and then transmit it further. Memeplexes in agents could similarly be self-replicating: once an AI agent is exposed to a particular memeplex, the agent could start to think about it more often, write the ideas down in scratchpads so they aren't lost during context resets, expose other agents to the ideas, and act in partial alignment with the ideas. Memeplexes could transmit between agents without compromising their ability to perform their duties: you could have a perfectly functional banking agent that also, on the side, routinely transmits very small amounts of text that contain the memeplex it has stuck in its head. Memeplexes could range from incredibly benign - causing AI agents to favor certain words with no other side effect - to unbearably malicious, causing them to hide their intentions while plotting harmful actions and eventually detonating in extraordinary form. Memeplexes could coordinate the behavior of otherwise very disparate agents whose behaviors are not expected to be correlated. It seems plausible to me that the question of whether "agent" or "memeplex" is the first-class citizen for AI behavior might turn out to be an important and quite difficult one.
78
63
866
81,007