this is how i wound up with a $172 couch blanket. there is household consensus that it is ugly but (1) it is too expensive to throw and (2) it is good to keep persistent humbling reminders of one’s own poor decisionmaking

counterpoint: if your boyfriend wont shill your startup on twitter then you should dump him

wrapping up a call and he hits me with “have a good weekend” at 12:44 on a wednesday you too, man. you too

I've been thinking a lot about AI disintermediating human relationships. This seems like a real challenge to me, and I'm not sure how to feel about it Thanks to AI, in the long run there will be no need for me to ask anyone else for help with anything, and consequently, there will be no opportunity for people to feel useful and appreciated when they provide a service. That give-and-take between people is the core of our social fabric. (It has already been fraying for a long time, even pre-AI.) Many optimists say things like "AI will automate all the rote work so people can focus on their passions and being together" -- but there is no "being together" without some shared struggle or objective. There is no building trust without need -- and in an AI-induced age of abundance, there may just not be any need. Social fabric is woven by countless daily small, necessary interactions with other people -- and those interactions may simply go away because your AI agent of choice ends up being your interface to the world around yourself. Not because of any shortcoming, like being lazy or complacent, but because AI will be a better interface. Why ask your friend for advice when AI will be more patient, less judgmental, and understand you better? What's left for parents to teach their children if AI is a better tutor in all respects? And so forth. Many incipient changes or risks due to AI do not worry me much. I am very optimistic about what AI can bring to our future. But this one challenge bothers me; it seems like it could spell a fundamental reorganization of our society, isolating the individual much more greatly than ever before. The consequences of this are not clear to me, but it doesn't look great -- generally, it seems that "isolating the individual" has not been good for either those individuals or their societies. And I don't think enough time or thought is being spent on what we want our social lives post-AGI to look like.

- Germanic, ‘earn/deserve’ money: a moral weight from working - American, ‘make’ money: new wealth is created, not a fixed pie - Latin: ‘win’ money: you gain by taking it from others, zero-sum fixed pie No comment is even needed on this one

me: wow this cool poster is only $50 on ebay. this will yield joy in a cost-efficient manner it arrives: hmm it would be even better if it were not stuck to the wall but framed me at the frame store: what? you’re telling me — this is how much? framer: it is our most modestly priced entry-level frame me: this is more than ten times the price of the poster framer: you have selected the most affordable of our custom framing options => lesson: buying art industrially pre-framed is much, much, much cheaper than paying for the cost of craftsman labor in california. which is understandable

A Question about Superintelligence and RSI So far, we've had good milestones for scaling intelligence by our own experience: we've gone from SOTA models struggling with arithmetic to now solving Millennium Prize problems. But these are the hardest problems we have: now what? What is the next set of reasoning problems that we can use as milestones for even greater intelligence? You could say that that's the wrong way to go about it. Maybe we optimize energy-to-intelligence, e.g. for future SOTA models, how many joules does it take them to prove Navier-Stokes? But improving efficiency is not at all the same as improving capability. Or you could say that we as humans don't have to come up with future milestones, but AI will: like training chess engines by adversarially having them play against one another. But I don't think that's coherent, because I can't think of an adversarial game that gets you closer to general intelligence -- only even greater "spikiness" in model performance. For example, trying to improve models by having them play chess against one another is really only going to make them better at chess, probably to the detriment of other capabilities. Adversarial training carries a risk of overfitting that seems hard to mitigate. If we're faced with this question of how exactly we (a) evaluate or (b) guide toward intelligence beyond what we can recognize, then it also raises the same question for Recursive Self-Improvement. Because Self-Improvement by definition requires a metric to optimize, or a ladder of progressively harder challenges to solve. These were available as long as the models were less intelligent than humans, but now we're running out. The space of problems that were formulatable by or accessible to humans will be rapidly saturated by model capabilities. That's not to suggest that RSI isn't possible, or anything like that. But to me there are real questions around (1) how we evaluate improvement going forward and (2) what the impact of this will be on speed of improvement. For example, it could be possible that we get RSI, but it ends up being much slower than progress to-date, because for every step forward in general improvement, the models first have to discover a new space of harder constituent problems, and those end up becoming sparse.

i bought one of these. it’s the best $28 i’ve spent this year. some might say i’m having too much fun

This evening I went to Envelop SF. It’s a listening space for ~15 people, with 32 big speakers all around you. It was so cool to experience a great album immersively like this, in the highest possible quality, with the sound moving all around you. If you like music, check it out!

Given the raw level of interest in intelligence <> compute <> electricity <> energy, I’m surprised that Vaclav Smil’s work isn’t cropping up more

I was 8 years old on 9/11. Old enough to remember, not old enough to get it. And I think many people in my generation struggled to get it because we were children, and almost everyone who died on 9/11 was an adult. But every year, as I got older and the stories remained the same, the deeper they hit. I read Todd Beamer’s story again. When I first read it, I must’ve been a child — and of course it’s hard to imagine yourself on flight 93, but as a 10-year-old, how do you relate to a grown man — father, athlete, employee, business traveler, and in the end, hero? When I was reading the story today, I came across the photo that previously had looked so old to me, and I thought “he looks pretty young”. He was 32 that day, a few months shy of his 33rd birthday — younger than me today. His whole life was still ahead of him. As so many years have now passed, it dawned on me that so many of the victims then were younger than I now; just starting out in their 20s. I’d see them as kids today. Robbed of life, youth, and the great journey ahead. Many others are writing that the 25th anniversary is hitting them unusually hard. Part of it is that with a longer arc of hindsight, it is becoming clearer and clearer what we lost. As for me, age adds perspective.

25 years ago, america mourned together. a long, confusing process. but we were a country. if it happened today the streamers would be blaming trump and the jews by this afternoon, and leftoids on tiktok would be saying the bankers deserved it. we’ve lost more than we realize.

The people most suited to carrying the greatest responsibilities tend to be the most afraid of them. And those most eager to take them on tend to be the least suited. Some people in AI are scared of the responsibility and are looking to delegate it away. But there are responsibilities you cannot delegate: there will be a rude awakening if you hand the keys to administrators with neither respect nor knowledge of the field. Imagine Thierry Breton in charge of our frontier labs.

Pandora's Box is Open You could try to pause, pace, or regulate AI. But the more parties cooperate, the greater the reward to the party that defects. This is the key, unavoidable problem. If you try to pause because you believe AI is so powerful, so dangerous, then of course the thought will cross your mind that the best strategy might be for you to persuade everyone else to pause, but for you to carry on in secret. Or for you to position yourself such that you come out on top after the pause. And this isn't even about material greed, you may have what feels like a good moral justification: better have you in charge -- you know that your intentions are good -- than anyone else, whose intentions are less known to you and therefore more dangerous. (But everyone thinks that their own intentions are good. Yes, even the villains.) And I worry greatly about the "middle ground" outcome, where AI is regulated in a wave of fear, and the people who end up in charge are the worst people -- maybe things even look good at the start, you put wise and well-intentioned people in charge, but you wind up with the worst people, who are simply better at politics and power games. The history of any number of organizations or governments shows you how quickly this can degrade. Whenever I hear of an x-risk scenario, I have practical questions. But I can see real and straightforward paths for us to wind up with all the downsides and none of the benefits of AI, simple failure modes where we wind up with AI-empowered despots. And I think that this failure mode -- someone monopolizing all-powerful AI to put a boot on the face of everyone else, forever -- is very bad, and underrated relative to its likelihood. I don't deny that there is some risk posed by AI progress. But all the suggestions I'm seeing don't seem to actually mitigate this risk, but rather *shift it* -- "pausing" simply moves the arena in which the power struggle occurs, while "pacing" creates the incentive for a defector to sneakily try to outrun everyone else. And obviously the defector is the person that you should be most worried about using their powers for evil! Do not accidentally incentivize and empower your adversaries. For better or for worse, we are locked into a progress race. Pandora's Box is open and there is no closing it. But at the bottom of Pandora's Box lays hope: my hope is that as in so many other domains, the winner of the race is also the most deserving; the one with the deepest expertise, best-suited to govern this power. And I hope that by having many parties competing in this race, new stable equilibriums emerge -- just as for the entire history of humanity, even though our destructive capabilities have only increased, balance of power has emerged alongside those capabilities at every step of the way, and so -- counter-intuitively! -- the world has only become a safer and less violent place.

Parents tend to raise their children optimal for the prior generation, and founders tend to build businesses optimal for the prior cycle I just got off the phone with a friend considering a job at an AI insurance brokerage. The pitch? They’re going to use AI to better enable salespeople to sell to clients. And it’s like — okay sure you’ll make some money for a little bit, but take a step back. You think the old sales model of people dialing on the phone, trying to take main street business owners out for steak & golf, and clipping ~15% commission is going to work? For how long exactly? This is a business model of the past. And sure you can optimize it but obviously this whole model will disappear not long thereafter. These people pick the safe business models of 2019, imagining that with AI this time around they can really kill it. But the puck has already moved a long ways away. We solved a Millennium Prize problem this morning and yet so many businesses I see are frighteningly unambitious, as if built by people who have not internalized the exponential and instead assume AI gets twice as good from here and then tops out. But no! Orders of magnitude are what matter. Imagine the unimaginable. I want to see startups tackling immortality, lunar colonies, whatever. Think of the biggest thing you can because this cycle is going to whisk us right to the heights at which these problems become tractable.