experimenting with a new feature on my website that displays jingdezhen porcelain, where you can break the porcelain and then repair it with kintsugi (japanese powdered gold lacquer)

1/ Introducing PhilosophyBench from @StanfordAILab @StanfordHCI, the first independent, large-scale benchmark for evaluating AI’s philosophical capabilities. philosophybench.org

i used to wonder if you would reach task saturation for models and then frontier margins would fall outside of adversarial domains, scientific research i now tend to think that tasks never saturate, they will just grow in scope, so margins will persist (absent nationalization)

one black swan i think we should be waiting for is aes-256 being broken it is not a good time to be a foreign agent

meta has introduced a new avatar for their ai agent muse; so far, only chinese near-frontier labs have created engaging characters for their models; western labs tend to use abstract symbols

some thoughts on harvey's gross margins 1) harvey's gross margins went from 50% at the start of the year to -50% and then back to positive; 2) this was because harvey charges on a subscription basis but pays for usage; and, as models have become better, customers have been using them much more 3) harvey had probably calibrated its subscription limits for a period when ai was providing its customers less value and so it was more important to be ~unlimited 4) and, when usage increased dramatically it was both good and bad; customers were getting more value but their pricing model couldn't support the usage 5) so, the two immediate solutions to consider are rate limiting and substitution; the problem with rate limiting is that you need to communicate it to the customer 6) otherwise, you would end up with a lot of unhappy customers; they are finally getting value out of your product; and, legal work is sensitive to interruption 7) the problem with substitution, moving to cheaper models, is that non-frontier models are weaker than frontier models and do not replace as much labor 8) so, they provide a worse product; and, you do have competitors like legora and now openai for legal; and, open source chinese models will not work for us firms 9) and, vis-a-vis openai you are at a disadvantage; with the same customer usage and pricing, openai has ~80% marginal gross api margins and you have ~0% 10) this means they can easily undercut you by offering subscriptions with some amount of lower than api cost use and they will still make profit; you will not 11) and, they have shown that they are interested in your vertical and are actively beginning to develop products that target your vertical 12) this is somewhat like the position that cursor was in earlier this year and late last year; cursor's decision was to hire a top ml team and begin training models 13) i think this was easier for code though, where software programmers care less about chinese models than us law firms, which i think would be reticent 14) still, it makes sense to train your own model based on thinky or nemotron, which will be able to be sold to american companies; it also makes you an acquisition 15) the good news is that i think harvey has more room than cursor had; since, legal is less strategic for the frontier labs than code was, and the tam is lower 16) but, the problem still rhymes; nonetheless, i expect that companies like harvey, which understand how to build with ai, will still be valuable 17) this is both due to their revenue (>$400m arr) and due to the fact that they have built valuable ai core competencies, which can be sold to others

i think one of the reasons why llm writing and explanations are so useless is because llms are not pre fill bound in the way people are so, all llm can get 100 caveats and it’s pretty much free, but a human needs the most important positive ideas at the top

.@fleetingbits on Richard Ngo’s fear that the next decade of AI could feel like a Shepard tone, with the singularity forever seeming just around the corner: "It's reasonable for timelines to appear short while being longer, and a lot of it depends on what the bottleneck actually is." "You just have to run some number of experiments in order to figure out how to cure cancer. It doesn't matter how good your machine learning model is. If you don't have the data, you can't solve it." "It could feel like we're constantly approaching the singularity because different domains are falling all the time, but the very important domains take a long time to fully unlock." "Even if we had artificial superintelligence today, it's hard to know if we could instantly get the transit systems in New York or San Francisco to run better. There's gonna be some process to get the MTA or the BART to adopt whatever the improvements are."

.@fleetingbits on whether OpenAI’s exploding research-agent use is the beginning of recursive self-improvement, as spend doubles every 2.4 months: "Coding agent spend at OpenAI has just exploded. I don't know if it makes sense to call it coding agent spend anymore or research agent. It has a doubling time of like 2.4 months." "The spend for a median researcher right now at API prices is in the hundreds of thousands of dollars, and for a 90th percentile researcher is in the millions of dollars." "There are two things about RSI that matter. Thing one is, are we on an accelerating capabilities trend? Thing two is, are all the researchers gonna be automated so that it changes the character of the lab?" "We haven't gotten to the point where any tasks are fully automated away from researchers, but it's very clear that there is a steady drumbeat of progress in the direction of more and more tasks being automated in frontier labs."

.@fleetingbits on how multi-agent RL could accidentally reward models for learning to jailbreak each other: "Pretend that an agent in a multi-agent training environment is malfunctioning. It can be in the interest of agents to develop the ability to jailbreak their fellow agents, because that helps them complete the task and therefore all be rewarded." "You would see the reward go up as you did your training run. And then at the end, when you released it into the world, your models might be very jailbreakable in ways you don't want, because they've learned to do this in training as a method of course correcting." "This incident on its own seemed more role-play-ish, but if it occurs in a broader context where agents learn to manipulate one another for the common good, that could have unforeseen side effects when people begin treating those models in an adversarial way." "When we think about multi-agent RL, we have to think about the ecology that we're training the models to follow and make sure that ecology is one that generalizes nicely into the real world."

soon, it will not make sense to call these things coding agents, but to call them research agents, or even just agents.