This is going to be a pretty long answer, but just to preface it: if you really disagree with this, or you feel strongly that you have data that proves me wrong, let’s discuss it properly. We can do a public roundtable with you, your research team, engineering guys, whoever else you think would add value to the discussion. My community is almost at 2 million people monthly now, and I think it would be a great way to show people how you guys are thinking about this, why you’re so confident in the approach, and where you think my analysis is wrong. The point would not be "debate" (I think that's corny af), but to have a good-faith discussion about our differences in opinion/technical thesis in a way that everyone can walk out with a better understanding of things. Happy to also talk privately if that's better. Or just argue with code, as they say. We'll define a task, define a budget, we build it in our own ways, and let’s compare the results. Now onto the answers. TLDR: training custom models would be wasteful for law firms and most challenges around sensitive agentic deployments are not a model layer problem. Trying to solve them in the model layer creates major security issues that law firms will not want to deal with. 1-- when you talk about helping law firms build their own models/building on their private data, you're likely suggesting post-training, because obviously they will not have the amount of money and data required to train a completely new intelligent model from scratch. Especially one that’s competitive against the really good open models out there currently. This is not useful to them because stacks of research have already shown us that most of the knowledge of an LLM comes from the pre-training stage, not from the post-training. Post-training is essentially just a way to change how the model samples from its latent space, but the way concepts are organized and information is actually learned is mostly coming from the pre-training stage. This is a problem for law firms because they don’t want their own custom knowledge, their client data, their own proprietary knowledge, to be confused with generic off-the-shelf internet insights (since you're not changing the pre-training). The whole point of training a model on your custom data would be that it’s much more precise, much more grounded in your data (including private data that isn't available in generic models). If you’re only post-training and not pre-training, most of your knowledge is still coming from that generic base. You can actually confirm this for yourself with your tenant models. Compare outputs on identical tasks between the tenant model and the base model it was post-trained from. You’ll see that the distributions are actually quite similar, especially when you account for the amount of resources you would have had to spend getting there. 2) let’s say you realize this and you’re post-training the model not to be more knowledgeable, but specifically to navigate a law firm’s spaces better: learn how to navigate through Google Drive connectors, any enterprise software they’ve bought, DISCO, Clio, etc. The problem there is most generic off-the-shelf LLMs, especially the really cutting-edge ones, will already have had access to the same tools and will know how to navigate these environments themselves. It’s not like these environments are alien technology that intelligent models will not be familiar with. So then you’re again stuck on the same issue: what is the ROI? Especially when you consider the fact that once you start getting into post-training, what happens when the next generation of model comes out? Are you just going to start post-training again? You’ll have to post-train that and redo all of your investment. So really all that gets left are the very, very specific law firm things: ways they want to practice, where the templates are, how to find their templates, how to use their templates, etc. That is incredibly valuable. But any generic intelligent model plus a well-defined harness and a well-defined customized context map can handle this much better. Customizing the harness (this includes setting up things like access controls, which we will talk about in a sec) and context map will be 10x cheaper and much better than training a model. If you're skeptical about this, let's back both of our claims with code. 3) Now let’s say for the sake of argument that your law firm is committed to investing a lot of money into pre-training and post-training a model. They have the data, they have the compute, and they have the stubbornness to go ahead with this. The final problem here is that agent deployment in sensitive enterprise settings is not blocked by intelligence. In most cases, it’s blocked by other important aspects, such as data access control, strict edit permissions, etc. Certain employees should be able to access certain kinds of data; other employees should not. For some questions, answering them might require the system to access data that the employee should not see directly. It should be able to use that data to produce an allowed answer without exposing the raw data, or simply say that at their level of access control it cannot do this. Both are valid, depending on the org rules, but monitoring and enforcing these rules should not be purely left to LLM calls. Similarly, what about edit controls? You should be able to edit your files, not edit files above your permission level and absolutely not touch client data. If you try to solve these kinds of problems in the AI model layer, you are just introducing unnecessary stochasticity. You will have mistakes. Look no further than coding models for prood. despite tens of billions of dollars invested into post-training coding models, coding models continue to do stupid shit like delete production databases and rewrite access controls. And coding models have orders of magnitude more investment behind them. Coding is more verifiable, so it’s easier to train with post-training techniques. Coding is also a domain AI researchers and engineers are familiar with, so there’s much greater domain understanding there compared to something like law, where your AI researcher and your AI engineer will usually not know law particularly deeply. Despite all of those advantages, coding models still have not solved these problems because these are not AI model-layer problems. Anybody trying to solve these problems by just training a law-firm AI model is introducing unnecessary security risks and unnecessary costs. All of which will compund when the agents spins out subagents in swarms. 4) And I haven’t even started to mention the fact that even if you do train your own model, you still have to worry about inference. There’s a reason AI inference companies and providers that optimize GPU utilization and inference utilization make so much money. What are you going to do? Go to a cloud/neo cloud provider and tell them to host your model? Host it in a VPC? Because otherwise you’re asking law firms to invest not just in research teams and model teams, but also inference engineers, cloud-serving engineers, DevOps guys, and everything required to make sure one person’s query doesn’t get messed up with another person’s query. These problems continue to compound. You’re adding a lot of unnecessary headache onto people, and you’re doing all of this for basically no gain. It is much simpler for a law firm to take, if they want, an open model hosted in their environment or through something like OpenRouter. Fine. Or if they want a closed model, they can use an API with ZDR and no training guarantees. Then worry about setting up the harness, the context map, the access controls, and building a system with the security and access-control structure in place. That will give you 10 times the results. Happy to hear where you see things differently

We audited thousands of agent rollouts in DeepSWE-1.1. Over 80% contained reasoning about an imagined grader. Yet no grader/verifier is mentioned in prompts nor accessible to the agents. Agents reasoned things like: "Let me look at the problem from the grader's perspective" and referred to "hidden tests", "test authors", and "the checker". We found this behavior across all six frontier models we analyzed, including recent models from OpenAI, Anthropic, Z ai, and Kimi. In 10-25% of cases, such reasoning pulled the agent's work away from the user's original specification (yet it often still earned full reward on the DeepSWE task). We call this speculative reward hacking where the agent's reasoning focuses on imagining a grader rather than what the user wants. Hui Wen & I published an article today with the problematic trajectories, quantitative findings, and a taxonomy of these reward hacking behaviors: joinhandshake.com/research/a… Our research community needs to do better. Better evaluations that penalize such reward hacking, and better model training that does not give rise to this grader obsession -- so that models focus instead on accomplishing what users actually want.

I turn 50 today. This is a message to my kids but I offer it to you in case any of it rings useful. I was born in Sri Lanka in 1976 in a country that, at the time, was about to enter a decades long civil war. My father was able to get stationed at the Sri Lankan High Commission in Canada for four years in the early 80s - when some of the fighting was most severe - and which from our vantage point in Canada gave us some respite from an otherwise omnipresent conflict. He came as a diplomat but then my father filed for refugee status after four years when his life was threatened by militants. We stayed in Canada as refugees. My dad never really landed on his feet and we grew up on welfare. My mom stepped up. She did an incredible thing and put our family on her back and kept marching forward. Sometimes the steps were very small but she always tried to keep moving forward. I worked at Burger King starting at 14, giving my paycheck to my mom and dad to support our family in the role that I was assigned - earn enough to pay for our bus passes because we couldn’t afford a car. I was a loser growing up. Some friends, some good memories but mostly forgettable and lonely. I wasn’t very studious but did get my act together in my final year and got into a great university. I immigrated to the US soon after graduating and found an opportunity. Things clicked and I have been traveling a path to find myself ever since. Only in America is this possible… I worked hard, followed the rules, paid my taxes and stayed in California out of loyalty for the opportunities it has given me despite its dysfunction. I found my wife here. Had my kids here. Found my tribe here. There have been some very hard times as well - deaths, divorce, financial pain, mistakes of omission and mistakes of commission. You can’t dwell on these things - they are weights that will pull you down if you’re not careful. Learn from them. Respect them. BUT MOVE ON. To think 50 years ago I could have ever gotten a chance to meet the people I’ve met and have had the experiences I’ve had would have been mind boggling to my parents. It’s mind boggling to me now. We forget often and try to convince others it’s not true but America truly is the best country in the world. Everything and anything is possible here. I can testify. Be grateful to be here. This is the only place that gives EVERYONE a chance to live a great adventure. The first 50 years have been that for me - an incredible adventure. Notice there is nothing here about accomplishments or money. Because it doesn’t matter and it won’t matter. You will be dust in less than 100 years from now and the decision to be remembered won’t be yours so stop sweating it. It’s not in your control. So what is in your control? Be kind. Have fun. Work on something meaningful to you. Find your tribe. Try to find perspective. Find an incredible partner. Have a bunch of kids. Live an adventure. Forgive your parents - they were doing the best they could even if that wasn’t what you deserved. I will do what I can to make my next 50 years a wild ride. You should too. I love you and thank you. ❤️ 🙏🏽🙏🏽

So it turns out China was building a hydro electric facility by a glacier and all the construction caused a portion of the glacier to collapse, tumble down, and turn into a mudslide that killed like 2,500 Nepalese people. But all you will hear about in the news is leftists claiming this was because of climate change

Industry breakdown of the millions of agreements that flow through Spellbook. The world runs on contracts. Contracts run on Spellbook.

This one triggered people on all sides, so let me expand: 1) agents have gotten way slower, more verbose, more paranoid, idiosyncratic 2) yet they still (or even more often) make massive blunders. 3) This is more costly now. Opus will take a day to build something totally incoherent, even after a very clear plan 4) meanwhile sol is hyper focused on the nearterm goal, can be steered, but cannot regain its original trajectory 5) they write too many dumb tests, catch too many errors, build too many guardrails 6) which makes the codebase slower to parse, harder to read, slower to build, costlier for them to work in 7) they solve all of these problems with MORE complexity. Let’s multithread your tests, let’s create a fast running CI script, let’s create more packages, more repos. 8) so the loop goes on 9) many (most!) codebases at this point are worse off than they were 18 months ago. 10) oh just have another loop refactor everything? Another loop clean up the tests? Spend twice as many tokens tearing things down that were put up yesterday? That’s the solution? 11) this doubles the change requests of the already verbose code. Oh just get code rabbit? Greptile my PRs? All I need is a factory? I mean, don’t get me wrong. We’re past the point of no return. The only way out is through, I just hope we get there.