people are underestimating what a big deal this is this is the ONLY open-weight model that's trained without distilling from OpenAI or Anthropic • Kimi distills • GLM distills • Qwen distills • Nemotron distills (Kimi & DeepSeek, which counts) basically a fully different tech stack. the first pure open frontier coding model. very exciting

In this episode, @EngramLab co-founder and CEO @dan_biderman joins @allenpark to cook Mediterranean meatballs with yellow rice and talk about building AI that actually learns from you: why long context, RAG, and compaction eventually break down, how Engram compresses knowledge into cartridges and model weights, what continual learning could unlock for long-horizon agents, why token efficiency is inseparable from intelligence, how personal models could improve like Tamagotchis, and what it takes to build the research and infrastructure for millions of continuously updated AI memories. Timestamps: 0:00 Intro 0:26 Engram’s $98M Launch and Meatballs 1:45 From Naval Special Operations to AI Research 4:32 Israeli Military Culture and Founder Maturity 7:12 Why Engram Is Betting on Context and Continual Learning 9:14 Knowledge Cartridges, Compression, and Model Intuition 14:10 Trillion-Token Company Knowledge and Context Rot 18:05 Long-Context Limits, Compaction, and Neural Memory 22:20 Test-Time Training and “Destroying Prefill” 24:31 Harvey and Holistic Enterprise Queries Beyond RAG 27:02 Personal AI Models and Tamagotchi Weights 30:00 What Belongs in Weights vs. Text 32:25 Autonomous Memory and User-Specific Feedback Loops 34:20 Token Efficiency, Model Routing, and Harder Tasks 38:03 Engram’s Research Team and Product Culture 43:02 Hiring Researchers and Infrastructure Engineers 45:25 Doing More With Less 47:41 Where to Find Engram 48:19 Final Taste Test

my paper won an award at icml 😁 some thoughts: • this work was rejected from NeurIPS. i cleaned it up a small amount and it got great reviews from ICML! don't give up • ICML received 24k submissions and only gives out 7 awards, which is crazy. feeling grateful • i distinctly remember sitting at my desk two winters ago wondering if i would ever finish this project. most of all this is the product of sitting down and forcing myself to keep working for several months straight. the results emerged from running the experiments over and over and fixing a long sequence of tiny details. eventually, the curves looked like that 👇 • also happy that the insights in this paper are becoming more widely accepted: 3.3 bits/param, thinking about capacity "LLM as flashdrive" mentality • the method here is used successfully for selecting midtraining data at least one frontier lab, which is cool! • i am grateful to my collaborators, but Meta is no longer a great place for academic research imo and this almost never got published for a number of reasons. i shall not elaborate further • for future work, i think analyzing the implications of on-policy algorithms on capacity, as well as LoRA and things like it, are fruitful potential research directions • sadly i'm not in Korea but am following the conference online from california and happy to chat! a nice end to one phase of my research career :)

Engram cofounder @jxmnop just raised $98M to build a new type of AI. He says models don't need to get smarter over time. Instead, they just need to know you better and better over time. Jack describes what he's building: "Our product is a new type of AI. We have a pretty different vision from a lot of the frontier labs, which are working on one model per lab, and trying to make that model smarter every month." "There's another way to think about it, which is that the model doesn't need to get smarter every month. It needs to know you better." "So we're working on a whole different stack, which is a way to train models that train themselves to know your world better and adjust to the things that you say." "So: new ways of training, new ways of running the models."

The models we use every day are brilliant strangers. They forget your organization the moment a chat ends, then relearn it on the next query. @EngramLab fixes that. It learns your world once and reuses that memory, matching frontier systems on 1-10% of the tokens. @Microsoft, @NotionHQ, and @Harvey are already testing it within their organizations. Congratulations to the team, and hear directly from @dan_biderman (CEO and co-founder) and Sabri Eyuboglu (CTO and co-founder) with @LM_Braswell ⬇️

The models we use every day are brilliant strangers. They forget your organization the moment a chat ends, then relearn it on the next query. @EngramLab fixes that. It learns your world once and reuses that memory, matching frontier systems on 1-10% of the tokens. @Microsoft, @NotionHQ, and @Harvey are already testing it within their organizations. Congratulations to the team, and hear directly from @dan_biderman (CEO and co-founder) and Sabri Eyuboglu (CTO and co-founder) with @LM_Braswell ⬇️

The models we use every day are brilliant strangers. They forget your organization the moment a chat ends, then relearn it on the next query. @EngramLab fixes that. It learns your world once and reuses that memory, matching frontier systems on 1-10% of the tokens. @Microsoft, @NotionHQ, and @Harvey are already testing it within their organizations. Congratulations to the team, and hear directly from @dan_biderman (CEO and co-founder) and Sabri Eyuboglu (CTO and co-founder) with @LM_Braswell ⬇️