Author has a history of lying about results "SWA beats Linear attention" when this is only a very specific case "TRM is a 7M parameter model that beats much bigger models" when its actually ~440M, with 7M active

Who dat

Actually lecun says exactly that in the slide before this I assume the subtext is "for human level ai", build these types of world models.

irl in a separate interaction I mean also found a video of this talk. He says "world models should be ..." youtube.com/watch?v=vJKC31Yp…

Yeah some of the tasks it can solve are not easy!

Then why does it outperform others with such a small transformer / such little training data? Probably good representations -- ablations point to 3D RoPE and the per-task embeddings Although I'm sure it'd perform the same if I scale it up very high

A big change -> No training on inputs Interestingly, this makes the test loss much worse and yet it scores better! Clearly a compression framework / val loss alone isn't a perfect metric for sample efficiency (I am bullish input training will make a comeback though)

44% on ARC-AGI-1 in 67 cents! Trained from scratch in 2hrs on a 5090 Matches TRM, beats HRM and is way faster & cheaper No recursion, just a transformer Also, 7% on ARC-2 🧵

Tribalism + incompetence + a super dumb mainstream media that loves ragebait

Sutton has been talking about the problems with gradient descent forever I was surprised Dwarkesh didn't know this watch this to understand: youtube.com/watch?v=gEbbGyNk…

a) Give it a pen and a stack of papers and its a turing machine[0] b) "Computer" referred to the human job till we realized machines can do this too. Turing's 1936 paper also uses it to denote a human ([1], pic) c) McChulloc and Pitts[2] showed that their model of the human neuron is an FSM and most modern models of human neurons are shown to be turing complete (iirc) --- [0]: yes, its not infinite, but none of the machine computers we have today are infinite either [1]: cs.virginia.edu/~robins/Turi… [2]: cse.chalmers.se/~coquand/AUT…

This was based on what I read in the blog (pic attached) I found what you're saying on the github readme. Maybe I misunderstood. @LiaoIsaac91893 could you pls confirm?

1) Also, I address this in the blog. I have been open and I say its necessary Does a human not learn when he looks at the test input? Why is doing unsupervised compression on test inputs bad?

This is no different from previous models on ARC-AGI. They have 2 phases, but the second phase is a continued pretraining I'm quoting @GregKamradt himself here on another approach: "relatively expensive to run because pre-training and inference are coupled" x.com/GregKamradt/st

In technical terms, this is joint self-supervised compression. To justify this, I rely on the Minimum Description Length principle (MDL) I take MDL to its logical conclusion by compressing every possible source of information

Every prev approach used supervised learning. This means the models were only learning the outputs. The main modification is forcing the model learn the inputs too This dramatically increases performance and reduces training costs

Announcing New Pareto Frontier on ARC-AGI 27.5% for just $2 333x cheaper than TRM! Beats every non-thinking LLM in existence Cost so low, its literally off the chart Vanilla transformer. No special architectures. Tiny. Trained in 2 hrs. Open source. Thread:

Simpler example The radial probability of the 1s orbital of hydrogen falls to zero at the origin

I always thought they overindexed on training images of old oil paintings that have the distinct yellow tint

weird one: pass@30 fails but gemini solves easily if you tell it to fix the mistake

You need a heavy agent harness. Otherwise you get stuff like this:

Hypothesis: You need a verification loop to close the final gap I got gemini to solve ~20% of the unsolved tasks in ARC-1 public eval this way (maybe more possible) Example:

Problem: Test inputs are distribution shifted (contain unseen information). Unless you compress this new info, you can't generalize Instead of compressing, solvers rely on hacks (like augmentations) This is anti bitter lesson, mathematically unnecessary, and can backfire!

MDL says better compression = higher intelligence. Yet no solver compresses all 3 - example inputs, test inputs, private puzzles Mutual information implies compressing them together is better (since this includes private puzzles, test time training is necessary!)

New blog post Why all ARC-AGI solvers fail today (and how to fix them) tldr they all violate minimum description length why test time training is necessary anti-bitter lesson hacks are unneccesary falsifiable predictions Link in thread

Information theory guys: We should always be unambiguous Also information theory guys:

This is one of the most beautiful illustrations I have seen. Check out his blog!

35x and maybe I can go faster

How far can vanilla NCAs go on ARC-AGI? Farther than you'd expect. I benchmarked fixed-step, unmodified NCAs on all public tasks from ARC-AGI-1 and ARC-AGI-2 They hold up remarkably well. Its also dirt cheap. Results + visuals:

Self organising systems are extremely resource efficient. I've been pushing a particular model for the last few weeks on @arcprize and here's what I got without a GPU:

Bruh its right there

This is the dream... Inventing and selling things. One day for sure. Simone Giertz -> foldable coat hangers youtube.com/watch?v=vREokZa4… kickstarter.com/projects/sim…