Naked apes are killing a human. en.wikipedia.org/wiki/Killin…

It would be better if you fix your MAC first.

🧵New paper: "Lost in Backpropagation: The LM Head is a Gradient Bottleneck" The output layer of LLMs destroys 95-99% of your training signal during backpropagation, and this significantly slows down pretraining 👇

Today, we're releasing LFM2.5-8B-A1B, a device-optimized model designed to power real-life applications on phones, laptops, PCs, robots, and fast & lightweight server-side use-cases. > 8B MoE, 1.5B active > Expanded 128K context > LFM2.5 flagship hybrid MoE architecture > Trained on 38T tokens + large-scale RL > fast, reliable tool calling, punching above its weight, comparable to models with up to 4x its size > customizable on a single GPU for any specialized task > LFM2 open-weight license 🧵

Single vs Multi-hand Attention by hand ✍️ Resize matrices yourself 👉 byhand.ai/qNmYKw The most important fact about multi-head attention: it has the same parameter count as single-head attention. The difference is purely structural — same total Wqkv weights, partitioned into smaller q–k–v triples. Look at the two diagrams below. Both Wqkv matrices have the same height — same number of weight rows, same number of parameters. What changes is how that single tall block is sliced. • Left. One head. The full Wqkv produces one big QKV: a tall Q (36 rows), a tall K, a tall V. One scoring computation runs over those full-width tensors. • Right. 3 heads. The same-height Wqkv is sliced into 3 smaller q–k–v triples — each 12 rows tall. 3 scoring computations run in parallel, each a thinner version of the left. The compute trade-off — kind of. Same Wqkv weights. Multi-head runs the attention scoring S = Kᵀ × Q once per head, so the dot-product count multiplies by H. • Single-head: seq × seq = 40² = 1600 dot products • Multi-head: seq × seq × H = 40² × 3 = 4800 dot products (3×) But each multi-head dot product is narrower — its inner dimension is head_dim instead of H × head_dim. So when you count actual scalar multiplications, the totals are equal: • Single-head: seq² × (H × head_dim) = 40² × 36 = 57600 • Multi-head: seq² × H × head_dim = 40² × 3 × 12 = 57600 Same FLOPs. Multi-head buys you H independent attention patterns at no extra weight cost and no extra arithmetic cost — it's the same total compute, sliced into H finer-grained heads.

Hey @DimitrisPapail we now have a 512 parameter model that does the job. I instructed opus 4.6 to explore along the direction of low rankness.

491 parameters. 10-digit addition. ≥99.97% accuracy. AI can just do things, lol. Told me to post this, so I did.

Latex report ready. NB this is the result from a long session, two compacts, where I've nudged it several times to try more things

i asked Claude Code and Codex to run a glove box experiment: train the smallest possible transformer that can add 10-digit numbers. both did it in <2hrs on my macbook. CC came back with 6k params, Codex with 1,6k. the interesting part is how they got there. writing more soon

🚀 Introducing the Qwen 3.5 Small Model Series Qwen3.5-0.8B · Qwen3.5-2B · Qwen3.5-4B · Qwen3.5-9B ✨ More intelligence, less compute. These small models are built on the same Qwen3.5 foundation — native multimodal, improved architecture, scaled RL: • 0.8B / 2B → tiny, fast, great for edge device • 4B → a surprisingly strong multimodal base for lightweight agents • 9B → compact, but already closing the gap with much larger models And yes — we’re also releasing the Base models as well. We hope this better supports research, experimentation, and real-world industrial innovation. Hugging Face: huggingface.co/collections/Q… ModelScope: modelscope.cn/collections/Qw…

Today ggml.ai joins Hugging Face Together we will continue to build ggml, make llama.cpp more accessible and empower the open-source community. Our joint mission is to make local AI easy and efficient to use by everyone on their own hardware.

Created tiny-infini-gram, a training-free language model which can generate Shakespeare 250x faster than nanoGPT! Last year, I read about unbounded n-gram language models, which solve the exponential space problem for classical n-grams that made using large n intractable. By using suffix arrays, we can simulate any arbitrary-sized n-gram lookup table in logarithmic time.

Since I’ve been testing different small language models recently, I decided to implement this n-gram variant, and was surprised at how good the results were.
 Previous papers (to my knowledge) haven’t used this for language generation due to previous sampling methods causing infinite perplexity and verbatim copying. I solved these issues by creating Selective Back-off Interpolation Sampling, which mixes probability distributions from multiple n-gram levels to balance quality and novelty. 
A detailed write-up is linked in the comments.

📢 « Partition Generative Modeling (PGM): Masked Modeling without Masks » is out! 🚯 Masked diffusion models waste FLOPs processing countless mask tokens that carry no real information. ⚡We show how partitioning can replace masking, boosting throughput by >5.3x on text and up to 7.5x on VQ-ImageNet! 📄 paper: arxiv.org/abs/2505.18883 💻 Code: github.com/jdeschena/pgm 🤗 Models: huggingface.co/jdeschena/pgm 1/9 🧵

24 dedicated people. $30M spent on development. Extreme specialization, speed, and power efficiency. Today we launch Taalas’ first product. Check it out: Details: taalas.com/the-path-to-ubiqu… Demo chatbot: chatjimmy.ai API: taalas.com/api-request-form/

Fuck it, a bit early but here goes: Monty: a new python implementation, from scratch, in rust, for LLMs to run code without host access. Startup time measured in single digit microseconds, not seconds. @mitsuhiko here's another sandbox/not-sandbox to be snarky about 😜 Thanks @threepointone @dsp_ (inadvertently) for the idea. github.com/pydantic/monty

You shared an AI photo, Tony. The officer has no head You knew this and did it anyway…

The issue started long before social media. It started with Reagan's cancellation of the FCC Fairness Doctrine, which opened the door to divisive propaganda outlets like Fox News and conservative talk radio. Then, badly run social media like this one made things worse. But political polarization in the US started increasing over 40 years ago.

“I’ve been betrayed.” Epstein survivor Anouska DeGeorgiou says the Trump DOJ released unredacted personal information she was compelled to provide for the Ghislaine Maxwell trial, despite assurances survivors would be protected. She is furious at the DOJ and House Speaker Mike Johnson. @AGPamBondi needs to answer for this! It’s left survivors feeling unsafe, exposed, and retraumatized. Who approved this? Why were safeguards ignored? Meanwhile the DOJ has redacted names of alleged co-conspirators.

BREAKING: Proof that the DOJ Is Scrubbing Epstein Files with Trump's Name in them. Today there are 1513 Results for "Donald Trump" Yesterday at 9:15 am there were 199 more files, 1712. If they are doing all of this in front of the public, just imagine what they hid behind closed doors.

Anyone that still didn't know that France's far-right darling, Marine Le Pen is funded by Russia. "was having a meeting yesterday with the French right wingers, including Le Pen, her husband, about refinancing the National Front Party because apparently a good part of their money comes from Russia. Two thoughts here, these guy seemed like idiots, pure Ruritania"

"Why do Epstein victims not speak out more about who raped them?" Virginia Guiffre was the most outspoken victim. She mysteriously died by suicide just as the Epstein Files reentered public discourse. Her dad maintains she was murdered.