We have a new paper on topic models! We introduce Mechanistic Topic Models (MTMs). MTMs model SAE feature counts rather than words or plain embeddings, combining the benefits of LLM-based and probabilistic topic models. Paper (published in TACL) : arxiv.org/abs/2507.23220

Sep 23, 2026 · 1:18 AM UTC

6
14
125
7,018
We show that MTMs are better than traditional and neural baselines at finding salient and abstract themes, while costing less than LLM-based methods. Moreover, working with SAE features allows you to construct topic steering vectors to steer LLM generation toward MTM topics.
1
4
512
We are presenting a poster at EMNLP in Budapest next month! Github: github.com/blei-lab/mechanis… With @carolinazheng_, @nicolasvelezb, @SwetaKarlekar, @causalclaudia, @AchilleNazaret, Asif Mallik, @amir_feder, and David Blei.

GitHub - blei-lab/mechanistic-topic-models

Contribute to blei-lab/mechanistic-topic-models development by creating an account on GitHub.

github.com
2
13
494
Sort replies: Relevant Recent Liked
Replying to @blei_lab
The nice property is that SAE features carry a decoder, so a topic gets a mechanism rather than only a label. The catch is that they are model-specific, so the topic describes the representation as much as the corpus. Do the same topics survive a different SAE?
1
74
Replying to @blei_lab
Modeling SAE features instead of words is the first topic model I have seen that actually knows what it is talking about.
84
Replying to @blei_lab
Topics now inherit the SAE dictionary. If you retrain the SAE at a different width or on another base model, do the same themes come back, or does each dictionary carve out its own topics?
1
32
Load more