Wouldn't it be great if you could watch a frontier model training run in real time? As of this week, you can.
All the big labs are training models right now, but none of us can see the details. When they finish, they'll publish their benchmarks and write some marketing blog posts, but they won't share the model weights, details of the training, or anything close to what you'd need to understand or replicate any of their work.
The open weight labs are slightly better: they write more detailed reports and share their weights. But they rarely share the infrastructure or software they train with or the problems they ran into along the way, and whatever they share is after the fact.
Enter the Marin, a 535 billion parameter mixture-of-experts model that began a three-month training run this week. Unlike all the other big models, this one is fully open, and not just open weights and open source software. The entire training and development process is happening in the open, right now, on 18 trillion tokens of data. All the work and discussion on the project is being done publicly, too.
Marin is a collaborative project built by researchers at Open Athena, a non-profit research lab, and Stanford's Center for Research on Foundation Models.
Check out the code, discussion, and live run dashboard here:
marin.community
(Fun fact: Open Athena's VP of Engineering is a
@RecurseCenter alum, as are a few other engineers on the team!)