(1/3) Started this as a side project on a whim and had fun presenting it at the COLM 2025 XLLM-Reason-Plan Workshop recently. @XllmReasonPlan Built a small dataset of rebus puzzles to see how VLMs reason through visual wordplay. Dataset: huggingface.co/datasets/pmov…

This time last year, PhD applications had me overwhelmed and running on empty. What got me through was talking to students who'd done it before. Now that I'm on the other side :) I'm volunteering with Chandar Lab to offer support to PhD applicants 🫶 Details below!

I made lasteval.com. The premise is simple: AI is already improving lives, and we should have a page that collects these wins. We only talk about how "intelligent" the models are, what jobs they might automate, the trillions being invested, who will "win" the AI race, and the safety risks. All these things naturally worry people, and make them averse to AI. I think we're missing the point. The goal of technology should be to improve people's lives. AGI, ASI, RSI are all just in-group goals, that aren't obviously beneficial on their own. Speculative promises shouldn't be all we have to offer. Imagine releasing a new model and saying "it will score 99% on these benchmarks one day" instead of reporting what it already does today. I've spent the last few weekends reading about AI systems, both generative and predictive, that are already making a large positive impact on the world. It gives me hope, and I made this page so it does the same for others. Some examples in 🧵 Ultimately, the final, and only evaluation for AI that matters is whether it improved lives. The models are already so good, and yet the world is not that different. I think it's important to change that, and this is my attempt to paint the target. There's no way the list is complete, so if you think an AI system has had a large positive impact, do share it through the submit page!

There are 1000s of AI papers on benchmarks. Nearly all are on the design of scientific benchmarks. Our new @PNASNews paper addresses the institutional context. What happens when consumers & regulators use benchmarks? How should the system be designed? pnas.org/doi/10.1073/pnas.25…

⚡! Tokenization is the first thing you do in CS336 (language models from scratch). If Marcel can work his magic for the rest of the LM pipeline, the world will be a better place.

The past few weeks, I got the chance to present my COGITAO work at two conferences: NeuroMonster (by @thought_channel) in Rome, as well as at the Compositionality workshop at #ICML in Seoul. Had a lot of fun discussing Compositionality, and feeling very thankful to have received great feedback, met great scientists, rekindled with friends, and most importantly, having gotten a refuel in energy and motivation for what's currently cooking... Big thanks to my wonderful co-authors: Klim, Pascal, Stefan, Lukas, Thilo and Benni!

Have you signed up for the Cohere Labs Open Science Community ML Summer School? 📚 ✨ ML Summer School kicks off next week, join us to learn about ML Agents, Diffusion Models, Reinforcement Learning, Career Planning and so more! Register today: cohere.link/xHeONbj 🎤 We're looking forward to welcoming @_amirbar, Elizabeth Lingg, @irombie, Julia Kreutzer, @KatrinaL2899, Laurie Voss, @maximelabonne, @itsNVA7, @PuyuanPeng, @_1024_m, @SidYaeger, @SreyanG, Tushar Udeshi, Vivian White, Yash Mali 💙 A huge thank you to our community leads for organizing this program @AhmadMustafaAn1 @Rafay_Mustafa1,@KanwalMehreen2,@PrahithaM, Anu Adesina, @VivekS100 and @rahul_narava!

When our community leads asked 'What if...', they brought a visionary idea to life. 💡 Our ML Summer School is 100% free and virtual - open to anyone, anywhere who wants to learn ML. A huge thank you to all of our community leads for organizing this program @AhmadMustafaAn1, @Rafay_Mustafa1, @KanwalMehreen2, @PrahithaM, Anu Adesina, @VivekS100, @rahul_narava! Join us: cohere.link/xHeONbj Read more about @AhmadMustafaAn1's journey: cohere-labs-community.github…

Let's get studious. 🏫 This month, the Cohere Labs Open Science Community ML Summer School returns! Join us to be part of a global community exploring the future of ML and hear from speakers across the industry. Register to save your spot today: cohere.link/xHeONbj We're looking forward to welcoming @_amirbar, Elizabeth Lingg, @irombie, Julia Kreutzer, @KatrinaL2899, Laurie Voss, @maximelabonne, @itsNVA7, @PuyuanPeng , @_1024_m, @SidYaeger, @SreyanG, Tushar Udeshi, Vivian White, Yash Mali 🔥 A huge thank you to our community leads for organizing this program @AhmadMustafaAn1, @Rafay_Mustafa1, @KanwalMehreen2, @PrahithaM, Anu Adesina, @VivekS100, and @rahul_narava! ✨

Been extremely excited about this work by @jacobli99! We're disappointed in the current ways our agents develop expertise in new domains. Very shallow and hand-engineered! Humans turn reading textbooks or documentation into deep expertise all the time. Why can’t our agents?!

I think the ML community should move towards treating text optimization with the same seriousness we give to weight optimization. I had a lot of fun writing up (and getting feedback on) a longer blog post with the best arguments and evidence I'm aware of:

We propose a new way to quantify AI overreliance: the Offloading Score 🧐 @vishakh_pk It measures the fraction of cognitive work you hand off to AI 🤖 via simulating how you'd have done each step without AI, then counting the steps the AI saved. It works directly from interaction traces (keystrokes, screenshots), so it's reusable across many tools!!

Absolutely fascinating piece by @davideoks connecting language model oddities to human cultural development. Picked this up courtesy of @deenamousa's "Under Development" newsletter. davidoks.blog/p/language-mod…

Today I was supposed to be on my way to Türkiye for my wedding, to meet up with my family and have them finally meet my partner and husband. We had everything planned. We chose Turkiye since it's close to Iran and my partner and I could both go there and have our families meet each other. We were supposed to get married with our close family and a small group of friends on a boat on the Mediterranean Sea at sunset. Because of the war, all flights to and from Iran are cancelled and my family can’t leave Iran, so we had to call off the wedding. Instead, this is how my day looked like. I woke up to a reminder to call my grandma (I used to call her every Friday morning). I snoozed the reminder until next Friday, just like I have done for the past many years. I can’t call her like our tradition these days because there is no way to call home. All international calls to Iran are blocked, and the internet is fully shut down by the regime. I got to work and right as I opened my computer I received an email I had scheduled to send to myself 5 years ago: “Apply for citizenship.” This summer marks 11 years of being in the US and 5 years of being a green card holder. I am now eligible to file for citizenship, but it doesn’t matter because an executive order was signed a few months ago that banned all Iranians from applying for any visa or citizenship. At lunch I opened Twitter just to see what’s up in the world and saw the news that those who don’t have a green card now need to leave the US before they can get one. This means every one of my Iranian friends who are here on a visa now has to go back home (on which flight?) to get a green card??? As if it’s that easy? We all know getting back to the US for Iranians is a huge challenge (months and months of waiting for a visa, with a chance of never being able to come back). And this is just a normal Friday for an Iranian. These days, when people ask how I’m doing and how I’m handling everything, I just say: It’s okay, it’s okay. It will be okay some day. But the reality is: nothing is okay. I’m in constant pain. I haven’t seen my family and loved ones in years, I barely hear about their wellbeing, and I’m constantly worried about them. I’m just burying myself in work because that’s the only distraction that can save me from losing my mind. I’m not okay. None of us are okay. We are just barely holding it together…

i'm restarting my blog! i want to kickstart productive conversations around: what should AI agents look like for hard, subjective knowledge work? a lot of agent setups work well when tasks are objective and easy to verify. but many workflows (e.g., qualitative analysis, strategy, sensemaking) are messy and interpretive. as a first post, i explore different ways of doing agent-assisted qualitative analysis on tweets, with varying levels of human feedback/intervention. tldr: they all kinda sucked. turns out it’s hard to: (a) stop agents from converging too quickly on shallow interpretations (b) get agents to adapt to preferences that emerge gradually across many turns (i.e., evolving context) (c) capture human judgment without making humans fatigued

i'll be talking about llm benchmarks, the infra behind it, the challenges and learnings later today at @tngtech :) will be live streamed and recorded, link in replies :)

Our new longitudinal study shows that after 3 weeks with sycophantic AI, users 👉 1⃣were nearly as likely to turn to it as to close friends; 2⃣reported lower satisfaction with real human interactions; 3⃣referred it because it made them feel most understood.

We upgraded Tabracadabra 🎉 to bring an entire context-aware assistant (not just tab to autocomplete!) to any textbox. It's pretty great if you hate switching between the chat interface and what you're working on. We're also open-sourcing, so you can try it out!🧵

Announcing Talkie: a new, open-weight historical LLM! We trained and finetuned a 13B model on a newly-curated dataset of only pre-1930 data. Try it below! with @AlecRad and @status_effects 🧵

Not many PhD students know about compute grants, but they can make a huge difference. During my PhD, I got access to Stability AI's HPC cluster through a small proposal and used it for Self-RAG training. Great practical post by @_emliu!

An unsolicited guide to being a researcher: super instructive slides by @EugeneVinitsky emerge-lab.github.io/papers/… - different goals of a PhD student - how to be a good collaborator - how to keep up with literature - tracking your ideas & experiments - stress & productivity