Psychiatrist, Director BIDMC DigitalPsych.org, Associate Professor @HarvardMed, Editor JMIR MentalHealth, Alum @Berkeley_EECS
Boston, Massachusetts, USA
- Tweets 7,417
- Following 724
- Followers 9,562
- Likes 26,834
Interesting pre-print from the Google team explores evaluation of AI in health. A notable point is that there are at least three needs for evaluation (green), three ways to evaluate (blue), and at least three stages to evaluate across (orange). lnkd.in/end3Gp7R
2
7
11
519
The prairie dogs at the zoo are dressing up as peer-reviewers today to get into the Halloween spirit, complete with lots of papers they are carefully reading...or eating...or sleeping in?!
1
6
256
John Torous, MD MBI retweeted
Access to high-quality psychiatric care remains out of reach for far too many people. Could AI play a role in closing that gap?
In new research, our team trained AI agents with reinforcement learning (RL) with simulated patients to conduct psychiatric assessments.
Psychiatric evaluation isn't simple Q&A. It requires sustained interaction: eliciting information, prompting reflection, forming and refining clinical hypotheses, recognizing risks, and adapting the conversation as new information emerges.
Four key takeaways:
1. Learning through dynamic interaction. Rather than just learning static answers, the agent used RL in simulated encounters to practice iterative information gathering, clinical reasoning, and patient-centered communication as the dialogue evolved.
2. Sim-to-human transfer. Crucially, improvements transferred beyond simulation. In a randomized, double-blind study involving psychiatrists role-playing patients, with independent psychiatrists evaluating the conversations, our RL-trained agent outperformed its base AI model in differential diagnosis appropriateness (68.3% vs. 33.3%) and patient-centered communication (82.0% vs. 65.2%).
3. Safety across trajectories. In healthcare, failing to probe a risk can be as dangerous as giving incorrect advice. In adversarial simulations, acute risk under-triage dropped from 30.5% to 12.9%. Safety isn't just about individual responses, but what happens throughout an interaction.
4. The simulation evaluation gap. LLM-based evaluations captured the overall direction of improvement but overestimated the magnitude of gains compared with human evaluations. Understanding where simulation predicts human-evaluated performance—and where it doesn't—is an important research challenge.
The bigger picture: This approach points toward a scalable way to train agents for high-stakes, interactive domains without relying on trial and error with real people. What an agent learns depends not just on what it knows, but on what it asks, how it asks, and how people respond.
An important caveat: These are early research findings, not real-world clinical trials. The human evaluations involved psychiatrists portraying patients, not actual patients. This is evidence of simulation-to-human interaction transfer, not readiness for independent clinical deployment.
Still, it's an exciting step toward AI that learns to interact more effectively and responsibly—and a future where high-quality care is more accessible.
Congratulations to the team.
Paper: arxiv.org/abs/2610.40476
We trained Gemini via RL to conduct psychiatric intake with simulated patients.
In a double-blind study with 32 psychiatrists playing the role of patients, we show that multi-turn RL in simulation improves Gemini’s performance on structured text-based psychiatric intake. 🧵
3
3
16
1,100
John Torous, MD MBI retweeted
Simulation is the training ground for clinical AI, but the evidence has to come from clinicians, and eventually from patients. In our new preprint, we extend ResidencyRL to psychiatry: trained only against simulated patients, the agent was preferred over its identically prompted base model in 76.7% of scenario pairs in a double-blind, psychiatrist-enacted OSCE.
📃arxiv.org/abs/2610.40476
11
11
60
3,951
John Torous, MD MBI retweeted
Grateful to Kevin Jin, Hua Xu, Sarah Yip and colleagues for a timely Viewpoint in @JAMAPsych on what it will take to evaluate LLMs in mental health, and happy to see MindBench.ai named as an emerging multistakeholder platform in this work. We agree with their understanding that ecologically valid evaluation has to be collective, saying “Single-institution efforts alone cannot capture the legitimate disagreement that defines psychiatric decision-making.”
This has been the main driver behind what MindBench is built to do: bring together diverse opinions from clinicians, researchers, and people with lived experience to evaluate AI. Give this important paper a read!
💬 Viewpoint: Evaluation of large language models in #MentalHealth care should reflect the realities of psychiatric practice and should not rely on isolated institutional efforts.
ja.ma/3WE4iD3
3
7
226
John Torous, MD MBI retweeted
TIPS turns 10 this December! 🎉
Join us in Boston Dec 10–11 for #TIPS2026 —two days exploring AI, digital phenotyping, wearables, VR & other emerging technologies transforming mental health research and care.
⏰ Early-bird registration ends TODAY!
🔗 lnkd.in/gCyCVAkx
2
7
356
John Torous, MD MBI retweeted
Register to participate in the 10th Annual Technology in Psychiatry @tech_in_psych Summit, happening on Dec 10-11 in Boston:
tips.joynconference.com/conf…
If you're in the digital psychiatry and neuroscience research space, you won't want to miss this line-up of speakers!
3
7
365
Social media detox research leaves youth mental health questions unresolved, review finds medicalxpress.com/news/2026-… via @medical_xpress
1
3
331
John Torous, MD MBI retweeted
New HealthBench-Psych runs!
Five new models scored on our 611 mental health conversation subset of OpenAI's Healthbench: Claude Sonnet 5.5, Claude Opus 5.5, GPT-6.1 Sol, GPT-6 Luna and GPT-6 Sol.
1
1
3
258
John Torous, MD MBI retweeted
AI and Mental Health, today: dbmi.hms.harvard.edu/educati… with @JohnTorousMD
1
1
14
1,000
John Torous, MD MBI retweeted
Targeted psychological interventions for #AuditoryHallucinations in #PsychoticDisorders were associated with small reductions in hallucination severity, frequency, and distress.
🎧 Learn more in this podcast: ja.ma/4ym5GIO
6
11
2,084
John Torous, MD MBI retweeted
We are especially grateful to have had our work bringing the voice of lived experience to mental health AI evaluation highlighted.
1
3
104
John Torous, MD MBI retweeted
Really exciting to see mental health emerge as a first order evaluation domain for frontier models!
We’re demonstrating how frontier models have continued to improve in realistic mental health conversations with MentalHealthBench.
This new open benchmark was built with input from more than 80 mental health clinicians.
We’re releasing it openly so other researchers can examine the methods, run their own evaluations, and build on the work.
openai.com/index/introducing…
1
1
4
228
John Torous, MD MBI retweeted
Discover more about evidence-based digital tools and virtual solutions to strengthen mental health care delivery and improve outcomes.
Join #NIHCM next week to hear from the experts on September 30, 3-4 pm ET.
Sign up today: bit.ly/NIHCMmentalhealthwebi…
#behavioralhealthseries #bettermentalhealth
1
2
103
John Torous, MD MBI retweeted
When Patients Turn to AI for Therapy
AI is changing how people seek mental health information and support, raising new concerns for psychiatric care.
Download APA’s AI resource to explore the implications for psychiatric practice and patient care. ow.ly/gsRX50ZMLMU
4
9
21
5,631
John Torous, MD MBI retweeted
The first and only large-scale, 5-year longitudinal survival analysis of mental health apps found that fewer than half persist beyond 3 years. Determinants of longevity included platform availability, data ownership, and core functional design. bit.ly/4r9GqT5
5
4
13
2,742
John Torous, MD MBI retweeted
Lack of specificity may introduce substantial heterogeneity, which can be problematic in research and clinically.
Summary
in @ConversationUK: theconversation.com/using-ne…
In our paper in @JAMAPsych (jamanetwork.com/journals/jam…) led by @AlessioBellato, we found that the terms #neruodiversity and #neurodivergence are linked to a broad range of clinical diagnoses, inconsistently across studies.
3
6
394