NEJM Medical Image Challenge of the Week vs. the Medical AI Council, too easy for AI models?! 🤔👀 All models predicted Fusarium keratoplasticum, consistent with the majority vote!

Claude Opus 5's performance on MedAgentBench drops substantially compared to its predecessors, Opus 4.7 and 4.8: roughly half their success rate. 🏥 We benchmarked Opus 5 as an autonomous clinical agent across 10 EHR task types, and it landed at ~40% pass@1 (avg of 3× 300-task runs), well below both earlier Opus models on the same benchmark. We dug into why 🧵

A closer look at both benchmarks: 🔹HealthBench Professional: Kimi K3 keeps pace with the top models on routine clinical cases (0.77, just past GPT-5.6 Sol), but falls off on the hardest cases and on adversarial / red-teaming prompts, that's where its distance from the leaders comes from. Its scores held steady across all 3 runs. 🔹MedAgentBench: Kimi K3 handles the lookup and lab-retrieval tasks cleanly, but stumbles on the multi-step protocols that chain several tool calls under a strict output format, potassium replacement (0.57) was its weakest task. 🔹vs Kimi K2.6: a clear step up on both benchmarks, most visibly on patient lookup (0.68 → 0.97). One task moved the other way, vital-sign recording, where K2.6 was perfect and K3 slips.

OpenAI's GPT-5.6 Sol just landed on MedAgentBench and claimed the top spot 🥇🏥 MedAgentBench is a medical benchmark of 300 test instances evaluating LLMs as autonomous clinical agents across 10 EHR task types (30 instances each).

A few observations from the runs: 🔹Aggregate rank hides task-level variation. No model leads on every task. The new #1 records its lowest score on HbA1c ordering, below the other three on that specific task. Winning overall means the fewest weak spots, not zero. 🔹The ranking is decided on the hard tasks. Age math, vital-sign recording and patient lookups are at or near 100% for all four. Separation comes almost entirely from multi-step write actions, ordering potassium replacement and HbA1c follow-ups, where scores swing by ~30 points. 🔹A narrow field. The top four sit within a few points of each other, with under 1 pt of run-to-run variance for most. Differences are real but small, and could shift with a handful of tasks. 🔹Failures are clinical, not formatting. All four reliably follow the tool-call protocol, near-zero invalid-action errors. Remaining misses come from wrong values or not terminating; muse-spark-1.1 most often ran to the step limit.