The wall between "what the doctor knows" and "what the patient can find" may crumbling faster than first thought
A new study out of NYU Langone published just last week seems to have landed a blow nobody in clinical AI saw coming. Researchers pitted OpenEvidence and UpToDate’s Expert AI — the purpose-built clinical decision tools doctors have come to rely on — against three general-purpose chatbots: ChatGPT, Claude and Gemini.
The frontier models won. Not narrowly, either. They beat the medical-grade tools on textbook knowledge, on alignment with expert clinical judgement, and — most damningly — on actual questions doctors asked during actual patient care.
OpenEvidence, the supposed gold standard of clinical AI evidence, had 52 flagged errors: incomplete clinical content, safety-critical omissions, disorganised answers while UpToDate refused to answer one in five real questions outright according to the study.
Gemini had eight errors, ChatGPT had 21 and Claude 19.
In other words, the tools built specifically for doctors, trained on curated clinical literature, with safety rails and liability protections baked in, lost to the same chatbot your patient used last night to ask why their knee hurts.
Not surprisingly the owners of OpenEvidence and UptoDate are disputing the results.
OpenEvidence says there is contamination (frontier models may have seen the benchmark questions during training), what they claim is misrepresented metrics on HealthBench scoring stylistic choices rather than clinical substance, and concerns about the peer review record.
Wolters Kluwer’s chief medical officer told Becker’s the study “confused clinical quality and complete-sounding answers” and that UpToDate’s refusal to answer underspecified or risky prompts may actually be the safer behaviour, not a deficit
What if the near-free consumer LLMs can match or beat the specialist clinical AI tools on real physician queries?
Well, the patient sitting across from you tonight, googling their symptoms at midnight, may be retrieving information of comparable clinical quality to what you’d get querying your own, at times very expensive, decision-support software.
Much worse, the wall between “what the doctor knows” and “what the patient can find” may not be slowly crumbling. It may already be mostly gone.
Clinical judgement is going thank heavens so you’ve still got a job. Maybe even a much more interesting one as you will start a lot more conversations with some patients from a very different position .
But that asymmetry on medical knowledge that doctors have tended to quietly lean on for decades without examining it too closely is certainly in trouble. Now the asymmetry is going to be much more around experience, communication and ability to orchestrate and help a patient navigate.
There are real caveats in this study though, so likely we aren’t anywhere near the headline happening yet.
But the deeper signal seems established. The tools built to give doctors an edge are being out-performed by the tools sitting in everyone’s pocket. The asymmetry of information is going.
The asymmetry of judgement is what’s left to defend.
That must have huge implications for doctor education and training, starting now, you’d think.
