DOCTORS ASK AI FOR PAPERWORK, NOT DIAGNOSES
Hospitals now give their staff secure AI assistants based on large language models. Whether those assistants are ready is judged mostly by benchmark scores — examination-style questions or carefully selected cases, on which models now score close to the maximum. The paper notes that of 519 evaluations of language models in health care published up to early 2024, only 5% used real patient-care data.
A benchmark only says something about real use if its questions resemble what clinicians actually ask. Krithik Vishwanath, Tim Requarth, Eric Oermann and colleagues, mostly at NYU Langone Health and New York University, set out to measure that.
127,833 real requests
The team analysed the first message of 127,833 conversations between clinicians and the secure AI assistant of a large, multi-hospital academic health system, from August 2025 to April 2026. They came from 6,342 people in 35 specialties: attending physicians, fellows, residents, nurse practitioners, physician assistants and nurses. Half of the requests came from nurses and advanced practice providers, groups rarely included in AI evaluations.
To label every request, the researchers built a 20-field grid called RCQ-Map (task, intent, missing information, potential harm, safest response…) and had an AI model, OpenAI’s GPT-5.6 sol, apply it. They first checked it against seven clinicians on 100 requests: the model agreed with the clinicians’ shared answer about as often as a clinician did, 84.6% versus 83.0%. The researchers never saw the text of the requests, only the labels.
What clinicians actually ask
- Documentation and administration: 36.2% — clinical notes, care plans, discharge instructions, chart summaries, replies to patient messages.
- Looking up knowledge: 28.9%, including drug information (14.1%).
- Clinical reasoning about a situation: 14.3%, of which diagnosis was only 3.7%.
- A quarter of requests were mere keyword fragments, and more than half used clinical abbreviations.
- Specialists often strayed outside their field: 43.9% of their clinical questions concerned conditions managed by another department.
Many questions cannot be answered as asked
More than a third of requests (36.1%) could not be answered well as written. In 22.2% of cases, a good answer needed information the message did not contain — the patient’s history, the hospital’s drug list or protocols. For about a quarter of requests, the safest reply was a clarifying question; refusing or passing the question to a human was almost never the best option.
The model judged 13.2% of requests as carrying high or critical potential harm if answered wrongly, but it was more cautious than the clinicians: recalibrated to their consensus, the figure falls to about 3.7%. These risky requests clustered around medication doses and around decisions or checks of a planned action.
Tests built for another job
The team then applied the same grid to 58 public medical AI benchmarks. The typical benchmark contained no documentation requests at all, and its mix of tasks overlapped with real use by only 31% — less than a benchmark spreading its questions evenly across all task types would (55%). Benchmarks from suites designed to resemble clinical practice were no closer. In MedQA, a widely used exam-style test, diagnosis made up 48% of items, and almost every item was best answered directly.
One hospital, one tool
The study covers a single health system and a single assistant; it did not look at the assistant’s answers, so it measures potential rather than actual harm. Clinicians agreed only modestly with each other on how risky a request was. One author declares consulting for Google among other interests, and another consulting for a health AI company. The authors release their grid and their benchmark atlas, so that other hospitals can profile their own use. Their message to test builders: evaluate AI on the work clinicians actually bring to it.