But some clinicians get more benefit than others. It’s all a bit underwhelming.
The most extensive study to date about the impact that AI scribe use has on clinician time and productivity has found the metrics are, frankly, underwhelming.
Clinicians using an AI scribe spent 13 minutes less a day on documentation and gained half an extra patient consult a week, according to the large, multi-site study. An accompanying commentary asked whether, even if those numbers were to increase, would patients be better off, and were the right questions being asked.
The US study found that there were differences in the measured benefits depending on clinician characteristics and AI scribe use intensity, and researchers said that even modest benefits seen in these specific time-saved metrics could influence clinician satisfaction and contribute to reduced burnout. The researchers did not measure the latter, however.
This article originally ran on TMR’s sister site, Health Services Daily. TMR readers can sign up for a discounted subscription.
The study looked at AI scribe use across five major US health institutions – Massachusetts General Brigham; Emory Healthcare; University of California, San Francisco; Yale New Haven Health; and the University of California, Davis.
All of these institutions made AI scribes available to some of their clinicians—attending physicians (including clinical fellows), advanced practice clinicians (nurse practitioners and physician assistants) and resident physicians, although they did not have to take it up. They were all called “adopters”. All the institutions also provided data for those who did not get access but who worked in areas where some of their colleagues did have access (“nonadopters”).
They all provided data for a minimum of 12 weeks, including six preadoption and six postadoption weeks for adopters. All the institutions used Ambience, Nuance DAX Copilot, or Abridge (individually or in combination) and Epic as their EHR vendor.
In total there were 1809 adopters (509 in primary care, 1080 medical specialists and 220 surgical specialists) and 6772 nonadopters – 8581 clinicians altogether. Nearly half of the sample (57.1%) was female.
Of the 1809 adopters, 780 used AI scribes for less than 50% of their notes, just 577 for 50% or more, and the proportion was unknown for 452.
Across the sample, in an eight-hour day, AI scribe use cut electronic health record (EHR) work time by 13.4 minutes (range: –17.70 to –9.10) and documentation time by 16 minutes (18.30 to –13.70).
It only cut EHR work outside work hours by 3.10 minutes per eight-hour day (range: –6.80 to 0.50) and added an extra 0.49 visits per eight-hour day (0.17 to 0.8).
“It is notable that estimates for documentation time reductions exceeded reductions in total EHR time expenditure and that AI scribe adoption was not ultimately associated with significant changes in work outside of working hours,” the study authors wrote.
“This is consistent with previous studies and suggests that while clinicians may save time on documentation with AI scribe use, some of those time savings may be reallocated to other patient care activities, such as reviewing current or prior documentation for accuracy, answering electronic inbox messages from patients, addressing test results, or conducting medical record review.”
Importantly, there were differences in benefits depending on the characteristics of clinicians and how much they used the AI scribes.
For instance, clinicians who used AI scribes 50% or more of the time spent 21.3 fewer minutes per eight-hour day on electronic health record work, 5.5 fewer minutes on EHR work out of work hours and added one extra patient consultation per week, compared with nonadopters.
They spent 27.3 fewer minutes on documentation, and there were incremental decreases in time spent on documentation commensurate with increases in proportion of AI scribe use.
Primary care clinician adopters gained more than other specialist adopters in some respects, spending 25 fewer minutes per eight-hour day on electronic health record use and 26.9 fewer minutes on documentation than primary care nonadopters. They only cut EHR work out-of-work hours by 3.4 minutes, however.
Female adopters spent 19 fewer minutes on electronic health records, 19.9 fewer minutes on documentation and 6.2 fewer minutes on EHR work outside of work hours, per eight-hour day, than all nonadopters.
Residents who adopted AI scribes spent significantly less time on electronic health record work (113.9 fewer minutes per eight-hour day), documentation (94.40 fewer minutes) and EHR work out of work hours (89.9 minutes) and had 1.4 more weekly patient visits.
“Use of AI scribes among residents is an area of ongoing discussion and study, given the critical nature of documentation for learning and unknown implications for resident learning,” the researchers noted.
The “disproportionately large benefits in reduced documentation time” for residents was of interest to the authors of an accompanying commentary in the same issue of JAMA, written by staff from the research division of US not-for-profit health insurance giant Kaiser Permanente.
They saw it as possibly a harbinger of the way AI could benefit clinicians in the future, even if that was not as evident for the clinicians we have now.
“Health care has already seen the shift to EHR-native clinicians, ie, those trained after the advent of the EHR, who have had minimal interaction with paper records.
“These findings may foreshadow an AI-native generation training alongside readily available, powerful ambient and generative AI tools. Future studies should assess implications of ambient AI scribe use on trainees’ clinical reasoning, documentation quality, skills acquisition, and supervision,” the commentary said.
The US study said that AI scribe adoption increased revenue for institutions, by increasing patient consultations, by US$167.37 per clinician who used it, per month.
But increasing the number of visits was not the only way to determine monetary benefit, the commentary warned.
“[N]arrow definitions of return on investment may obscure other financial benefits, including improved clinician retention, reduced turnover, and the avoidance of costly errors and preventable utilization. If health systems rely on increasing visit volume to justify the cost of ambient AI adoption, they risk squandering the benefits of time savings if that time is simply converted into more visits per clinician, rather than investments in higher-quality care,” they wrote.
Previous studies looked at data from single sites, with varied results, and differences between AI scribe products, the researchers wrote.
“Given their derivation from pooled data from 5 health systems, [our] estimates are less likely to be biased by site-specific implementation details and represent generalizable results regarding the relative time savings associated with clinicians adopting AI scribes in real-world settings,” they wrote.
Related
The study had some limitations, the authors said.
Implementation varied across the five study sites; “thus, estimates reflect averages and assessments of real-world implementation experiences rather than measurements of maximal efficacy of AI scribes”, they wrote.
All of the sites were academic institutions with about 20 visits per week for clinicians – generally much lower than many clinical settings where AI scribes might be used.
It was observational, not a randomised controlled trial, so the characteristics of adopters and nonadopters weren’t perfectly matched. Cofounders coannot be ruled out completely, despite adjustments and models used in the study.
And the costs of products and service delivery are different, so the savings estimate wouldn’t apply to most cases.
The study didn’t measure how AI use affected burnout or the way time was reallocated.
And estimates for the effect of AI scribes on these metrics was only available for five months after adoption, which really wasn’t very long for people to become comfortable with a new technology, and they might have seen greater benefits down the track.
“What is known today about the effectiveness of ambient AI scribes reflects outcomes that are easiest to count: electronic health record (EHR) time, documentation minutes, visit volume, and billing,” said the accompanying editorial.
“While these measures matter, the question is no longer whether ambient AI can save documentation time for clinicians but whether that time is reinvested in ways that measurably improve outcomes and equity for patients.”
The commentators said that although time savings for individual clinicians seemed modest, the aggregated savings could “substantial”.
“These findings reinforce prior literature showing that ambient AI scribes are associated with reduced documentation-related workload… This success has contributed to rapid uptake, with nearly two-thirds of health systems using Epic EHRs also deploying ambient AI,” they wrote.
However, “If ambient AI is becoming a default component of health care delivery, evaluation objectives must evolve accordingly,” they said.
What we needed to be measuring now were the effects on “the core priorities of the quintuple aim—patient experience, population outcomes, and health equity”.
“Without this shift, the field risks optimizing ambient AI for what is easiest to count (EHR workflow metrics), rather than what the quintuple aim demands: improved health, greater equity, reduced clerical burden, and a more financially sustainable and humane care experience.”
Read the full paper here.
Read the commentary here.
