Leaders from Oracle Health and Heidi say Australia's regulatory framework needs to catch up with rapidly evolving clinical AI, as questions persist over how potential safety incidents are identified and monitored.
Australia’s AI scribe developers are preparing for tighter medical device regulation, with senior executives from Oracle Health and Heidi predicting that more clinical AI products will fall under the Therapeutic Goods Administration’s regulatory framework.
Speaking at Health Services Daily and The Medical Republic’s Hospitals Summit in Melbourne on Thursday, Oracle Health lead healthcare executive, Associate Professor Angela Ryan said she expected AI scribes would increasingly be classified as medical devices, while Heidi’s Dr Tom Kelly said the uncertainty surrounding regulation was a greater problem for developers than regulation itself.
“I think the TGA is abundantly clear, and I think that it won’t be very long before they all are medical devices,” Professor Ryan told the conference.
This article originally ran on TMR’s sister site, Health Services Daily. TMR readers can sign up for a discounted subscription.
Dr Kelly agreed that tighter regulation was likely and said Heidi would welcome greater certainty.
“It will happen, and it’s genuinely good for the business if it does,” he said. “The problem is the uncertainty.”
He said Heidi was already regulated as a medical device in the UK.
“If it was medical device tomorrow, we are medical device in the UK, and that’s fine,” he said.
The comments followed the TGA’s preliminary review of 12 digital scribe products, released on 16 October, which found that eight had functions likely to meet the legal definition of a medical device.
None of the 12 suppliers had identified their products as medical devices, and none was included in the Australian Register of Therapeutic Goods at the time of the review.
The regulator has declined to identify which eight products were flagged, saying it is continuing discussions with suppliers to determine the regulatory status of particular functions.
The TGA stressed that its review did not involve testing products or assessing their safety, performance or detailed regulatory compliance.
Software used solely for administrative transcription and documentation is generally unlikely to meet the medical device definition, but functions that generate clinical conclusions, recommendations or information capable of influencing diagnosis and treatment may bring products within the regulatory framework.
Some clinical decision support software may qualify for an exemption from ARTG inclusion while remaining subject to medical device regulation.
Safety incidents may be going undetected
Professor Ryan raised concerns that the healthcare system did not yet have adequate mechanisms to identify and measure potential safety incidents associated with AI scribes.
“We don’t actually know if there are issues because they’re not actually being measured,” she said.
“We are from Oracle. We have a scribe product as well, which is now on the market. But no one is actually measuring that.”
She warned against assuming the absence of reported incidents meant the technology was operating without risk.
“I think we’d be kidding ourselves if we think there’s no safety events because they are just not being captured.”
Professor Ryan said large language models were continuing to evolve, but their ability to generate convincing language did not necessarily establish their reliability in every clinical setting.
“I don’t think the LLMs are quite there yet,” she said.
“Producing sort of plausible language is not the same as having a clinical use case in every context.”
Related
Oracle had been working with clinicians to test AI-generated outputs across different healthcare environments, including ambulatory care, inpatient settings and emergency departments.
Professor Ryan said different versions of the same model could produce substantially different outputs.
She warned that errors could be particularly difficult to detect because AI-generated clinical documentation often appeared credible.
“The outputs look really credible. So it is hard to spot where there is something in there unless you’re really looking at it.”
Dr Kelly argued that clinician supervision provided an important safeguard for AI scribes.
“When you’re generating a note, ultimately the clinician is still signing the name, they’re still reviewing the content,” he said.
“I think in practice, when supervised by the clinicians, the harm is actually relatively low.”
However, he distinguished supervised clinical applications from AI systems operating without clinician oversight or providing information directly to patients.
“It’s when it becomes unsupervised and when it’s in the hands of patients that that’s, it’s like a different bar, in my opinion.”
New approaches to monitoring AI safety
Professor Ryan said Oracle was working with researchers examining whether indirect measures of clinician behaviour could help identify potential problems with AI-generated documentation.
The research involved the concept of “technomarkers” – indicators that might reveal safety concerns without necessarily identifying a clinical incident directly.
One possible measure was the time clinicians spent correcting or editing AI-generated notes.
“When a clinician is using ambient scribe, is there some sort of [measure of] how long is it taking them to edit it?” she said.
“Does that mean there are more hallucinations and so on and so forth?”
She said it was important to evaluate the technology against clinical outcomes rather than focusing exclusively on time savings.
“It’s great to say we’ve saved five minutes or 10 minutes or whatever, but is there an actual other outcome as a consequence? What about the safety?”
Dr Kelly also raised the question of post-market surveillance, arguing that regulators and developers needed ways to distinguish potentially dangerous AI hallucinations from more obvious errors.
“Unless you have post-market surveillance monitoring for dangerous hallucinations, which are different from just obvious hallucinations, and you actually submit that to a regulator, how can you prove that the tool is safe?” he said.
Regulation as AI moves beyond documentation
The regulatory discussion comes as developers increasingly move beyond conventional AI scribes towards systems capable of interacting directly with clinical software.
Dr Kelly described Heidi’s work on AI agents that could navigate existing software interfaces, retrieve information and enter data into healthcare systems.
He argued that such technology offered a potential solution to the fragmentation of clinical information across incompatible software platforms.
But Professor Ryan cautioned that AI systems interacting with electronic medical records required appropriate controls.
“We need to have a sanctioned interface between AI and EPR,” she said.
She said healthcare could not simply allow AI systems unrestricted access to clinical software because the technology was capable of automating tasks.
“The risk of AI is AI at scale,” she said.
The TGA’s recent review similarly highlighted the growing difficulty of distinguishing administrative software from clinical decision support as developers introduce new functions.
The regulator found that some suppliers appeared not to have considered whether their products were subject to medical device requirements, while others relied on disclaimers stating that their software was not intended for diagnosis or treatment.
The TGA said such disclaimers were not sufficient to determine whether software was legally a medical device.
The Medical Software Industry Association has defended the regulator’s preliminary approach but called for clearer guidance on regulatory boundaries, exemptions, validation requirements and clinician oversight.
MSIA CEO Emma Hossack told Health Services Daily last week that the industry needed greater certainty about which functions triggered medical device requirements.
“We need more specificity around boundaries; it is a complex area,” she said. “That is why we are meeting with TGA to work through the requirements.”
The MSIA met the TGA on 26 October to discuss the review and possible regulatory responses.
Speaking in a later session at today’s summit Ms Hossack said the TGA was “certainly trying”.
“The next stage is more consultation from us, and they’re open to that,” she said of Monday’s meeting.
“Everyone was really forthcoming. The next step is for us as an industry broadly, and that means everyone, to actually explain the problems to [the TGA] simply, and ask them to start lobbying to the Commonwealth government for some funding so that they can start some of their enforcement where it’s required because they don’t have that.
“Secondly, [we need] to actually get the education and the training and awareness out there because we’ve got a lot of laws out there, well and truly over 100, that are currently applicable for this kind of technology.
“I’d hate to see us suddenly going, ‘oh, we need a new AI law’,” she said.
“Let’s look at what’s there, and if there are gaps, and I think that’s what [the TGA is] doing now. It was a pretty collegiate meeting.”
The TGA has said it will use its findings to guide further compliance activities, education and clarification of regulatory obligations, while seeking to avoid unnecessary disruption to clinical care.
