We have built the digital infrastructure and seen the necessity for it but have not yet put it in every Australian's hands or devices. Maybe commercial AI platforms with appropriate governance could be the way.
The launch of ChatGPT Health and Claude for Healthcare in January 2026 marks a watershed moment.
Purpose-built commercial AI health assistants have become the primary way millions of consumers make healthcare decisions. Over 40 million people globally now ask ChatGPT health and wellness questions every single day.
These platforms are displacing internet search as the starting point for health questions, offering conversational, seemingly personalised guidance rather than lists of links.
This isn’t theoretical. Australians are using AI assistants to interpret symptoms, plan treatment, adopt preventative behaviours and interpret test results. Uptake ranges from 9.9% in Australia to 32.6% in the US, and the numbers are climbing fast. For mental health support, the pattern is even more pronounced. Nearly half of AI assistant users report using them for psychological support to address anxiety and depression, with 63.4% reporting improved mental health conditions.
Why millions are choosing AI over their doctor
The potential benefits are substantial.
AI assistants improve access to appropriate services by providing better initial guidance in forms consumers understand. Recent research has found that AI assistants improved engagement with traditionally underserved populations. Another study showed that patient actors rated conversational AI highly on 25 out of 26 dimensions, including perceived politeness, ability to put patients at ease and rapport building.
But significant risks accompany these benefits.
AI assistants fall outside regulatory frameworks for quality. The recent removal of certain health-related summaries from Google AI Overviews following safety concerns exposes the dangers of unmonitored consumer-facing AI. These assistants could intercept consumers who clinically need professional assessment, diverting them from health services that operate within regulatory or accreditation schemes governing safety and quality.
Despite this massive uptake, AI assistants disclaim responsibility in their terms of use. The de facto front door to healthcare is not accountable for the advice it provides.
AI companies acknowledge limitations in their terms of use, stating their tools don’t constitute medical advice. OpenAI limits liability to $100, Google to $500, Anthropic to $100. They position AI assistants as general wellness tools, yet consumers are using them for diagnostic and treatment purposes that would normally demand rigorous review.
Whose values are embedded in your health advice?
Last year I wrote an article with my peer Daniel McInerney, published in NEJM AI, about the benefits and potential hazards of AI powered virtual front doors.
We discussed how AI encodes human values through training and fine-tuning, and how prompting steers recommendations by framing specific perspectives. We considered a 2024 review that highlighted how an identical case of a 14-year-old boy was given to ChatGPT-4, which was prompted to respond from the perspective of a physician, an insurance company and the boy’s parents.
ChatGPT-4 generated three different recommendations: commence hormone therapy, deny it or delay the decision.
As AI becomes ubiquitous in virtual front doors, the values driving triage decisions will reflect system operators’ priorities in ways that are opaque to consumers. Virtual front doors generally don’t provide options for care pathways, but refer consumers into a single pathway: “go to ED”, “see a doctor within 24 hours” or “self-care at home”.
Without safeguards, economic pressures could scale distortion to a system level.
Building trust through evidence and accountability
Realising the benefits while mitigating the risks requires ensuring AI assistants operate to quality standards and with accountability. The core safety question remains: how do we determine when AI advice crosses from helpful guidance into territory requiring clinical oversight?
This boundary-setting challenge cannot be solved by technology alone.
Industry-developed quality benchmarks such as OpenAI’s HealthBench and Stanford’s MedAgentBench provide the first steps in measuring safety and accuracy of AI assistants. They establish baseline performance standards and demonstrate technical capability, although recent research shows the limitations of benchmarks and identifies the need to identify how humans interact with AI assistants and systematically conduct safety testing with diverse, real users.
Application of emerging standards can assist in establishing oversight. Interface standards like Model Context Protocol enable secure integration. Data and interoperability standards like SNOMED and FHIR ensure clinical information flows reliably across systems. Emerging ethical and privacy standards provide guardrails for responsible AI deployment. The highly anticipated AAAi-H 3rd Policy roadmap for safe and effective national implementation of AI in healthcare will update recommendations on how to embed values and clinical governance in this fast-paced environment.
Together, these frameworks address the intended use dilemma and the liability gap, incentivising private developers to submit their tools to oversight that provides the gateway to safe use at scale and clinical data integration.
An Australian vision: AI that works with our health system, not against it
Australia could build its own sovereign AI health assistant, trained on Australian clinical guidelines and governed by Australian health authorities – using an interface like the 1800MEDICARE website and app.
This is a worthy approach and the right approach.
But as hard as it might be to accept it, this approach ignores where Australians are already showing up – millions are already using commercial AI platforms for health queries. Rather than competing with this reality, perhaps we can leverage it.
The emergence of health AI assistants presents an opportunity for governments to set clear boundaries and harness industry innovation. Governments could enable health AI assistants to access consumers’ health records held in public systems, in accordance with consumer consent and conditional on platforms meeting rigorous safety, security and data governance standards.
Critically, governments retain control. Data access can be revoked at any point and penalties applied for recklessness or misuse, providing leverage in negotiations with developers who are already calling for this information to be unlocked.
Imagine an Australian who consents to turn on an API allowing their AI assistant to access their My Health Record, alongside other data streams like Fitbit, Oura, Strava, Woolworths shopping history, Healthdirect self-care advice and hospital records such as those in the NSW Single Digital Patient Record.
The API would not allow the AI assistant to store this information, only process it in the context of the consumer’s prompt. This would bring together multiple sources of disparate data, much of it based on lifestyle factors, complemented with highly reliable structured and SNOMED coded clinical data from the My Health Record and hospital systems. The AI synthesises this information into meaningful insights tailored to that individual Australian.
Setting the guardrails: when AI must hand over to healthcare
Here’s where boundaries matter.
The system must know when a healthcare provider is required and hand off to services with clinical oversight and accountability – smarts that may not be in these tools yet.
It could connect consumers seamlessly with Healthdirect’s digital symptom checker, which is subject to rigorous clinical monitoring and oversight, or make a call to Healthdirect allowing them to speak with a nurse immediately.
Where it is clearer that a higher scope of practice is required or for ongoing care, the system makes a booking with their regular GP (checking first if they have registered with a practice through MyMedicare), or helps them find a GP or telehealth provider such as 13SICK using Healthdirect’s National Health Services Directory.
Australians already have free access to these services – this approach simply meets them where they’re already seeking health information and at no cost to the consumer.
This approach could deliver the best of all worlds – the deep empathy and large user base of commercial AI platforms meets consumers where they are. Multiple sources of health and lifestyle data are synthesised into actionable guidance. The system glides seamlessly into digital channels that have clinical oversight and accountability. The hazards of unregulated AI assistants are mitigated by services the state, territory and Australian governments are already providing, using data governments already hold. This data is finally unlocked and synthesised into meaningful insights that consumers can start to act on every day to prevent the onset of chronic disease and better manage their health.
This is the moment when all the investment we as a country have been making could deliver transformational benefits.
The vision so many of us in the digital health community have looked towards is within our reach.
We have built the digital infrastructure and seen the necessity for it, but have not yet landed it in every Australian’s hands or devices. Maybe commercial AI platforms with appropriate governance could be the way.
Related
Author’s note: use of AI in article preparation
I used Claude Sonnet 4.5 Extended (February 2026 version) while writing this piece. Not just for grammar checks; I used it to organise my thinking, test arguments, challenge my assumptions (including asking it to tell me why I’m wrong), and refine the narrative flow. It pushed me to be clearer and more direct.
But every idea in this article is mine; I take full responsibility for everything written here, with Claude as my thinking partner.
I’m disclosing this because transparency matters, especially when writing about AI governance. If we’re going to advocate for accountability in AI health assistants, we should model it in how we work with these tools ourselves.
Bettina McMahon has worked in the Australian digital health sector since 2009. She is the Chief Executive Officer of Healthdirect Australia and Board Director of SNOMED International, and a former executive at the Australian Digital Health Agency and Board Chair of the Australasian Institute for Digital Health. The provocation in this article does not represent the views of any organisation, but are ideas shaped by working as a public official in technology programs alongside industry since the 1990s, and service on Boards responsible for data standards and informatics.
