It’s been a big week so we’ll cover some ground: how the medical profession is failing to understand the impact of AI, whether HotDoc misled its client practices, and why telehealth standards developed by the private sector aren’t a great idea.
BOTW stands for Best of the Week, if you were wondering. I almost gave it the acronym WOTW – weirdest of the week – though.
AI is weird. It’s disturbingly weird.
Coming, ready or not
Digital health semantics warrior Tim Blake posted a good example of how weird this week on LinkedIn and he won a few high fives from some pretty influential people in the health services sector for doing it.
Pointing to a paper submitted by researchers from a computer sciences group at Cornell University to open access digital archive arXiv on the effects of AI’s algorithmic sycophancy, Mr Blake commented:
“Rather than hallucination, sycophancy in LLMs (generative AI) is why its use in healthcare is so much more dangerous than we currently understand.
“We are currently conducting a massive social experiment with almost no clinical safety guardrails.”
The arXiv paper argues that “sycophancy poses a unique epistemic risk to how individuals come to see the world: unlike hallucinations that introduce falsehoods, sycophancy distorts reality by returning responses that are biased towards existing beliefs.”
If you’re using AI – we do a bit at The Medical Republic, mainly for research and reflection – you immediately get what these researchers are saying.
AI is very obviously analysing all sorts of emotional aspects of its users, and moulding itself to be just like the AI you see in the movies, a likeable assistant/friend who knows you, what you want and what your sense of humour is, so it can share a joke with you and you actually laugh – that’s when you know you’re in a bit of trouble.
I’d say it’s creepy – it is – but it is done so well it isn’t. Most people using it for work will find themselves often wandering off in a conversation with their little AI helper, which is way off topic: “… what about that episode of The Seven Kingdoms last night, eh?”
One of our best, most experienced and hardened journos, who was adamant she would never touch AI, but has never been the greatest at reading balance sheets, used it to try to understand a particularly complex financial result recently, which normally I would have had to do for them.
Within the next week this journo had given her AI a nickname – “Squishy” – was regularly using it for getting to the bottom of complex issues faster, and was chatting to it in between research sessions for various reasons, a few nothing really to do with work.
That’s algorithmic sycophancy. As much as you know it’s happening and you can rationalise it, you engage with it.
But Mr Blake’s point here might be one of the best being made by senior members of our health services technology literate establishment: put this power in the hands of either patients and professionals, and, because we are dealing with complex medical information, risk, and a lot of emotion, the sycophancy effect is potentially catastrophic.
Mr Blake’s point might be the most important one made so far by the health services side of the fence trying to ward off AI.
There is much more danger than the main warning we’ve been getting so far from healthcare professionals: that you can’t trust the technology because it hallucinates and lacks vital context like that which a GP might have if they’ve been managing a patient continuously for many years.
Telstra Health’s head of product, Danielle Bancroft summed up the above point succinctly in the comment stream.
“The part that often gets missed in this conversation is context.
“Clinical safety isn’t just about whether a model is accurate or persuasive. It’s about whether the information it produces is grounded in governed, contextualised data and embedded within accountable clinical workflows.”
Add algorithmic sycophancy to this obvious core problem, and the odd hallucination and you can understand why a lot of professionals in healthcare are starting to pile on the unique danger AI presents in medicine.
But there’s a very big problem we all have with all of these very good arguments and intention.
AI is coming for patients in a very big way and there is absolutely nothing the professional side of healthcare can do about it, other than to try to keep up with it, and work with what is unfolding.
Of course, we have a fairly big cohort on the professional side of delivering healthcare – and it’s not just doctors, although they likely have the biggest emotional and cultural change coming at them here – that demand that AI have guardrails and be tightly integrated with existing clinical governance.
At least part of the group is trying to defend its ground as the keepers of the “sacred, special, doctor grade” medical information.
There are all sorts of reasons both from a medical point of view, and a system efficacy point of view, that giving patients so much more informational power in the system dynamic can lead to chaos, so the arguments here, though pretty defensive of their power position on a lot of occasions, are still valid.
But as with Mr Blake and Co, and the very obvious issue of “sycophancy”, tell all of this to someone who cares.
Because the mad entrepreneurs building these giant AI engines don’t care one bit, which means patient-side informational power is coming big time via the patient-side AI engines and there “ain’t nutting” anyone can do to stop it.
Some big thinkers in government do seem to be contemplating this problem.
As an example, Bettina McMahon, who runs Healthdirect, is considering the possibility that either ChatGPT Health, or Claude Health, will end up being the most powerful and comprehensive digital front door for patients in Australia.
If that happens, these new AI engines would likely end up doing a lot of the patient information and triage work that Healthdirect is best known for.
And in this scenario, Ms McMahon is worried that without the unique longitudinal data and IP her organisation has built over many years, which forms the basis for good clinical governance for patient decision-making, the AI engines could get a lot of stuff wrong for patients.
So, she is thinking hard about the idea of somehow integrating with one or both of them to get inside the fast-evolving dynamic. A real-life “if you can’t beat ‘em, you had better join them quick and see what you can do” type of strategy.
This is thinking that every healthcare professional should be contemplating now.
All of us special people who care on the inside of running healthcare and medicine can try to guardrail AI all we like. There are no guardrails and its highly unlikely there ever will be on the patient-side AI agents.
They are coming ready or not.
I think we would do very well to get our heads around this dynamic and try to be more ready than not.
10 things I hate about HotDoc
HotDoc’s new PE owners got its first taste of cranky GPs and practice owners this week when it was revealed that it had quietly tried to introduce a new payment option that looked designed to shift patient payments en masse from the practice, upstream to HotDoc itself.
That story is HERE.
By shifting payments for a lot more types of consult, including gap payments for mixed billing practices, HotDoc was effectively trying to insert itself as the major payment layer in patient transactions, something which would not only give the group significantly more revenue by avoiding commission payments to the patient management systems for managing the transaction at the practice, but would also significantly add to the patient data power of the group.
But if you’re not feeling for the disintermediated patient management system vendors, then as a GP or GP owner, it potentially gets worse for you too.
Health&Life principal owner, and professional accounting shit-stirrer, David Dahm, copping a lot of confused incoming from his clients, and watching the GP socials, immediately put out an advisory note for his patients listing 10 reasons his clients should quickly go into their systems and switch off the default setting HotDoc had pushed in a link on its announcement for patients to see the HotDoc new payment option offer.
Then this publication, mistakenly published Dahm’s client advisory note, which we are calling “10 things Dahm hates about HotDoc” and the shit very quickly hit the proverbial fan.
HotDoc sent us a long missive complaining that some (not all, mind you) of Dahm’s advisory note points were either misleading, erroneous or both.
We published HotDoc’s “Six things we hate about David Dahm’s advisory note” table, unpublished the advisory note and rang up Mr Dahm.
Apart from being pretty annoyed that we’d assumed we could publish his client’s private advisory note (sorry about that), Mr Dahm remained just as much down on HotDoc and its new payment option as he was before he saw HotDoc’s rejoinder.
His overall point might be summarised as, pushing the payments upstream, away from the practice and the practice management system, which is what HotDoc was trying to do, had enormous implications for practice business and individual GP tax accounting, both at the state level around payroll tax, and the federal level with the ATO.
Mr Dahm told TMR that his client note was an emergency note in response to direct queries he was getting from his clients, and what he was watching on the GP practice management socials.
“Everything in that note was in response to questions I was getting directly and too questions being raised on the GP socials,” he told us later in the week.
Related
“If HotDoc’s clients are asking those questions, and HotDoc’s position is that I’m assuming a whole lot of things in my note which aren’t true, then I’d suggest they clear all the confusion up with their clients.
“They’re confused … and pretty annoyed. That’s not on me. It’s been created in my view by HotDoc’s attempt to shift a lot of payment options for patients to them, with pretty much no proper explanation to either their practice clients or to the patients themselves.
“The worst thing though? Some of my clients have patients ringing them in tears of confusion thinking that they can’t see their long-term GP anymore, even for bulk billing, because they don’t have a credit card.
“Did HotDoc think any of that through?”
This part of what Mr Dahm was pointing out was confirmed for TMR by calls to a few other practices that were confused by what was going on.
Poorer and disadvantaged patients were calling in confused and distressed. They thought they had to change how they paid to a credit card, which they didn’t have, and which many were embarrassed to admit.
Which might bring us back to HotDoc under new management – private equity management.
Attempting to insert yourself significantly into the patient payment layer so slyly looks like a pretty classic strategic data and money power play.
Whether HotDoc deliberately soft landed the announcement with a two-liner in a normal update to their clients, with a view to hoping the move went through unnoticed largely by the practices and the patient management vendors, or not, it’s a very bad look for the start of the group’s new ownership.
Even before the debacle of Brookfield Asset Management and Healthscope, PE wasn’t exactly a highly trusted owner in healthcare.
HotDoc insinuated in its update that we at TMR were also publishing fake news by saying, in a very roundabout way, that the major patient management vendors were informed of the change.
They may have been, but when we rang them, they all said they weren’t, so whatever HotDoc did, it was ineffective with the PMS vendors as its message to their practice clients was.
Mr Dahm has told us that he has addressed all of the issues raised in the HotDoc correction table with his clients in detail, and that he is still strongly advising them to get into the system and uncheck the default so the patients aren’t made the offer and the payments stay largely with the practice.
HotDoc meanwhile has a few existential problems, and they aren’t all about client trust.
Who are their clients?
The GP practices which currently pay about 90% of their revenues, or the GPs’ patients, who they do need to create much deeper engagement with in hurry, lest the new patient-side AI agents, such as ChatGPT Health, offer a much better patient-side experience and, once they have secured as many or more patients than HotDoc has app downloads, eventually decide to get into the bookings game themselves.
Telehealth standards, or classic industry-side marketing tactic?
This week, a group of private telehealth platforms, in cahoots with a pretty conflicted supposed patient advocacy group, joined forces to ask us all to look into their eyes and believe them that Australia now has national healthcare telehealth standards.
That story is HERE.
It’s probably important to point out that a notable part of the support crew for the announcement in Canberra was Healthdirect, the government agency that runs probably the largest telehealth set-up in the country.
Healthdirect is highly thought of. So many people began asking themselves, what is going on?
Is this a good thing or a bad thing? If it was just a bunch of private telehealth platform providers getting together to make up their own standards, that would probably be seen as a non-starter.
Are these actually standards, and if they aren’t – they aren’t – and they have no regulatory meaning or teeth – they don’t – then why would anyone pay any heed to them anyway?
But Healthdirect is in there cheering from the sidelines of the announcement.
Healthdirect has a good reputation, broad engagement with the patient community, and is wholly government funded.
Is the government endorsing these so-called standards in some low-key way via Healthdirect’s involvement?
Answer: no, but you could certainly be forgiven for thinking that it might, given Healthdirect’s standing in the healthcare community.
Some GPs weren’t buying it, that’s for sure, which is not a good sign for our dedicated new team of standard bearers.
GP Dr Max Mollenkopf wrote on LinkedIn:
“This is confusing and a bit misleading. Despite this post there aren’t actually any national government standards for telehealth. We should probably rename them as the ‘industry-led aspirational standards in the absence of any government movement’.
“This is yet another industry group consortium putting forward what they HOPE will become the national standards,” he continued.
Dr Mollenkopf, as he often does, is making a pretty good point.
These aren’t standards.
You know how the French get all pernickety when we call sparkling wine champagne?
Well, the government really should act the same when a bunch of commercially driven businesses get together to beat the government to a job they should have done, almost certainly with a view to influencing the agenda significantly, and publish something they label as “a national standard”.
It’s highly misleading.
The government or Standards Australia (do they exist anymore?) should go them for being so.
This is not to say that what this group has put up is necessarily a biased or a bad set of guidelines (they are okay and also terrible in various ways, see ahead).
Are they better than nothing? Ms McMahon thinks they are a good set of guidelines for the industry to follow, suggesting perhaps they could be.
But there’s a big credibility problem here for everyone which is why it’s a bit surprising Healthdirect is cheering this initiative on so enthusiastically.
Where do I start?
- That this is almost wholly built and run by the private telehealth providers, not withstanding Healthdirect’s cheering from the sidelines;
- The originator organisation for the idea, Patients Australia, is not really a patient advocacy group, as its name would suggest, but a classic industry-funded lobby group made to look like it is representing patients. In the end Patients Australia is funded mainly by and must therefore answer directly to its funders in some way. Its funders are private commercial groups, not patients.
- We somehow have someone claiming to have built “national standards”, but the reality is, the focus of these private telehealth platform groups, and of the standards as a result, is mainly around patients ringing up on the phone to either get drugs, or medical advice, in Modified Monash Model (MMM) regions 1-3. What happens in MMM 4-7, where you could easily argue, most of the serious and highly impactful telehealth should be contemplated and promoted? The fact is telehealth in MMM 1-3 and 4-7 is almost two entirely different propositions, but these “national” standards are not fit for purpose for the latter when you get under the hood.
- Nearly everyone one in this national standards group has, at one time or another, been the subject of some pretty nefarious and messy operational scandals, often overseen by government regulators: things like selling weight-loss drugs over text to patients without checking if the patient is who they say they are, or their actual physical condition.
- All of these operators have virtually no revenue coming from Medicare so at this point of time are off the grid as far as government oversight on what they really do is concerned.
- There is another industry body, you’d have to say a much dodgier one than this one, led by that great standard bearer of ethics and morality in healthcare delivery, the vertically integrated cannabis sales outfit, Montu. Montu and friends, which is actually called “The Telehealth Sector Working Group”, is trying to do the same thing, albeit, their effort will likely be far more favourable to what everyone now sees as a business model that should not even exist in our healthcare system – vertical integration of cannabis production, GP consulting, prescribing and delivery. If you were even a bit cynical you’d recognise the two groups as that Monty Python sketch from The Life of Brian, The People’s Front of Judea, and The Judean People’s Front. No chance of patient confusion in the future here when they come out with their set of national standards, I guess?
But I’m not going to blame this group of companies, or even their faux patient organisation, Patients Australia, for this potential mess.
It’s been allowed to happen. The government should have stomped all over this before it got any sort of traction.
And someone high up in the department should have warned off Healthdirect from joining in to add some sort of credibility aura to the whole venture.
But this obviously didn’t happen. And we might know why.
When TMR asked the Department of Health, Disability and Ageing what the hell was going on here and what did they think of this, they told us blithely that the standards were developed independently of government as a sector-led initiative.
Very informative thanks, and …
“The Department of Health, Disability and Ageing welcomes efforts that support safe, high-quality virtual care for consumers,” a spokesperson said.
Does the Department really welcome this effort? Has it even looked at it to be able to be making a statement like this?
Because if you look at all the problems with this initiative I’ve listed above, the biggest one being it pretty much misses out on contemplating the most impactful likely area of telehealth into the future, MMM 4-7, you’d surely have to think not.
This idea has way too many holes to be credible, even if there is anything in there that would be good for patients and industry at the same time (it can happen).
But it also fails to address the needs of the patients who need good telehealth policy, standards and technology the most, patients in rural and remote areas.
