Prediction models based on inauthentic or unreliable datasets should never be used... yet a new study has found that they are.
Two datasets in a popular online repository informed 125 peer-reviewed clinical prediction models across 32 countries. But a new study published in BMC Medicine found they had no verifiable provenance.
Three of those models were likely applied in clinical practice, and one was even cited in a medical device patent.
A further 11 models were built into computer- or phone-based tools with a user interface, and two of these – both stroke risk calculators – are still publicly accessible online today.
Practical recommendations were made in 67% of stroke articles and 80% of diabetes prediction articles, researchers found.
“Data that is not fit for purpose can place patients at increased risk of harm as poor predictions will lead to patients being denied necessary treatments or receiving unnecessary treatments,” the BMC Medicine report read.
Kaggle, a popular data repository, hosts over 550,000 datasets and 28 million users.
Using nine data-provenance items from the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis and Artificial Intelligence (TRIPOD+AI), researchers found no information on when, where, why or how these datasets were collected.
Clinical prediction models assist clinicians in diagnosing patients or making prognoses, influencing treatment plans and subsequent health outcomes.
Related
Journals such as Scientific Reports and the PLOS have introduced policies requiring high data quality standards and rejecting fast-churn research – yet PLOS itself published at least one of the clinical prediction models.
Verifying data quality, then, remains difficult when repositories or publishers lack a mandatory process for reporting data provenance.
These findings add to broader concerns about the integrity of Australia’s health data infrastructure, just a day after the AIHW’s Australia’s Health 2026 report highlighted persistent data silos and gaps.
BMC Medicine study co-author Alexander Gibson, a PhD candidate at Queensland University of Technology, told The Medical Republic the issue was twofold – the pressures of fast-churn research and data repositories’ vested interests in housing expansive data.
“Data collection is a very time- and resource-intensive process, so if you can find data that’s already available, it speeds up the research process dramatically,” Mr Gibson said.
He told TMR his team of researchers raised the synthetic data issue with Kaggle, who quickly “washed their hands” of the concern, saying the datasets didn’t breach any policies and were legitimate.
While none of the clinical prediction model studies was cited in policy documents, the researchers said the true concern remains determining the extent of the global issue.
“We are doing a follow-up study at the moment to look for the prevalence of how many prediction models are using datasets with unknown provenance,” Mr Gibson said.
MDCalc & Offcall founder Graham Walker, whose company builds clinical decision tools, wrote on LinkedIn that “dataset slop” reflected a failure at every gatekeeping stage – from modellers who didn’t check the data to review-article authors who failed to check reviewers.
“Each layer assumed the previous one had done the work. Nobody did,” he wrote.
“‘Which model is best?’ is no longer the only question. Now we have to ask: which of these are even allowed to count as medicine?”
The chain doesn’t end here but also extends to GPs who may not have time to assess the validity of the prediction model, Mr Gibson said.
A recent BMJ scoping review of international clinical guideline development documents for prediction models found a lack of available guidelines.
“Legislation and the regulation of these tools are lagging behind how fast [prediction models] are moving,” Mr Gibson said.
“We encourage all journals to include mandatory data availability as described in TRIPOD+AI guidelines for clinical prediction models,” the report read.
Mr Gibson encouraged GPs not to assume published research is error-free and said pre-registering and pre-printing studies before publication makes it harder for poor-quality data to go undetected.
