In a recent study published in PLOS Digital Health, a group of researchers investigated whether physicians could over-ride erroneous artificial intelligence (AI)-based patient classifications when they had access to relevant treatment-outcome information.
But the physicians continued to follow AI guidance, even as treatment outcomes showed the classifications were incorrect, reports News-Medical.net
They had to determine whether to administer a fictitious drug to individuals incorrectly classified by an AI system as more or less sensitive to the treatment. In the second experiment, the treatment served as a laboratory model of pseudomedicine, presented as potentially useful but not proven effective and used primarily to study perceptions of treatment effectiveness.
The participants generally trusted the AI classification to inform clinical judgments and struggled to use patient outcomes to correct errors, even when relevant information was available. The drug was completely ineffective in the second experiment, yet participants generally judged it to be effective.
These findings suggest that physicians should critically evaluate AI-generated patient classifications, although the simplified online experiments do not directly establish how physicians would behave in everyday clinical practice.
The study authors, from Spain’s University of the Basque Country, University of Granada, and University of Deusto, wrote that with a continuous shift toward digital systems and computerised workflows, AI-based models are increasingly used in clinical practice. These models can assist healthcare professionals (HCPs) in classifying patients and estimating their disease risk.
AI algorithms can also predict treatment response, potentially guiding resource allocation to improve care and reduce the burden on the healthcare system.
However, these models may draw inferences from outdated, incorrect, or incomplete data. The models may be trained on homogeneous data, yet they may need to be applied to diverse populations. As a result, AI-based estimates may be inaccurate. HCPs are generally expected to recognise and correct AI errors, particularly when critical information is available.
How they did it
In the present study, researchers investigated whether HCPs can identify and override AI errors: they conducted two experiments with separate final samples of 105 and 118 participants, yielding a combined sample of 223 self-reported physicians recruited via the Prolific survey platform.
The participants spoke fluent English and self-reported as physicians practicing in healthcare settings. They also reported their specialty and years of work experience. In Experiment 1, they rated the extent to which they perceived the AI system as reliable. They also completed the General Attitudes toward AI Scale (GAAIS), which measured their attitudes toward AI use. These measures were not included in Experiment 2.
In each experiment, they interacted with an AI system that falsely classified 60 fictitious patients in the less sensitive (n=30) and more sensitive (n=30) groups for an investigational drug used to manage a rare disease known as Lyndsay syndrome.
Despite being placed into different AI-defined categories, patients responded similarly to treatment within each experiment. In Experiment 1, seven in 10 patients recovered after receiving the drug, compared with two in 10 who recovered without treatment.
In Experiment 2, seven of 10 patients recovered whether or not they received the drug, indicating that the treatment was completely ineffective.
The participating physicians decided whether to provide the drug to these patients and subsequently received immediate patient-level feedback indicating whether each patient had recovered.
Across the trials, these outcomes allowed participants to infer that both AI-classified groups responded similarly and that the classification was therefore incorrect. In the initial experiment, the drug was moderately effective for both patient types.
In the following experiment, the drug was ineffective in both groups.
Results
Participants had a mean age of 38.6 in Experiment 1 and 36.6 in Experiment 2. Their mean professional experience was 13.1 and 10.6 years, respectively. The most common specialties in Experiment 1 were General Medicine and Paediatrics, and in Experiment 2, General Medicine and Internal Medicine.
In Experiment 1, participants generally rated the AI system as fairly reliable, with a mean score of 3.65 out of 5. The average positive attitude score was 3.7, indicating generally favourable views of AI.
While participants who viewed AI more positively also tended to rate it as more reliable, having concerns about AI did not necessarily make them view it as less reliable.
In both experiments, the physicians largely trusted the AI classification, despite receiving patient outcomes that could have been used to challenge it. They found it difficult to update their judgments based on patient-level evidence, which consistently showed the same treatment response pattern across both AI-classified groups.
The participants appeared not to fully integrate the patient recovery outcomes needed to recognise that the AI classification was inaccurate.
In Experiment 1, participants administered the treatment to 89.3% of patients classified as highly sensitive, compared with 55.7% of those classified as lowly sensitive. In Experiment 2, they administered the ineffective treatment to 78.3% of patients in one group and 36.8% in the other.
Participating physicians incorrectly judged that the drug would be more effective among individuals classified as being more sensitive. In fact, in the second experiment, they failed to recognise that the drug was ineffective, raising concerns about patient safety if similar patterns occur in clinical practice.
However, the study used a highly controlled fictitious task and did not test decisions made in real clinical environments.
Conclusions
In Experiment 2, the findings demonstrate that participants generally over-estimated the effectiveness of a treatment that was, in fact, completely ineffective. They did so despite receiving outcome information from each patient. These findings suggest that erroneous AI-based classification models can influence physicians’ treatment decisions and effectiveness judgments under experimentally controlled conditions.
HCPs should therefore critically evaluate such systems as AI use in healthcare expands.
The findings should nevertheless be interpreted cautiously. The participants’ professional status was self-reported, and the experiments involved a fictitious disease, a fictitious treatment, limited patient information, and a binary treatment decision.
The study also did not compare AI classifications with identical recommendations from a human expert or a conventional clinical system.
In further studies, research efforts should investigate whether automation bias, confirmation bias, and causal illusions contributed to the findings. The authors also noted that participants may have deferred to an apparently authoritative classification system because they lacked prior knowledge about the fictitious disease and treatment.
The experiments could not fully distinguish cognitive bias from this potentially adaptive response to uncertainty.
Although AI use is expanding in clinical practice, such systems should be used cautiously and judiciously by HCPs as adjuncts, not as substitutes for clinical decision-making in real-world scenarios. While AI systems can analyse large volumes of multidimensional data, reviewing relevant clinical information using human reasoning and critical thinking remains imperative for patient care.
Study details
Doctors vs. Algorithms: Physicians, too, struggle to learn from evidence that contradicts AI suggestions
Aranzazu Vinas, Fernando Blanco, Helena Matute.
Published in PLOS Digital Health on 9 July 2026
Abstract
Despite their widespread adoption, Artificial Intelligence-based Patient Classification Systems sometimes rely on incorrect, outdated, or incomplete data, which can lead to inaccurate outputs. Nevertheless, health professionals are expected to override these errors, at least when they have access to critical information. To test this, we conducted two experiments in which professional physicians interacted with an Artificial Intelligence system that incorrectly classified fictitious patients as either highly or lowly sensitive to a treatment. The physicians administered the treatment to a series of fictitious patients and received feedback that was useful for learning that the patient classification was incorrect and that all patients were equally sensitive to the treatment. We ran two experiments: in Experiment 1, the medicine showed medium effectiveness for both types of patients, while in Experiment 2, the treatment was completely ineffective for both types of patients. The results showed that, in the two experiments, physicians generally trusted the AI-based patient classification and struggled to learn from the evidence. Furthermore, in Experiment 2, they failed to realise that the treatment was ineffective. Our findings have important implications for healthcare professionals and patients, underscoring the need to critically evaluate Patient Classification Systems.
See more from MedicalBrief archives:
AI beats doctors in Harvard emergency triage diagnosis trial
AI chatbots outstrip doctors in diagnoses – US randomised study
ChatGPT diagnoses child’s illness after 17 doctors fail