1300 166 138 Tell us what you need
News / Health AI

Harvard trial: AI spotted illness more often than doctors

By Monika Tantau • Published 7 months ago
Harvard trial AI emergency room triage diagnosis study at Beth Israel Deaconess

Key takeaways

• A Harvard Medical School trial published in Science tested OpenAI's o1 against ER doctors at Beth Israel Deaconess
• The AI diagnosed 67% of Boston cases correctly from triage notes alone, against 50-55% for the human pairs
• On longer-term treatment plans, o1 scored 89% versus 34% for 46 doctors using search engines
• Authors framed AI as a triadic care model partner, not a replacement, and flagged a missing accountability framework

Most Read

A Harvard Medical School trial published in Science found that OpenAI’s o1 reasoning model identified the right diagnosis in 67% of emergency department triage cases at Boston’s Beth Israel Deaconess Medical Center, against 50-55% for the human doctor pairs the AI was tested against, as reported by The Guardian.

The result is one of the cleanest signals yet that frontier AI can match — and at the most data-starved moment, beat — clinical reasoning from trained physicians. For Australian healthcare operators weighing AI decision-support, the study sharpens questions about deployment, governance and accountability that have been pending since the TGA opened its Software as a Medical Device pathway.

What the Harvard team tested

The trial, led by Arjun Manrai of Harvard Medical School and Adam Rodman at Beth Israel Deaconess with Stanford collaborators, set OpenAI’s o1 against pairs of expert physicians on 76 real ER patients, as Harvard Magazine noted. Each side received the same standard electronic health record — vital signs, demographics and a few sentences of nursing notes — and was asked to produce a diagnosis.

At the initial triage moment, when information is thinnest, o1 named the exact or very close diagnosis 67% of the time, against 50% and 55% for the two doctor pairs. When more clinical detail was added, o1 reached 82% accuracy and the doctors 70-79% — a gap the authors said was not statistically significant.

Beyond the snap diagnosis

A second part of the study tested longer-term clinical management. Forty-six doctors were given five complex case studies and asked to plan treatments — antibiotic regimes, end-of-life care — using whatever resources they normally would, including search engines. The AI scored 89% against the doctors’ 34%.

The authors flagged the limits clearly. The model only saw text. It never read a patient’s facial distress, body language or skin colour, all of which feed clinical judgement. In one case highlighted in the paper, a patient with worsening symptoms after a lung clot baffled doctors who assumed anti-coagulants had failed; o1 caught a lupus history in the record and inferred lung inflammation as the real cause.

Why Australian healthcare teams should pay attention

Australia’s emergency departments are operating under chronic load — the National Emergency Access Target has not been hit at the national level for years, and the Australasian Triage Scale is sometimes applied by overworked staff with limited information at the door. A model that performs well on text-only triage is exactly the kind of tool a busy ED could integrate as a second-pair-of-eyes assistant.

The regulatory path is also clearer than it was 18 months ago. The TGA now treats clinical AI as Software as a Medical Device, AHPRA has issued specific guidance for registered practitioners, and several state health departments are running governed pilots. Vendors building conversational AI assistants for clinical settings will need to align with those frameworks before any pilot becomes routine.

Accountability gap in AI-assisted clinical decisions for Australian healthcare

The accountability gap

The biggest open question is liability. ‘There is not a formal framework right now for accountability,’ Rodman said — a problem Australia shares. Existing professional indemnity arrangements were not written with model error in mind, and AHPRA has flagged that registered practitioners remain responsible for any AI-assisted decision.

There is also a behavioural risk. Wei Xing of the University of Sheffield warned that doctors may unconsciously defer to AI answers rather than reason independently — a dynamic that grows as systems become routine. Hospital workflow design and document automation for clinical records will need to keep that human-in-the-loop pressure visible.

A second opinion, not a replacement

Manrai called the results a ‘really profound change in technology that will reshape medicine’ but stopped well short of replacement language. Rodman described the future as a ‘triadic care model — the doctor, the patient, and an artificial intelligence system’. The Edinburgh University team that reviewed the work called o1 ‘a useful second-opinion tool’.

For Australian health services, the practical question over the next 12 months will be less whether AI can diagnose, and more which deployments earn the trust of patients, regulators and the clinicians who carry the legal risk.

Scroll to Top