OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

Original Article ↗ Hacker News Discussion ↗

An AI model from OpenAI reportedly outperformed emergency room doctors in diagnosing cases from text-based medical records, correctly identifying conditions about two-thirds of the time versus roughly half for physicians. Commenters question how meaningful this is, noting that doctors were handicapped by being limited to nurse notes and vignettes rather than real patient encounters, and that benchmark design, data leakage, and model hallucinations can all skew results. Many see near‑term value in AI as a triage aid or second opinion, but stress unresolved issues around bias, liability, real‑world validation, and the irreplaceable role of human clinicians in examining and communicating with patients.

Study design & limitations

  • Many argue the trial is closer to a “paper quiz” than real ER work: AI and doctors saw text from electronic records and nurse notes, not real patients.
  • Doctors were forced to diagnose from notes alone, which they rarely do in practice; physical exam, conversation, and observation were excluded.
  • When both AI and humans had fuller case details, the performance gap shrank and became statistically insignificant, weakening “AI beats doctors” claims.
  • Some note the study uses older models and vignette-style cases, which are useful early steps but far from real-world validation.

Comparisons with other AI-medical studies

  • A recent chest x‑ray benchmark is cited where an AI outperformed radiologists even without seeing images, highlighting how flawed benchmarks can be.
  • Another study with “ChatGPT Health” reportedly mis-triaged about half of emergency cases, showing inconsistency across setups and models.

Human vs AI capabilities

  • Supporters think diagnosis is largely pattern recognition over vast knowledge; specialized medical models will likely surpass most doctors over time.
  • Skeptics emphasize:
    • Physical exam, nuanced history-taking, and detecting deceit or missing info.
    • Judgment under uncertainty, and knowing when to say “I don’t know, we need more tests.”
    • Emotional presence during crises (e.g., cancer diagnoses) as fundamentally human.

Bias, trust, and patient experiences

  • Multiple anecdotes:
    • Missed or delayed diagnoses by human doctors, especially for women and complex or rare conditions.
    • Others report LLMs helping identify conditions (e.g., long Covid, MCAS) or interpret labs better than rushed clinicians.
    • Some had AI completely miss serious issues (e.g., hip problems on x‑ray), reinforcing caution.

System-level incentives and risks

  • Concerns that:
    • AI may be optimized for liability or billing, not patient welfare.
    • Metric-driven use (e.g., ranking doctors against AI) could lead to gaming, overreliance, and eventual de-skilling.
    • Insurance and private equity may use AI to cut costs or deny care.

Proposed roles for AI

  • Common middle-ground view: AI as:
    • Triage aid, second opinion, and guideline-following checker.
    • Research assistant and note-taker.
    • Tool that should augment, not replace, accountable human clinicians.