LLMs approach expert-level clinical knowledge and reasoning in ophthalmology

Methodology and Validity of the Study

• Several commenters question the study’s design: the exam questions came from a standard ophthalmology textbook, not newly written items.
• People note that such textbooks are likely in common LLM training sets (via LibGen, “Books2”, or direct scanning), so test leakage is considered very plausible.
• Because of this, some argue the paper mainly tests memorization rather than reasoning, and might be “bunk” without stronger guarantees about training data and novel questions.

Limits of Exam-Based Benchmarks

• Multiple comments stress that passing written exams is not equivalent to real clinical practice.
• Standardized exams often emphasize recall and pattern-matching over true diagnostic reasoning, communication, or dealing with uncertainty.
• Some link this to broader critiques: LLMs exposing how professional exams and medical education overweight rote knowledge rather than real-world skills.

Capabilities and Risks of LLMs in Medicine

• The reported performance (GPT-4 near expert level on questions) aligns with many users’ personal experiences of LLMs being useful for symptom triage, blood test interpretation, or differential suggestions.
• Others, including at least one clinician, say LLM outputs look impressive to laypeople but break down on nuanced differential ranking and management decisions, sometimes suggesting dangerous treatments.
• Hallucinations are seen as a central blocker for direct medical use. Proposals include multiple sampling and self-consistency, but commenters note lack of solid evidence these fully solve the problem.

Broader Reflections on Healthcare Quality

• The thread contains many anecdotes of misdiagnosis, delayed diagnosis, and perceived superficial thinking by doctors, fueling optimism that AI tools could eventually outperform average practitioners.
• Others attribute problems to systemic constraints: 10-minute visits, per-visit payment models, long and expensive training, fragmented records.
• There are calls for better outcome-based metrics for physicians and recognition that LLMs’ exam success also highlights weaknesses in how clinicians are trained and evaluated.

Attitudes Toward Future Models

• Some are highly bullish that future models (e.g., a hypothetical GPT-5) will surpass top diagnosticians and be widely accessible.
• Others are skeptical, citing possible performance plateaus and unresolved safety, ethics, and reliability issues.