GenAI and erroneous medical references

Original Article ↗ Hacker News Discussion ↗

Large language models are increasingly used for medical queries and even diagnostic support, but their inability to reliably cite and substantiate medical claims is raising serious safety and accountability concerns. Commenters contrast LLM performance with real-world medicine—where doctors also make errors but operate within regulated, guideline-driven systems—and argue that current GenAI tools should be limited to narrow, assistive roles (like structured data extraction or RAG over vetted corpora) rather than standalone clinical decision-makers. The thread also highlights how promptable, authoritative-sounding outputs can reinforce patient misinformation, the importance of high-quality retrieval and evaluation layers, and the regulatory gap between certified medical AI tools and general-purpose chatbots that clinicians are already informally using.

Role of LLMs in Clinical Practice

  • Safest current use suggested: extract entities (symptoms, signs, test results) and feed into transparent clinical scores (e.g., Wells, Centor), not direct diagnosis.
  • Many expect future explainable multi-diagnosis models, but agree none exist yet.
  • Several see LLMs as “expert assistants” that speed up reference lookup and synthesis for trained clinicians, not as standalone tools.

Comparison with Human Doctors

  • Debate on whether LLMs’ inability to substantiate claims is worse than doctors, who rarely cite on the spot and often follow guideline algorithms/flowcharts.
  • Some point to large estimates of medical errors as a reason to try AI; others challenge those statistics and note many errors are systemic, which LLMs don’t fix.
  • View that difficulty of medicine is either a reason to avoid untrusted tools or a reason to welcome better ones.

Diagnostic Accuracy and Anecdotes

  • Cited studies claim GPT‑4 can match or beat experts on specific diagnostic tasks, but others note training leakage, narrow scenarios, and authors’ own cautions.
  • Some anecdotes: GPT‑4 correctly diagnosing missed conditions and providing effective emotional support; others warn against overgeneralizing.
  • Skeptics emphasize that LLMs can’t physically examine patients and that any “superhuman” claim is extraordinary.

Citations, RAG, and Hallucinations

  • The discussed study tests GPT‑4’s web-browsing (with Bing) and finds ~30% of statements unsupported by provided sources; some call this more an indictment of that RAG setup than of LLMs generally.
  • Multiple posters note that naïve RAG is hard; document curation and low-level tuning matter a lot.
  • Examples of specialized systems (e.g., drug–drug interaction tools, large medical-model startups) reportedly achieving very low or zero error in constrained domains.
  • Suggestions include using a second LLM to verify each claim-source pair; others point out that verifier models can also hallucinate.

Patient Behavior and Risk Amplification

  • Concern that LLMs act as mirrors: with leading prompts, they reinforce user biases, especially for hypochondriacs or supplement enthusiasts.
  • Some argue classic web search can do the same, but others say LLMs are more fluent, faster, and more convincing than scattered search results.
  • Worry that plausible, well‑worded but unsupported explanations make it hard for non‑experts to distinguish fact from fiction.

Regulation, Liability, and Fairness

  • Startups offering medical AI report heavy certification burdens; frustration that generic LLMs are used in care without similar regulation.
  • One side argues liability lies with clinicians who ignore warnings and misuse non‑medical tools; the other sees this as an unfair competitive advantage for big LLM providers.

Conservatism vs Progress in Medical Knowledge

  • Some users find GPT‑4 now over‑constrained to mainstream sources (e.g., Mayo, Cleveland Clinic) and resistant to citing less orthodox or early‑stage research.
  • They argue this may entrench current dogma and slow scientific progress, preferring models that surface a wider range of evidence and let users judge.
  • Others justify strong guardrails as necessary for reducing harm, legal risk, and public backlash.

Emerging and Proposed Use Cases

  • Beyond diagnosis, discussed uses include: drug interaction checking, mining large patient event histories, and “resource routers” (e.g., a cancer bot that links to grants, guidelines, and support resources rather than answering medically).
  • General consensus: LLMs will likely become one component in larger medical toolchains, not a replacement for clinicians.