Why LLMs still have problems with OCR

Original Article ↗ Hacker News Discussion ↗

Claims that multimodal LLMs like GPT‑4o or Gemini 2.0 “solve” OCR are meeting pushback from people doing large‑scale document extraction, who report subtle but frequent errors, hallucinated digits, and layout confusion—especially with scans, complex tables, and long or messy PDFs. Commenters contrast these failures with traditional OCR pipelines and specialized models that offer explicit confidence scores, bounding boxes, and human‑in‑the‑loop correction, arguing that current vision–language architectures and training regimes are optimized for semantic interpretation rather than exact transcription. Others note that LLM‑based OCR can work impressively well for many everyday images and digital PDFs, but is not yet reliable enough for mission‑critical domains like finance or medicine without additional tooling, verification passes, or hybrid systems.

Hype vs. reality of VLM-based OCR

  • The thread centers on whether multimodal LLMs/VLMs (e.g., GPT‑4o, Gemini 2.0, Claude) “solve” OCR or are fundamentally unreliable compared with traditional OCR pipelines.
  • Several people report impressive, near‑frictionless results on simple, single‑page tasks (screenshots, basic PDFs, grocery lists, product labels, small tables).
  • Others working in large‑scale or high‑stakes document extraction say these success stories don’t generalize to complex layouts, large volumes, or mission‑critical domains (finance, medical, regulatory).

Hallucinations, determinism, and reliability

  • A recurring complaint: VLMs “helpfully” infer missing or unclear text (e.g., truncated grocery items, incomplete recipes, financial rows), which is useful for casual use but unacceptable for production OCR.
  • Unlike classic OCR, which typically surfaces confidence scores and obvious failure modes, VLMs produce fluent text that hides errors and is hard to systematically validate.
  • Some suggest post‑verification passes (“is the image text identical to this text?”) or separate verifier models, but current models reportedly still hallucinate or ignore instructions when scaled.

Architecture and training debates

  • The article’s critique of ViT/CLIP‑style vision stacks (patch size, positional embeddings, “semantic over precise”) is challenged as technically inaccurate or overstated.
  • Counter‑arguments: vision encoders can and do capture fine‑grained text; can output bounding boxes and confidence (e.g., CLIP derivatives, OWLv2, Florence‑style models).
  • Broad agreement that current VLMs are weaker on vision than on text, largely due to training data and benchmarks rather than inherent architectural limits.
  • Some argue better training (synthetic complex documents, RL with strict verifiers, 2D attention) could close the gap; others think hallucination is structurally hard to constrain.

Use cases, scale, and layout complexity

  • People distinguishing “OCR as a step” vs “vision‑based RAG / semantic querying” note that VLMs can excel at high‑level understanding even if raw transcription isn’t perfect.
  • Where high character‑level fidelity is required (financial tables, historical archives, long multi‑page documents, nested/50‑page tables), practitioners report persistent digit drops, misaligned columns, and layout confusion.
  • Reported acceptable error rates differ: some are happy with “99%+” for business use; OCR veterans call that “terrible” compared to traditional pipelines plus human review.

Tools, hybrids, and future directions

  • Multiple traditional or hybrid systems are mentioned: Tesseract + LLM cleanup, PaddleOCR, Surya, Mathpix, Florence‑2, Moondream, and bespoke pipelines focused on bounding boxes and layout.
  • Some believe pure OCR will fade as VLMs subsume it; others argue that specialized OCR/layout engines plus LLMs for higher‑level tasks will coexist for the foreseeable future.