How Does GPT-4o Encode Images?

Original Article ↗ Hacker News Discussion ↗

How GPT‑4o internally represents images is prompting speculation about convolutional encoders, vision transformers, and VQ‑VAE–style “image tokens,” especially given OpenAI’s claim that a single image counts for roughly 170 tokens yet can yield far more text on output. Commenters share practical experiences with image-based OCR, noting both impressive performance and sudden hallucinations on large or complex PDFs, and debate whether OpenAI is augmenting the model with separate OCR systems such as Tesseract. The thread underscores frustration with sparse documentation, concern over reliability for tasks like data journalism, and a broader wish for modern, open‑source OCR and clearer guidance on multimodal model limits.

Experiments on GPT‑4o Image Encoding

  • Commenters like the 7×7 colored‑shape grid test; similar experiments with 512×512 images found that an “85‑token” image can yield more text than 85 text tokens.
  • People confirm the article’s observation that models can extract >170 text tokens worth of information from a single image and see this as a potential context‑window optimization.

How Images Might Be Tokenized

  • Several hypotheses: CNN pyramid with 13×13 feature tiles, ViT‑style encoders, or VQGAN/VQVAE compressing 512×512 into ~13×13 image tokens.
  • Others suggest a separate vision encoder that projects tiles into the same embedding space as text tokens (CLIP/LLava‑style).
  • Debate over whether image “170 tokens” is real tokens vs an accounting approximation; most assume they do become tokens so attention can work.

Image Generation and Multimodality

  • Some argue GPT‑4o must have native image tokens because its claimed image‑generation and fine control surpass what prompting a separate text‑to‑image model could do.
  • Others note current ChatGPT behavior still looks like it’s calling DALL·E; there’s disagreement and no definitive proof in the thread.

OCR Performance and Failure Modes

  • Multiple reports of excellent OCR on single pages but severe hallucinations on large or multi‑page images, likely due to downscaling/tiling limits.
  • Users respond by splitting PDFs into page images and feeding them separately; some are building pipelines around this.

Documentation, Resizing, and Tiling Concerns

  • Strong frustration at limited, vague documentation about image handling, especially resizing thresholds, tiling behavior, and how cross‑tile content is treated.
  • OpenAI’s stated low/high‑resolution modes (e.g., 1024 vs 2048 with tiling) help, but many practical questions remain “unclear.”

Use of External OCR and Tesseract Debate

  • One claim in the article that Tesseract might be in the loop is widely doubted; commenters cite Tesseract’s poor accuracy on handwriting, distortion, and complex layouts.
  • Others say combining a rough OCR output with an LLM can improve overall OCR; one anecdote mentions an internal error referencing a Tesseract script.
  • Some expect any OCR component to be a modern, in‑house model if used at all.

Tokens, Embeddings, and Future Directions

  • Discussion clarifies: text tokens use a lookup table for input embeddings, but images can be mapped by any learned encoder.
  • Debate on whether image tokens share the same vocabulary as text, or are marked by “mode” tokens.
  • Several speculate that future systems may move from discrete tokens toward more flexible, variable‑length embeddings.

OCR Tools and Ecosystem

  • Many note that dedicated cloud OCR APIs outperform open‑source tools and current vision LLMs on reliability and structured extraction, especially for tables.
  • Open‑source alternatives mentioned as better than Tesseract include PaddleOCR, SuryaOCR, Doctr, and some multimodal LLMs; experiences vary from “disappointing” to “remarkably good.”
  • There is a strong call for a modern, open‑source, non‑LLM OCR system, including for handwriting.

Hallucinations, Reliability, and Use Cases

  • Commenters stress that image LLMs can confidently hallucinate plausible but false content, making them risky for data journalism or high‑stakes workflows without verification.
  • Suggested stance: treat them like fallible human workers; design processes that detect or tolerate errors, or avoid them where that’s impossible.

Cost, Arbitrage, and Practical Workarounds

  • The article’s point that sending text as an image can be cheaper in tokens prompts talk of “arbitrage” by API wrappers, though added image‑generation overhead might erase benefits.
  • Some are curious whether response time scales with textual content inside the image, which could indicate hidden OCR steps, but this remains untested and unclear.