Vision language models are blind
A new paper argues that state-of-the-art vision-language models like GPT‑4o, Gemini 1.5, and Claude 3.5 fail at very simple visual tasks—such as counting line intersections or judging whether two circles overlap—despite impressive benchmark scores and marketing claims about “seeing” and helping blind users. Commenters debate whether these failures are “embarrassing” or expected given current architectures, pointing to heavy image compression, tokenization limits, and gaps in low-level geometric reasoning as likely causes. Many see the work as a useful stress test that exposes narrow training and generalization limits, while noting that VLMs remain highly effective for real-world tasks like OCR, photography assistance, and accessibility, albeit with clear constraints.
Overall reaction to the paper
- Many find the failure cases “shocking” given claims that VLMs can “understand” images, guide the blind, or tutor children.
- Others argue the results are not embarrassing for the models but for humans who overinterpret or overmarket them.
- Several commenters see the title “Vision language models are blind” as hyperbolic or clickbait.
Observed strengths vs weaknesses
- Strong performance reported on:
- OCR and handwriting recognition (including non-Latin scripts).
- Real‑world photos: navigation help for low‑vision users, identifying hardware issues, gardening advice.
- Very poor or inconsistent performance on:
- Counting line intersections, overlapping/touching circles.
- Counting shapes in logos and grids.
- Following paths in mazes, spatial relationships, reading circled letters, calendar-style highlights.
- Some small open models and specific prompting styles appear to do better on selected examples, suggesting sensitivity to prompts and setups.
Technical explanations discussed
- Images are heavily compressed into tokens/embeddings (e.g., patches, CLIP-like encoders), losing fine-grained spatial detail.
- Embeddings are not trained for faithful reconstruction; visually different images can map to similar vectors.
- VLMs seem optimized for tasks with abundant training data (captions, OCR) and weak at low-level geometry, counting, and precise spatial reasoning.
- Some note that models can redraw images reasonably but still fail logical questions about them, implying a reasoning gap, not just “bad eyes.”
Evaluation, training, and generalization
- Debate over whether these synthetic tasks are “toy tricks” or important evidence of non-general reasoning.
- Some say failures could be fixed by targeted synthetic training; others warn that patching benchmarks doesn’t address underlying generalization.
- Counting and spatial relations are known weak spots; auxiliary methods (segmentation, object detection, “set of marks”) can help.
Implications and ethics
- Concern that marketing for low-vision assistance and “general” vision understanding overstates reliability; calls for stronger safeguards and clearer limitations.
- Others counter that despite imperfections, current systems are already practically useful and often better than prior tools.