No "Zero-Shot" Without Exponential Data

Original Article ↗ Hacker News Discussion ↗

Paper’s Claim and Scope

  • Thread centers on a paper arguing “zero-shot” generalization in multimodal models is largely illusory: rare concepts require exponentially more data for linear gains in performance.
  • Authors test 34 models across ~3M–400M pretraining samples on 4,000+ concepts and release a long‑tail benchmark (“Let it Wag!”).
  • Several commenters emphasize: results are on CLIP-like image–text models, not LLMs; some think this distinction is crucial, others think it may generalize.

Zero‑Shot and Evaluation Nuances

  • Multiple comments note that much “zero-shot” work is really just held‑out test data with overlapping classes, not truly unseen concepts.
  • Example: training on many animal images and then “zero‑shot” testing on ImageNet labels is questioned; true zero‑shot would be training on horses and testing on zebras.
  • Some argue the paper mostly shows that models are bad at recognizing nouns they rarely see, and that concept difficulty wasn’t controlled.

Data Scaling, Long Tail, and Synthetic Data

  • Several discuss that performance vs. data often looks logarithmic/S‑shaped; exponential data requirements for rare concepts are not surprising.
  • Worry that augmenting long‑tail concepts with generated data can create feedback loops and “model collapse”; others counter that synthetic data can be information‑rich and is already used effectively.
  • One cited work suggests iterative training on model‑generated data degrades performance in the long tail.

Human vs. Machine Learning

  • Debate over whether humans also need “exponential” data for true zero‑shot abilities.
  • Comments stress that humans come with evolved priors, continuous multimodal experience, and abstraction abilities; comparing “a few images for a child” to a tabula‑rasa network is seen as misleading.

Hype, AGI, and AI Winter

  • Some see this paper as evidence current scaling is hitting limits and fear another AI winter driven by unmet AGI promises.
  • Others argue: even without AGI, current models are already economically useful; engineering and optimization alone can sustain progress and commercial deployment.
  • There is tension between enthusiasm about applications (RAG, domain tools, productivity gains) and skepticism about overhyped claims of general reasoning and zero‑shot intelligence.