No "Zero-Shot" Without Exponential Data
New research on image–text models like CLIP suggests that so‑called “zero-shot” generalization may be far more limited than advertised: performance on rare concepts improves only linearly even as their frequency in training data grows exponentially. Commenters connect this to broader concerns about scaling laws, dataset quality, and whether today’s architectures can ever match human-like abstraction from few examples. The findings feed into a larger debate over the sustainability of current AI hype, risks of an “AI winter,” and the tradeoffs of relying on ever-larger, often synthetic, datasets.
Paper’s Claim and Scope
- Thread centers on a paper arguing “zero-shot” generalization in multimodal models is largely illusory: rare concepts require exponentially more data for linear gains in performance.
- Authors test 34 models across ~3M–400M pretraining samples on 4,000+ concepts and release a long‑tail benchmark (“Let it Wag!”).
- Several commenters emphasize: results are on CLIP-like image–text models, not LLMs; some think this distinction is crucial, others think it may generalize.
Zero‑Shot and Evaluation Nuances
- Multiple comments note that much “zero-shot” work is really just held‑out test data with overlapping classes, not truly unseen concepts.
- Example: training on many animal images and then “zero‑shot” testing on ImageNet labels is questioned; true zero‑shot would be training on horses and testing on zebras.
- Some argue the paper mostly shows that models are bad at recognizing nouns they rarely see, and that concept difficulty wasn’t controlled.
Data Scaling, Long Tail, and Synthetic Data
- Several discuss that performance vs. data often looks logarithmic/S‑shaped; exponential data requirements for rare concepts are not surprising.
- Worry that augmenting long‑tail concepts with generated data can create feedback loops and “model collapse”; others counter that synthetic data can be information‑rich and is already used effectively.
- One cited work suggests iterative training on model‑generated data degrades performance in the long tail.
Human vs. Machine Learning
- Debate over whether humans also need “exponential” data for true zero‑shot abilities.
- Comments stress that humans come with evolved priors, continuous multimodal experience, and abstraction abilities; comparing “a few images for a child” to a tabula‑rasa network is seen as misleading.
Hype, AGI, and AI Winter
- Some see this paper as evidence current scaling is hitting limits and fear another AI winter driven by unmet AGI promises.
- Others argue: even without AGI, current models are already economically useful; engineering and optimization alone can sustain progress and commercial deployment.
- There is tension between enthusiasm about applications (RAG, domain tools, productivity gains) and skepticism about overhyped claims of general reasoning and zero‑shot intelligence.