Kimi K3, and what we can still learn from the pelican benchmark
A long-running joke benchmark—asking large language models to generate an SVG of “a pelican riding a bicycle”—is used here to probe how newer models like China’s Kimi K3 compare on creativity, code generation, cost, and speed. Commenters argue over whether the test has been “poisoned” by appearing in training data, how much it still says about general capability versus overfitting, and what better benchmarks for real-world tasks (like tool use, codebase refactoring, or complex planning) might look like. The thread also touches on model economics and geopolitics, noting the rapid progress of Chinese labs despite compute limits and raising questions about the sustainability and reliability of different AI providers.
Nature and Value of the Pelican Benchmark
- Many see the “pelican riding a bicycle” SVG task as a playful, non-rigorous benchmark that persists mainly as a running joke and shared “hello world.”
- Others argue it still usefully tests composition of novel concepts, SVG generation skills, and a model’s “taste” or aesthetic judgment.
- Several commenters think it has become too “polluted” to be meaningful: widely publicized, probably in training data, and potentially a target for explicit optimization.
Training Data, Overfitting, and Style Convergence
- Ongoing debate about whether labs intentionally train on pelican SVGs versus them appearing incidentally in large web crawls.
- Some report models doing markedly better on pelican-on-bike than similarly absurd prompts (e.g., sloth on skateboard), suggesting possible overfitting or benchmark-specific RL.
- Others counter that general SVG and image improvements plus broad training can explain the gains.
- People note strong stylistic convergence: left-to-right motion, similar colors, generic cartoony look, even when the prompt doesn’t specify style.
Benchmarks, Cost, and Practical Utility
- Several participants distrust frontier-model benchmarks in general, preferring personal tests (e.g., video generation, web/UX design, complex workflows).
- Disagreement over the importance of small cost differences: some say a 10× token price gap matters greatly at large scale; others say all are cheap compared to human labor.
- Suggestions to evolve benchmarks include:
- Randomized “creature-vehicle” SVG tasks instead of a fixed pelican.
- More realistic tasks (codebases, refactors, removing dead code, long-horizon tools/agents).
- Hybrid tests such as SWE-style coding with intermittent SVG interruptions.
Chinese Models, Scale, and Competition
- Kimi K3 discussed as roughly competitive with high-end Western models on some tasks, but slower and with language and deployment caveats.
- Debate over how Chinese labs achieve large MoE models given compute and funding constraints; ideas include architectural optimizations, domestic chips, and workarounds to export controls.
- Parameter count alone is seen as a weak proxy for quality; attention mechanisms, expert size/density, and RL matter.
Community Reaction and Meta-Discussion
- Some users enjoy the long-running pelican “bit” and ask for galleries; others see repeated posts as spammy or promotional.
- Several stress Goodhart’s law: once a cute benchmark becomes a target, its value as a measure of general capability degrades.