How many legs do ten elephants have, if two of them are legless?
Large language models routinely stumble on seemingly simple math and logic puzzles, such as counting the legs of elephants when some are “legless,” revealing errors that range from basic arithmetic slips to surreal explanations. Commenters use these failures to highlight how LLMs generate statistically plausible text rather than perform genuine reasoning, and to debate whether bolting on tools like Python interpreters meaningfully closes that gap. The thread also touches on tokenization quirks, overfitting to common riddles, and what these behaviors imply about the limits of current AI systems versus aspirations for AGI.
Overall theme: LLMs and basic math/reasoning failures
- Thread centers on a simple riddle (“ten elephants, two legless”) and how many LLMs give obviously wrong answers.
- Reported answers include: 32 (correct), 36, 38, 40, 64, 78, and even “eighteen,” often with confident but incoherent reasoning.
- Some models assert elephants have 8 legs, mis-apply subtraction, or double-count “missing” legs. Others mis-handle inverse questions (given total legs, infer number of elephants).
Why LLMs struggle with math (as described in thread)
- Several comments stress that LLMs are probabilistic next-token predictors, not symbolic math engines or logic systems.
- Numbers are just tokens: 0, 10, 100 may each be single tokens; 98 might be multiple. Models don’t inherently switch into a “numeric reasoning” mode.
- Arithmetic/maths are a tiny portion of training corpora and map poorly to tokenization, so performance is fragile and uneven.
- Some see these mistakes as evidence LLMs lack genuine understanding or reasoning; others argue they are still powerful pattern matchers that just weren’t optimized for math.
Tool-use vs. “pure” LLM behavior
- Multiple people note GPT‑4 often gets the elephant and date questions right by writing and executing Python code in a sandboxed interpreter.
- There’s debate on whether this is impressive “generalization plus tool use” or just bolting a calculator onto an autocomplete engine.
- Clear distinction is drawn between the raw LLM and the surrounding system (LLM + tools).
Inconsistency across models and prompts
- Same prompt yields different answers across Bard, GPT‑3.5, GPT‑4, Claude, Mixtral, and Bing; even the same model can change behavior over time.
- Some smaller or newer models answer correctly; others give bizarre logic (e.g., rounding negative legs up to 1).
- Overfitting to common riddles is observed (e.g., always saying “they weigh the same” to feather/brick variants).
Broader reflections
- The failures are used to argue both that LLMs are “just autocomplete” and that they are rapidly improving (especially with larger models and tools).
- Several comments frame LLMs as highly capable “sifters” or assistants rather than reasoners, and see these puzzles as good stress tests for their limits.