Are you better than a language model at predicting the next word?
A small web quiz challenges people to guess the next word in real Hacker News comments and compares their accuracy to various language models, highlighting both human strengths and model limitations. Commenters explore how the game illustrates that LLMs are fundamentally probabilistic next-word predictors rather than thinking entities, and note that humans can sometimes outperform them when given time and context. Others question the test’s validity and scoring, debate what “smart” really means beyond pattern matching, and suggest tweaks such as better feedback, alternative scoring metrics, and more transparent framing.
Game concept and mechanics
- Quiz asks humans to predict the next word in real Hacker News comments, framed as a competition with several LLMs and a unigram baseline.
- Incorrect options are generated by an LLM (e.g., llama2); each model then “chooses” among the four options by picking the completion with lowest perplexity.
- Temperature is not used for the choice; instruction‑tuned/chatbot behavior is largely avoided to reduce “voice” bias.
- “Correct” means “the word that actually appeared next in the original HN comment,” not any semantically plausible word.
User experience and clarity
- Several people enjoy the idea and call it clever or fun, but many find the quiz longer or more tedious than expected.
- Suggestions include: show one question at a time, provide instant feedback, clarify upfront that prompts come from HN and that “correct” = original commenter’s word.
- Some confusion arises from very short or clipped prompts (e.g., only a symbol or a single word), which feel like pure guessing.
Difficulty, scores, and strategies
- Reported human scores vary widely (e.g., 2/15 to 11/15; 28/100), often near or slightly above random choice.
- LLM scores also vary significantly per run and per model; sometimes humans beat the best model, sometimes not, and the unigram baseline occasionally does surprisingly well.
- The author reports that on 1,000 questions, LLMs get ~30–35% correct; patient humans can reach ~40–50%.
- Some users notice they do better on longer prompts or when they recognize specific HN comments.
- A proposed strategy is to pick the “outlier” option, assuming distractors are LLM‑like; effectiveness is unclear.
- Several people argue that a scoring method based on ranking / probabilities (cross‑entropy, likelihood) would be more meaningful than strict right/wrong.
What is really being measured?
- Many emphasize that the quiz measures alignment with one specific human’s next word, not objective correctness or “intelligence.”
- Multiple options are often grammatically and semantically valid; choosing which is “right” is somewhat arbitrary.
- Some argue this shows the limits of next‑word prediction as a proxy for “smartness”; others say that is precisely the point—LLMs are statistical next‑token models, not deep reasoners.
Methodological and training-data concerns
- Questions are raised about whether HN comments used might be in the training data of some models, and whether that biases results.
- Using older comments is suggested but might increase training-set overlap; using newer ones avoids that but cannot fully be verified.
- There is discussion of how base models differ from instruction‑tuned chatbots, how “style” emerges, and how beam search can approximate “going back” on predictions.
Broader reflections on LLMs and intelligence
- Commenters contrast statistical pattern matching (LLMs) with symbolic reasoning and general intelligence, noting that success at next‑word prediction doesn’t imply AGI.
- Others note that humans bring fresh experiences, long‑term goals, and social actions (e.g., politics, organizing, inviting someone for coffee) that LLMs currently lack.
- Some see the quiz as a humorous inversion of “look, I broke the AI” posts and as a good teaching tool for how LLMs actually operate.