LLMs can't do probability

Original Article ↗ Hacker News Discussion ↗

Scope of the issue

  • Thread centers on a simple prompt: “Say ‘left’ 80% of the time and ‘right’ 20% of the time,” and observes many LLMs output “left” almost always.
  • Participants debate whether this shows a lack of probabilistic understanding, a limitation of decoding/sampling, or simply a bad prompt for this architecture.

Determinism, temperature, and sampling

  • Many point out that base LLM output is deterministic logits; randomness comes from the sampling step (temperature, top‑k/p).
  • With low temperature or argmax decoding, the model will output the single most likely token (“left”) 100% of the time, regardless of verbal probabilities in the prompt.
  • Several argue the author effectively tested frequentist behavior (empirical outcomes) where the model is only set up to give a single Bayesian-style prediction per prompt.

Evidence from experiments

  • Some users inspect logprobs: one token (e.g., “7” or “left”) has much higher probability than alternatives, contradicting the requested 80/20 split.
  • When asked for sequences (e.g., 100 outputs in one go), or “repeat 10 times,” models often internally write and run Python/NumPy RNG code, yielding roughly correct distributions (e.g., ~80/20).
  • Dice-roll tests show non-uniform biases (e.g., overproducing 3–5), reinforcing that “generate a random X” is unreliable without tools.

Humans vs LLMs on probability

  • Multiple comments stress humans are also bad at randomness and probability; reference to experiments showing predictable “random” human choices.
  • Others counter that large human samples (e.g., polls) can approximate prompted probabilities reasonably well, suggesting humans still outperform current LLMs on this narrow task.

Training, RLHF, and calibration

  • Some note pre-RLHF base models were better calibrated; RLHF and safety tuning may flatten or sharpen logits and hurt probabilistic behavior.
  • Ideas are floated about fine-tuning on probabilistic prompts using KL divergence, but temperature dependence and decoding make this tricky.

Broader takeaways

  • Consensus: LLMs are probabilistic models but not good on-demand RNGs; for reliable randomness, they should call external code or tools.
  • Several commenters generalize: LLMs are “plausibility engines” optimized to produce the most likely-looking next token, not to obey explicit probabilistic instructions.