LLMs can't do probability

Original Article ↗ Hacker News Discussion ↗

Large language models are being tested on seemingly simple probabilistic tasks—like saying “left” 80% of the time and “right” 20%—and often fail, defaulting to deterministic answers such as always choosing “left.” Commenters attribute this to how LLMs actually work: they output a probability distribution over tokens and are optimized to give the most likely next token, not to act as random number generators, with sampling temperature and RLHF further distorting any intended probabilities. Many note that humans are also poor at intuitively generating randomness, and suggest that reliable probabilistic behavior from LLM-based systems will require explicit tools (e.g., code execution with real RNGs) or different training approaches rather than prompt engineering alone.

Scope of the issue

  • Thread centers on a simple prompt: “Say ‘left’ 80% of the time and ‘right’ 20% of the time,” and observes many LLMs output “left” almost always.
  • Participants debate whether this shows a lack of probabilistic understanding, a limitation of decoding/sampling, or simply a bad prompt for this architecture.

Determinism, temperature, and sampling

  • Many point out that base LLM output is deterministic logits; randomness comes from the sampling step (temperature, top‑k/p).
  • With low temperature or argmax decoding, the model will output the single most likely token (“left”) 100% of the time, regardless of verbal probabilities in the prompt.
  • Several argue the author effectively tested frequentist behavior (empirical outcomes) where the model is only set up to give a single Bayesian-style prediction per prompt.

Evidence from experiments

  • Some users inspect logprobs: one token (e.g., “7” or “left”) has much higher probability than alternatives, contradicting the requested 80/20 split.
  • When asked for sequences (e.g., 100 outputs in one go), or “repeat 10 times,” models often internally write and run Python/NumPy RNG code, yielding roughly correct distributions (e.g., ~80/20).
  • Dice-roll tests show non-uniform biases (e.g., overproducing 3–5), reinforcing that “generate a random X” is unreliable without tools.

Humans vs LLMs on probability

  • Multiple comments stress humans are also bad at randomness and probability; reference to experiments showing predictable “random” human choices.
  • Others counter that large human samples (e.g., polls) can approximate prompted probabilities reasonably well, suggesting humans still outperform current LLMs on this narrow task.

Training, RLHF, and calibration

  • Some note pre-RLHF base models were better calibrated; RLHF and safety tuning may flatten or sharpen logits and hurt probabilistic behavior.
  • Ideas are floated about fine-tuning on probabilistic prompts using KL divergence, but temperature dependence and decoding make this tricky.

Broader takeaways

  • Consensus: LLMs are probabilistic models but not good on-demand RNGs; for reliable randomness, they should call external code or tools.
  • Several commenters generalize: LLMs are “plausibility engines” optimized to produce the most likely-looking next token, not to obey explicit probabilistic instructions.