Why is everything based on likelihoods even though likelihoods are so small?
Statisticians and practitioners unpack why likelihoods in statistical models often take on extremely small numerical values, especially when multiplying many probabilities, yet still yield meaningful inferences. They emphasize that likelihood is typically a probability *density* (not a probability itself), that only relative or log-likelihoods matter for tasks like parameter estimation and model comparison, and that maximum likelihood methods work well under broad conditions but can struggle with multimodal or poorly behaved distributions. The exchange also touches on Bayesian vs frequentist interpretations, the likelihood principle, and how these abstractions connect to real-world decision-making and risk.
Conceptual role of likelihood
- Likelihood is defined as the probability (or density) of the observed data given model parameters, not the probability of the parameters themselves.
- The likelihood function is a function of parameters with data fixed; it is not a probability distribution over parameters and does not integrate to 1 in parameter space.
- Several comments note the original question confuses likelihoods with posterior probabilities and with “how probable” a parameter value is.
Continuous vs discrete and “tiny numbers”
- For continuous variables, the probability of any exact value is 0; only intervals have non‑zero probability. Densities can be small or large and can even exceed 1 depending on scale and variance.
- Joint likelihoods for many observations multiply many terms, so the product is almost always extremely small or extremely large; this is expected behavior, not a problem.
- Because only ratios matter, log‑likelihoods and likelihood ratios are emphasized rather than absolute magnitudes.
Bayesian, frequentist, and the likelihood principle
- One line of discussion: maximum likelihood estimation (MLE) approximates Bayesian inference with weak/regular priors and has good convergence properties.
- Another line stresses the likelihood principle: all information about parameters, given a model, is contained in the likelihood function; any admissible estimator must “do something sensible” with this function.
- Debate appears around whether frequentist treatment of parameters is conceptually awkward, and how Bayesian posteriors, priors, and MAP estimates relate to likelihood.
Limitations, pathologies, and practice
- Likelihood-based methods can behave poorly for multimodal or ill‑behaved distributions and in high‑dimensional spaces where mass is far from the MLE.
- Model comparison via AIC, BIC, likelihood‑ratio tests etc. is described as heuristic and implementation‑dependent; reproducing results across software can be difficult.
- Some argue statistical methods are fragile and heavily approximation‑driven; others stress they work well in many practical settings.
Intuition, risk, and real‑world reasoning
- Multiple analogies (coin flips, lotteries, yardsticks, raindrops, bowling balls) are used to show that individual exact outcomes are extremely unlikely even under the true model.
- Discussion extends to decision‑making: importance of probability × consequence, challenges in reasoning about tail risks and “low‑probability” bugs that affect every user at least once.