Bayesians moving from defense to offense
Bayesian statistics is presented as increasingly compelling for fields like medicine and online experimentation, where prior information from past studies or trials can meaningfully shape new analyses. Commenters contrast Bayesian and frequentist approaches on issues such as explicit assumptions, the role and subjectivity of priors, interpretability of p-values and confidence intervals, and their contributions to the replication crisis. Several practical resources are shared for applying Bayesian methods to A/B testing and multilevel modeling, while critics warn that mishandled priors can entrench bias rather than correct it.
Publication bias and use of prior information
- Several commenters note that publication bias can distort both Bayesian and frequentist conclusions.
- Bayesian methods can explicitly model publication bias or encode skepticism via weak/less-informative priors; not accounting for bias when it’s measurable is seen as a larger mistake.
- There is disagreement on how much one should lean on past trials as priors vs. simply collecting more data and meta-analyzing.
Frequentist vs Bayesian: strengths and use cases
- Frequentist stats is described as computationally easier, with strong analytical theory, abundant literature, and established “recipes” for hypothesis testing.
- Many report being taught only frequentist methods, often without explicit discussion of assumptions.
- Bayesian stats is praised for making assumptions and biases explicit, fitting generative and hierarchical models naturally, and working better with small samples or complex multilevel structures.
- Some say the controversy is largely historical; both frameworks often give similar answers with large data.
Interpretation, priors, and philosophy
- Supporters argue priors are unavoidable; the real choice is whether to state them explicitly.
- Critics find priors philosophically dubious, worry about reinforcing prior errors, and prefer to separate “what the data say” from subjective belief.
- There is discussion about “uninformative” or weakly informative priors, maximum-entropy ideas, and whether “absence of belief” is representable.
- Frequentist confidence intervals vs Bayesian credible intervals are contrasted; frequentist intervals are about long-run coverage, not the probability a specific interval contains the true value.
Replication crisis and hypothesis testing
- Several tie the replication crisis (especially in psychology and medicine) to overreliance on p-values, point null tests, and ignoring effect sizes.
- Some argue frequentist practice enabled “garbage science”; others say the root problem is bad reasoning and incentives, not the math itself.
Teaching, tooling, and A/B testing
- Multiple resources are suggested for learning Bayesian modeling, with emphasis on generative models over test “recipes.”
- For A/B testing, commenters highlight beta–binomial models, Bayesian multi-armed bandits, contextual bandits, and profit-maximizing “test and roll” designs.
- Practical caveats include difficulty with many arms, need for good priors, and handling multiple metrics.