A ragtag band of internet friends became the best at forecasting world events
An article on “superforecasters” who use base rates and Bayesian-style reasoning to predict geopolitical events like a Chinese invasion of Taiwan prompts debate over how meaningful such probability estimates really are. Commenters explore the value of base rates, calibration metrics like Brier scores, and prediction markets, while critics argue that one-off, rare events and complex systems like war or climate are ill-suited to this kind of quantification. Others question why, if these methods work, they aren’t more dominant in financial markets, and whether apparent forecasting success reflects genuine skill or survivorship bias.
Role of Base Rates and Forecasting Methods
- Many comments focus on “base rates” as an anchor for probabilistic forecasts.
- Supporters say base rates provide an uninformed prior and a sanity check before adjusting for new information.
- Critics argue some base-rate constructions (e.g., mixing different historical periods or arbitrary Laplace priors) are hand-wavy and can look like numerology.
- Several note that base rates are only one ingredient; good forecasters combine multiple models and qualitative analysis.
Calibration, Scoring, and Validation
- Question raised: how can single-event probabilities (e.g., “China invades Taiwan”) ever be validated?
- Answers: look at large sets of predictions; if 8% events happen ~8% of the time, you’re calibrated.
- Mention of Brier scores, cross-entropy, and “proper scoring rules” that reward honest probabilities and contrarian correctness.
- Some remain skeptical, arguing heterogenous, unique geopolitical events are hard to aggregate meaningfully.
Comparison to Experts, Finance, and Markets
- Debate over whether such forecasters should be competing in financial markets (“major leagues”) if they’re really good.
- Responses: markets are different (noise, incentives, domain knowledge) and many forecasters prioritize social impact projects over profit.
- Some argue if predictions are +EV, they should translate into trades; others suspect survivorship bias or luck.
- Prediction markets are viewed as a useful mechanism to encode beliefs and measure forecasting skill.
Black Swans, Nuclear War, and Risk Perception
- Several argue forecasting rare catastrophic events is structurally biased toward “no,” and underweights impact.
- Others counter that proper scoring and long track records still make it meaningful to estimate low probabilities.
- Discussion of climate change forecasts: climate models provide physics but forecasting emissions and policy is more political and uncertain.
Usefulness and Limits of Forecasting
- Some see forecasting as intellectually rewarding and practically useful for policy, finance, and risk management.
- Others doubt decision-makers will act on forecasts, or compare the practice to horoscopes when applied to unique geopolitical events.
- A few note that even if probabilities are philosophically messy, bets and markets force coherent, testable beliefs.