Simpson's paradox
Simpson’s paradox — where trends in separate groups reverse when the groups are combined — is highlighted through real-world examples from university admissions, marketing analytics, web performance, and COVID data. Commenters emphasize that aggregate statistics can be deeply misleading without understanding causal structure, subgroups, and confounding variables, and point to related ideas like Goodhart’s law, Berkson’s paradox, and hierarchical models. The recurring lesson is that sound inference requires looking at both overall metrics and disaggregated data, and choosing metrics that align closely with the underlying question or decision.
Overview of Simpson’s Paradox
- Described as an apparent contradiction where trends in every subgroup reverse when data are aggregated.
- Several commenters emphasize it’s not a logical contradiction, just “two ways of looking at the same data.”
- Key lesson: sometimes the aggregate matters; sometimes subgroup patterns matter. You must interpret both in context.
Causality vs. Pure Statistics
- Strong theme: passive observation only reveals correlation; causal understanding requires experiments or a causal model.
- UC Berkeley admissions example: data alone can’t say where “bias” lies without knowing how admissions decisions are actually made (department vs university, funding, competitiveness).
- Normalizing or re-scaling doesn’t resolve the paradox; you must decide what to condition on, which is a causal question, not a purely statistical one.
- References to causal-inference treatments (e.g., Pearl) and mixed/hierarchical models as formal ways to handle subgroup structure.
Real-World Examples
- E‑commerce: category-level marketing efficiency improved but overall marketing cost ratio worsened due to a mix shift toward a high-cost category.
- SRE / performance: optimizations reduced latency for all user segments but global latency metrics worsened after induced usage growth in high-latency regions. Debate over whether this is Simpson’s vs. Jevons (induced demand); some argue both apply at different stages.
- COVID: cited example where subgroup case-fatality rates vs. national aggregates reversed.
- Housing: US data where houses with AC are more expensive within each state, but nationally houses without AC are pricier due to state mix.
- ML / model evaluation: dataset with more “easy” cases made overall accuracy look better despite per-class degradation.
Practical Lessons and KPIs
- Naively chosen metrics (e.g., overall p99 latency, overall marketing ratio) can be misleading under mix shifts.
- Better practice: define KPIs that are tightly tied to the underlying problem (e.g., p99 latency for large customers, per-category metrics) and then also inspect aggregates.
- Main takeaway repeated: always “keep both the parts and the whole in mind at once.”
Related Concepts and Meta
- Related ideas: Berkson’s paradox, Goodhart’s law, Lord’s paradox, induced demand.
- Some philosophical side-discussion on types of paradoxes, Epicurean multi-hypothesis thinking, and a brief tangent on physics experiments and whether statistics are needed.
- Minor meta-thread on the proliferation of bare Wikipedia links on HN without added context.