Simpson's paradox

Overview of Simpson’s Paradox

• Described as an apparent contradiction where trends in every subgroup reverse when data are aggregated.
• Several commenters emphasize it’s not a logical contradiction, just “two ways of looking at the same data.”
• Key lesson: sometimes the aggregate matters; sometimes subgroup patterns matter. You must interpret both in context.

Causality vs. Pure Statistics

• Strong theme: passive observation only reveals correlation; causal understanding requires experiments or a causal model.
• UC Berkeley admissions example: data alone can’t say where “bias” lies without knowing how admissions decisions are actually made (department vs university, funding, competitiveness).
• Normalizing or re-scaling doesn’t resolve the paradox; you must decide what to condition on, which is a causal question, not a purely statistical one.
• References to causal-inference treatments (e.g., Pearl) and mixed/hierarchical models as formal ways to handle subgroup structure.

Real-World Examples

• E‑commerce: category-level marketing efficiency improved but overall marketing cost ratio worsened due to a mix shift toward a high-cost category.
• SRE / performance: optimizations reduced latency for all user segments but global latency metrics worsened after induced usage growth in high-latency regions. Debate over whether this is Simpson’s vs. Jevons (induced demand); some argue both apply at different stages.
• COVID: cited example where subgroup case-fatality rates vs. national aggregates reversed.
• Housing: US data where houses with AC are more expensive within each state, but nationally houses without AC are pricier due to state mix.
• ML / model evaluation: dataset with more “easy” cases made overall accuracy look better despite per-class degradation.

Practical Lessons and KPIs

• Naively chosen metrics (e.g., overall p99 latency, overall marketing ratio) can be misleading under mix shifts.
• Better practice: define KPIs that are tightly tied to the underlying problem (e.g., p99 latency for large customers, per-category metrics) and then also inspect aggregates.
• Main takeaway repeated: always “keep both the parts and the whole in mind at once.”

Related Concepts and Meta

• Related ideas: Berkson’s paradox, Goodhart’s law, Lord’s paradox, induced demand.
• Some philosophical side-discussion on types of paradoxes, Epicurean multi-hypothesis thinking, and a brief tangent on physics experiments and whether statistics are needed.
• Minor meta-thread on the proliferation of bare Wikipedia links on HN without added context.