DeWitt and Stonebraker's "MapReduce: A major step backwards" (2009)

Original Article ↗ Hacker News Discussion ↗

Novelty, Prior Art, and “Anti‑Memory” Culture

  • Several commenters argue MapReduce was framed as novel despite overlapping with decades of database ideas; they see value in papers that re‑situate “new” work in older literature and revisit earlier critiques.
  • Others criticize a broader CS/adjacent culture that prefers citing recent work and reinventing old ideas instead of building on them.

MapReduce vs. Relational Databases: Right Tool, Right Job

  • Strong agreement that many teams grabbed Hadoop/MapReduce where a single Postgres instance (with indexes) would suffice.
  • Counterpoints: indexes are overhead for full scans; if the job is inherently a sequential pass, a DB won’t beat a streaming pipeline.
  • MapReduce is framed as a batch, fault‑tolerant compute model for massive data (e.g., web indexing, log crunching, data transforms), not as a general SQL engine.
  • Some note that early RDBMSs couldn’t handle web‑scale workloads or cheap‑hardware failures in those use cases.

Evolution: From MapReduce to Spark, BigQuery, Snowflake, etc.

  • Commenters note that later systems absorbed the paper’s core critiques: schemas, richer operators, better execution engines, in‑memory caching, and more SQL‑like interfaces.
  • Massively parallel “split–shuffle–combine” patterns persist under many modern distributed SQL and analytics systems, even if they no longer look like classic Hadoop.

Operational Realities and “Worse Is Better”

  • A recurring theme: a system you can use today (MapReduce clusters, cloud) is worth more than a theoretically better system that takes months of approvals and hardware procurement.
  • This is compared to why cloud won: faster provisioning and autonomy beat pure cost efficiency.

Hype, Overengineering, and Parallels (Kubernetes, Microservices, GraphQL, Data Lakes)

  • Many see MapReduce as an early big‑data hype cycle, mirrored later by microservices, Kubernetes, GraphQL, and data lakes: widely adopted where simpler setups (single DB, monolith, on‑prem) would do.
  • Others defend Kubernetes/containers as neutral deployment tools that don’t constrain application design and can reduce operational toil.
  • Several commenters emphasize that most organizations don’t truly need large‑scale distributed systems; complexity often becomes tech debt.

Implementation Critiques (Hadoop, Spark)

  • Hadoop MapReduce is frequently described as inefficient and awkward, especially given knowledge of vectorized execution and better engines.
  • Spark is praised as a powerful abstraction but criticized for brittle debugging and hidden complexity, with some alleging that vendors retain key fixes in commercial offerings.