Is the "modern data stack" still a useful idea?
Vendors and data practitioners are increasingly questioning whether the “modern data stack” — a loosely defined bundle of cloud ETL, warehouses, transformation tools like dbt, and BI platforms — ever delivered enough value to justify its cost and complexity. Commenters describe a hype-driven ecosystem that pushed fragmented, SaaS-heavy stacks onto companies that often lacked clear analytics goals, solid engineering practices, or realistic cost accounting, leading many to reconsider integrated platforms or bespoke solutions built by experienced engineers instead. The emerging consensus is that tooling choices should be driven by concrete business needs, scale, and observability practices rather than marketing terms like “modern,” with some predicting a shift toward simpler, more tightly integrated “analytics stacks.”
Usefulness of “Modern Data Stack” (MDS) as a Concept
- Many see “MDS” as a buzzword that lost meaning once co‑opted by vendors and VCs.
- Critiques: ill-defined, “modern” ages quickly, mostly resonated with investors and analysts, not practitioners.
- Some argue the label helped marketing and partner ecosystems but added little descriptive value.
Vendor-Heavy vs Integrated/Bespoke Approaches
- Strong criticism that the multi-vendor MDS created unnecessary complexity and high spend (e.g., multiple tools just to run simple pipelines).
- View that companies now prefer integrated platforms or building in-house, especially as they realize hiring a few engineers can be cheaper than stacking many SaaS products.
- Counterpoint: modular “pick-your-stack” tooling gives flexibility, avoids lock-in, and lets teams tailor components; integrated “all‑in‑one” systems are often clumsy.
- Consensus that the right answer is context-dependent (company size, skills, requirements).
dbt and the Analytics Stack
- Widely acknowledged as a major step forward for organizing SQL: version control, DAGs, tests, documentation, CI/CD hooks.
- Also criticized as slow and awkward compared with dataframe tools, lacking a great local IDE, and becoming hard to manage at scale (hundreds+ models).
- Some see dbt as best suited to “last‑mile” transformations, not messy raw-data wrangling.
Software Engineering Practices vs Data Engineering
- Many argue the data space lags a decade behind standard software engineering in CI/CD, testing, observability, and deployment discipline.
- Others say the core problems are actually the same as general SWE: dependency management, contract changes, monitoring, and error handling.
- One camp stresses data-specific challenges: schemas and distributions change without code changes; version control isn’t the sole gate on system behavior.
Data Collection, Costs, and Over-Instrumentation
- Debate over “measure everything” vs hypothesis-driven collection.
- Some justify broad ingestion when storage and built‑in exports are cheap and low effort.
- Others highlight hidden costs: engineering time, connector maintenance, MDS tool bills, and opportunity cost, arguing many pipelines exist without clear business questions or ROI.
What a “Modern” Stack Looks Like in Practice
- Suggested stacks range from heavyweight (Kafka, Flink, Iceberg, Spark/Ray, metadata tools) to very simple (BigQuery + dbt + basic BI; or even MySQL + file server + simple reporting).
- Common advice: keep it as simple as possible, match tools to concrete needs, and avoid chasing trends for résumé or hype.