Get me out of data hell

Writing and Reception

• Many commenters praise the post’s prose, humor, and “acerbic” style; some compare it to classic tech-humor series and say it helped them process burnout.
• Several go down the author’s blog rabbit hole, highlighting earlier essays on burnout, toxic positivity, quitting, and AI as particularly resonant.
• A few readers dislike the “theatrical” tone, but others defend it as a necessary creative outlet.

Serverless, Architecture, and “Cutting-Edge”

• The “serverless cutting-edge” line is widely enjoyed, but some argue Lambda is no longer cutting-edge.
• Multiple people describe painful “all‑in on Lambda” architectures: chains of many small functions, cold starts, brittle failure modes, and complexity that a simple monolith on EC2 would have avoided.
• Others counter that traditional servers can be just as self‑harming; choice of architecture is context‑dependent.

Nature of Data Engineering Hell

• Many report similar “data hell” experiences: over‑engineered pipelines, absurd DAGs that move data S3→S3→S3, brittle Airflow jobs, and scripts chaining Python/shell/R where SQL would suffice.
• Job definitions for “data engineer” and “data scientist” are seen as wildly inconsistent, sometimes meaning “distributed systems expert,” sometimes “fills Excel sheets.”
• Several note that data work often lacks visible, verifiable outputs; broken pipelines can “fail silently” while still supporting careerist narratives around “AI” and “big data.”

Organizational Dysfunction and Incentives

• Repeated theme: the real problem is organizational—politics, siloed access, risk‑averse or confused leadership, and incentives that reward onboarding data sources and buzzwords, not correctness.
• People describe environments where fixing fundamentals is impossible: Jira roadmaps forbid refactors, approvals are politicized, and “comfort” for executives (often via big consultancies) is valued over outcomes.
• Some argue many enterprises are structurally incapable of modernizing critical data systems; old mainframes persist partly because large “modernization” efforts so often fail.

Burnout, Mental Health, and Coping

• Numerous commenters say the piece is intensely triggering because it mirrors their own burnout, “pain zone navigation,” and dread of bad codebases.
• Strategies mentioned: pair‑programming just to survive, retreating to smaller companies, taking long sabbaticals, switching careers (e.g., trades, restaurants, solo software businesses).
• There’s tension between “just quit, life is short” and the realities of mortgages, kids, and a cooler job market.

Data Quality, Testing, and “Right from Day One”

• Many tie the logging fiasco directly to lack of tests and poor observability; several assert anything important must be test‑backed.
• Suggested remedies: treat data engineering like high‑performance software engineering—fast tests, CI/CD, refactoring to keep feedback loops short, and clear contracts for upstream data.
• Some stress this is hard once a platform is declared “done” and measured only by number of sources onboarded; leaders rarely accept the temporary slowdown needed to rebuild properly.

Tools, Platforms, and Technical Debates

• Vigorous discussion of ELT vs ETL: bulk‑load to a DB or lakehouse tables, then transform in SQL, versus pushing heavy logic into external code.
• Debate over SQL Server/Postgres vs object storage + Iceberg/Delta/Hudi; trade‑offs in cost, performance, DBA culture, and complexity.
• Microsoft‑centric stacks (SQL Server, PowerBI) are described by some as brittle and deadlock‑prone; others defend pragmatic, simple SQL‑centric designs.
• Observability-as-data and event‑driven architectures are proposed; others warn that making observability mission‑critical increases blast radius.

Consultancies, Industry Culture, and Geography

• Several note that big‑firm consultants often deliver “comfort” and slideware more than working systems; good boutiques are seen as rare and capacity‑limited.
• Multiple comments criticize the Australian corporate tech/data scene (especially Melbourne) as hype‑driven, politically clogged, and status‑oriented, though a few pockets of excellence exist.
• Broader point: median competence in hyped fields like “data” is lower than outsiders expect, and titles (e.g., Chief Data Officer) may fade or be renamed as this becomes obvious.