Trillions spent and big software projects are still failing

Domain complexity and irreducible rules

• Many failures stem from attempting to encode extremely complex, path‑dependent business rules (e.g., tens of thousands of payroll rules across many unions).
• Participants note that simplifying such rules is politically near‑impossible: every special case has a vocal beneficiary, and neither unions nor taxpayers want to accept loss or higher cost.
• Committees are seen as both punchline and necessity: the only way to surface legal/contractual constraints and negotiate simplifications.

Scale, modularity, and project shape

• Large, centrally managed systems with many stakeholders and moving parts are viewed as structurally failure‑prone.
• Several comments invoke Gall’s Law: complex systems that work tend to evolve from simpler systems that work; “big bang” rewrites rarely succeed.
• Successful patterns cited: many small systems with stable interfaces, incremental rollouts (start with a low‑risk subset), and consciously designing for scale while deploying small first.

Management, incentives, and accountability

• Recurrent theme: projects usually fail for organizational and political reasons, not technical ones.
• Power often sits with non‑technical decision makers who don’t understand feasibility, complexity, or risk; requirements never converge; optimistic timelines are enforced.
• Middle management and contractors frequently face little real accountability; retrospectives focus blame on developers while shielding leadership.
• Principal–agent problems and misaligned incentives (cost‑plus contracts, promo‑driven culture, resume‑driven development) encourage overscope and under‑delivery.

Learning from history and other engineering fields

• Many argue software does a poor job of studying past systems and failures compared to hardware or civil engineering.
• Counterpoint: software’s flexibility, rapid change, and weaker “laws of physics” make convergence on stable methods harder; churn and fashion (languages, frameworks) exacerbate this.
• There is debate over professionalization: some call for licensure and liability akin to civil engineering; others argue economics and contracts already dictate appropriate rigor.

Developer culture and practice

• Critiques of developer ego, cargo‑cult adoption of trends, and excessive abstraction/“mindfacturing” that build towers of leaky layers.
• Others push back that most engineers would happily “do it right” but are constrained by deadlines, process, and management.

AI’s role

• Consensus that AI is an amplifier: in disciplined organizations it can increase throughput; in chaotic ones it accelerates failure.
• Skepticism that AI will fix governance or incentive problems; worry it will enable even less skilled, less attentive work on high‑stakes systems.