Writing correct software is hard and math alone won’t help us (2016)
Formal methods and mathematical proofs can guarantee correctness for small, well-specified pieces of code, but commenters argue they break down quickly in the face of real-world software: shifting requirements, messy operating systems, unreliable hardware, and human-centered workflows. Many see verification as essential for safety‑critical or security‑sensitive components, yet impractical as a universal approach, advocating instead for a mix of types, testing, careful specifications, and empiricism. The rise of AI-generated code sharpens these concerns, both by increasing the volume of brittle software and by tempting people to overestimate what automated reasoning and formal proofs can realistically ensure.
What “Correctness” Means and Why It’s Hard
- Many comments stress that correctness is relative to a specification, and getting that spec right is the hardest part.
- Real systems have fuzzy, changing, and politically contested requirements (e.g., “what counts as a sale”, “draw something that looks good”).
- For lots of business and UX behavior, full formalization is seen as impractical or impossible; empiricism and testing dominate.
Limits from Theory (Halting, Rice, Gödel)
- The halting problem and Rice’s theorem are cited as showing that fully generic verification is impossible, but several argue this is mostly about “arbitrary programs,” not typical real-world code.
- Some point out that undecidability does not forbid proofs for specific programs; it forbids a universal procedure.
- There is some disagreement over whether the article leans too heavily on these results and overstates pessimism.
Formal Methods in Practice
- Formal verification is viewed as essential but expensive in domains like safety-critical, medical, military, cryptographic, and some high-security systems.
- Multiple people report that overemphasis on proofs can stall progress, especially when proofs rest on unrealistic assumptions (e.g., ideal sensors).
- A recurring theme: proof efforts are fragile under changing requirements; large proofs can become unsalvageable when assumptions shift.
Partial Guarantees: Types, Properties, and Tools
- Strong type systems, capability systems, and lightweight formal methods are praised as practical middle ground.
- Ideas like subtypes, pure functions, and property checking can guarantee important invariants without proving whole programs.
- Static analysis and bounded reasoning are underused but relatively cheap benefits.
Environment and OS Complexity
- Several argue that OS primitives and filesystems are intrinsically hard to model; much of the difficulty is at this boundary rather than in core algorithms.
- Microkernels, capabilities, and data‑centric OS designs are discussed as ways to make reasoning easier, but still very hard.
AI, “Vibe Coding,” and Reliability
- Some worry that LLM‑generated code and “vibe coding” normalize a dismissive attitude toward correctness and could reduce reliability.
- Others note that software has been unreliable long before AI; LLMs may also help with specification, proofs, and tooling if carefully constrained.