Reducing undefined behavior in the C language

Original Article ↗ Hacker News Discussion ↗

Undefined behavior in C and C++—operations for which the language guarantees nothing, like signed integer overflow or dereferencing null—remains a central source of bugs, security vulnerabilities, and surprising compiler optimizations. Commenters weigh historical reasons for UB and the need for aggressive optimization against calls to convert more cases to implementation‑defined behavior, add opt‑in modes that trap or sanitize UB, and push new work in C2y to shrink the “UB zoo.” Many argue that truly robust safety will come from stronger languages (like Rust) or hardware/ABI-level mechanisms (such as CHERI and Fil‑C), while C should be reserved for low-level systems code where developers consciously accept these trade‑offs.

Scope and nature of undefined behavior (UB) in C

  • UB is described as “what happens when you violate a mandatory precondition,” giving compilers freedom to assume those cases never occur.
  • Several comments argue this makes whole‑program reasoning extremely hard, since the compiler can optimize based on “UB never happens” and change behavior even before the UB point.
  • Others note that UB in C/C++ is unusually aggressive compared to many other languages, effectively allowing arbitrary behavior (“nasal demons”) rather than just “whatever the hardware does.”

Optimization vs. correctness

  • Defenders of UB say it is essential for optimization: assuming no UB enables inlining, constant propagation, dead‑code elimination, register allocation, and removal of redundant checks.
  • Critics counter that compilers now “weaponize” UB, leading to surprising removal of null checks or branches and making debugging harder.
  • Some suggest compilers should at least warn when UB‑based reasoning eliminates code, while others argue this is infeasible because nearly all optimizations depend on assuming no UB.

Implementation‑defined vs undefined

  • Multiple commenters argue many behaviors (e.g., null dereference, uninitialized reads) should be implementation‑defined or guaranteed to trap, rather than fully undefined.
  • Opposing view: exotic hardware, WASM, and performance constraints make fully defined behavior impractical; C committees are reluctant to break existing implementations.

Integer overflow

  • Strong debate on whether signed overflow should be defined (e.g., two’s‑complement wrap, traps, or configurable).
  • Pro‑definition side: easier runtime checks, less UB exploitation, consistent behavior; flags like -fwrapv/-ftrapv already exist.
  • Anti‑definition side: UB lets compilers warn about obvious mistakes, use sanitizers, and keep powerful optimizations; unsigned overflow is already defined yet still bug‑prone.

Uninitialized memory

  • Example code shows compilers exploiting reads from uninitialized malloc’d memory to optimize in surprising ways (e.g., constant folding to always take one branch).
  • Some want “indeterminate but stable” values or implementation‑defined semantics; others state C2y will not do this because it would break too many implementations.

Alternatives and mitigations

  • Suggestions include: use memory‑safe languages (Rust, Go) for applications; keep C for low‑level kernels; rely on sanitizers (UBSan, ASan, TSan), Valgrind, and static analyzers despite their limits.
  • CHERI and Fil‑C are cited as ABI‑level approaches that tightly constrain memory behavior and reduce exploitability, with tradeoffs in compatibility and deployment.

Standard evolution and layering

  • C2y reportedly removes some UB instances, but many remain for optimization and portability.
  • There’s discussion about which layer (language, OS, ABI, hardware) should bear complexity for safety, with no clear consensus.