Out-of-bounds read and write in the glibc's qsort()

Original Article ↗ Hacker News Discussion ↗

A recently identified out-of-bounds read/write in glibc’s `qsort()` arises only when programs pass it a “bad” comparison function, such as one that overflows on integer subtraction or violates ordering rules with floats and NaNs. Commenters debate whether this is truly a security vulnerability in glibc or simply undefined behavior caused by incorrect caller code, and whether it merits a CVE given the rarity and exploit difficulty. The thread broadens into a comparison of safety cultures in C/C++ versus Rust and other languages, arguing over how much responsibility libraries should take to defend against misuse versus preserving performance and simplicity.

Nature of the qsort bug

  • The bug is an out-of-bounds read/write in glibc’s qsort when the caller supplies a non-transitive (buggy) comparison function.
  • Many see it as “user error” (invalid parameter), analogous to calling memcpy with bad pointers, but still agree fixing bounds checks is good defensive practice.
  • Others stress that the “obvious” comparator implementation (a - b) is subtly wrong due to integer overflow, so this isn’t purely an exotic edge case.

“Holding it wrong” vs security vulnerability

  • One camp argues this is primarily a bug in application code, since standards explicitly require a total ordering; undefined behavior is on the caller.
  • Another camp says that if a common misuse can cause memory corruption, that’s a legitimate safety issue and merits a CVE for the library implementation as well.
  • There is criticism of the “it’s UB so anything goes” attitude from C/C++ culture, contrasted with more defensive norms elsewhere.

Comparators, NaNs, and floating point

  • Sorting floats with NaNs or poorly handled infinities can violate comparator requirements, causing crashes or incorrect behavior.
  • Examples show how NaNs break reflexivity and transitivity, confusing sort algorithms. Some suggest that trying to sort NaNs is itself usually a logic bug.

Language safety cultures and integer overflow

  • Rust is cited as treating “you’re holding it wrong” cases as safety issues: bad comparators may give wrong results but not UB; traits like Ord are safe, with separate “unsafe traits” for invariants that can cause UB if broken.
  • Discussion of overflow handling:
    • Rust’s checked/saturating/wrapping operations and debug-vs-release behavior.
    • C/C++ signed overflow as UB; unsigned wraparound often not what programmers intend.
    • Ada, Pascal, and WUFFS as examples of languages or DSLs that encode ranges/refinements for extra safety.

Performance and alternatives to qsort

  • qsort is described as relatively slow; for high-performance code people recommend C++ std::sort or specialized libraries, sometimes via a thin C++ wrapper callable from C.
  • Some worry about the performance cost of extra safety checks; others argue the cost is usually negligible.

Exploitability, CVEs, and scoring

  • Exploitability is debated: requires an already-buggy comparator and often a privileged (e.g., setuid-root) context; no in-the-wild exploits are known in the thread.
  • Mixed views on whether a CVE should be issued:
    • Pro: encourages users to audit comparators and update.
    • Con: CVEs are costly for regulated environments and should be reserved for clearly exploitable issues.
  • Broader frustration with CVSS scores: some severe, easily exploitable bugs scored low, while impractical DoS-style bugs get high scores, making prioritization hard.