OpenBSD – pinning all system calls

Original Article ↗ Hacker News Discussion ↗

OpenBSD’s new syscall “pinning” feature — which restricts where and which system calls a binary may invoke — is prompting debate over how much real-world protection it offers against modern exploit techniques like ROP, and whether it mainly hardens already rare attack paths. Commenters weigh OpenBSD’s history of proactive, sometimes speculative mitigations against concerns about added complexity, unclear threat models, and the discovery of a buffer overflow in this very implementation. The thread also touches on broader questions of security engineering practice, including reliance on C vs. adopting Rust or stronger compiler checks, and how to measure the value of mitigations that may prevent, rather than just fix, vulnerabilities.

Scope of the new syscall pinning

  • Kernel now constrains syscalls more tightly:
    • Previously: syscalls only allowed from processes’ libc text region.
    • Now: pinned to specific libc syscall stubs.
    • For static binaries: only syscalls actually referenced in the binary are allowed; others are effectively disabled.
  • Program .text can no longer directly execute syscalls; they must go through libc (or ld.so during startup).
  • Language runtimes typically call into libc, so they remain usable; “weirdo” runtimes or JITs that synthesize raw syscalls at runtime may be blocked.

Security impact and exploit techniques

  • Pinning reduces some ROP/JOP freedom:
    • You can no longer turn any reachable syscall instruction into an arbitrary syscall; only the recorded syscall number is allowed.
    • You can still misuse the permitted syscall’s arguments (e.g., still call execve if the program normally does).
  • For static binaries, attackers cannot ROP into syscalls the program never used (e.g., execve if absent), which is seen as useful containment.
  • JITs can still jump into allowed syscall thunks; browser JavaScript typically doesn’t directly syscall anyway.
  • Another mitigation effect: syscalls restricted to libc complicate ASLR bypass where an attacker only knows main binary layout and hunts for in-text syscall byte patterns.

Debate on effectiveness and threat modeling

  • Supportive views:
    • If the mitigation is cheap and doesn’t add attack surface, it’s worth shipping.
    • OpenBSD has historically deployed mitigations before classes of attacks were public (e.g., disabling hyperthreading pre-Spectre/Meltdown; “needless” randomness that later blocked DNS cache attacks).
    • Low CVE counts and mitigations that preempt vulnerabilities are cited as evidence of good intuition.
  • Skeptical views:
    • Mitigations without explicit mapping to real-world exploits/CVEs and tested exploit chains are criticized as “amorphous” and potentially “security theater.”
    • Every mitigation adds complexity and long-term maintenance cost; without a clear threat model, this may degrade security overall.
    • Some argue that if a mitigation doesn’t clearly stop current in-the-wild techniques, it’s mostly an academic exercise.
    • Others counter that focusing narrowly on “fixed CVEs” rewards past mistakes and undercounts prevented bugs.

Discovered implementation bug

  • A trivial but real bug is found in the new code:
    • Signed/unsigned mixing in a MAX macro lets an attacker drive an internal count (npins) negative, leading to under-allocation and out-of-bounds heap writes indexed by attacker-controlled syscall numbers.
    • Example exploit sketch uses crafted “pinned syscall” table entries to first poison memory, then force npins to -1, which gets clamped to a small positive value before allocation.
  • Discussion on tooling:
    • Some note that Rust-style integer/typing checks might have prevented this class of bug.
    • Others point out C compilers already have flags (e.g., sign-compare warnings, sanitizers, overflow traps) that could have caught it, but aren’t universally enabled.

C vs. Rust and OpenBSD’s constraints

  • Rust is praised for safer integer semantics and type checking, but:
    • OpenBSD has ~tens of millions of lines of C, many platforms, and very limited manpower.
    • Rewriting or deeply integrating Rust into core OS code is seen as high-risk and resource-intensive.
  • Some argue new security-focused OSes should start with Rust; others emphasize that Rust is not a silver bullet and that mature C code plus careful tools can remain viable.

Meta: OpenBSD security posture & ecosystem

  • Some participants say OpenBSD’s user base is too small for there to be a rich, empirically known “exploit meta,” so much of this is extrapolated from other platforms.
  • There’s tension between:
    • People who view OpenBSD’s slogan and reputation as overstated or misleading.
    • People who see sustained hardening work and relatively few serious holes as sufficient justification, and are puzzled by aggressive attempts to dissuade others from using it.