The Rust calling convention we deserve

Original Article ↗ Hacker News Discussion ↗

Rust’s lack of a dedicated, stable calling convention tailored to its own type system prompts debate over how much performance and binary-size gains are realistically available beyond the C ABI, especially for features like `Result` and complex enums. Commenters weigh tradeoffs between more aggressive register-based conventions and today’s stack-heavy, C-like patterns, noting that modern CPUs and compilers are heavily tuned to the latter and that inlining often hides call overhead anyway. The conversation branches into related issues such as dynamic linking, debugger support, interop with managed languages and Go, and Rust’s data layout choices (like niche optimization and struct field reordering), all framed by the tension between long-term ABI stability and freedom to optimize.

Rust ABI, calling conventions, and performance tradeoffs

  • Many agree C-style ABIs are suboptimal for Rust semantics (Results, enums, large structs, multiple returns).
  • Suggestions include richer register use (e.g., more return registers, flags for Result-like success/error) and different conventions for internal vs escaping functions.
  • Some argue Rust shouldn’t be locked to LLVM’s current limits; others note Rust already inserts checks (e.g., div-by-zero) to avoid LLVM UB.
  • There’s debate over per-function or type-driven calling conventions vs a single fixed ABI; debug support and dynamic linking complicate non‑C‑like approaches.
  • Several participants stress: design must be guided by measurement on real code and hardware, not aesthetics. Surprising wins from “ugly” stack-passing are reported, especially given CPUs tuned on C compiler output.

Inlining, code size, and dynamic dispatch

  • Heavy inlining reduces call overhead, but not all calls can be inlined (e.g., dynamic dispatch such as dyn Trait, cross-crate boundaries).
  • Cheaper calls could allow less aggressive inlining, improving code size and compile times.
  • For small/embedded targets, current Result-heavy patterns are seen as code-size-unfriendly; ideas include returning discriminants in registers while large variants live on the stack.

Struct layout, niches, and Option

  • Rust already reorders fields (unless repr forbids it); there’s an unstable flag to detect layout assumptions.
  • A recurring theme is generalizing niche optimizations to reduce enum and Option size and padding.
  • Example complaint: many Option<T> fields inside a struct waste space because each carries its own discriminant; manual bitpacking with MaybeUninit can be more compact but loses direct &Option<T> borrows.

Interop and managed vs unmanaged memory

  • Rust–Go interop is commonly done via extern "C" and cgo; described as workable but tedious due to manual type marshaling.
  • Strong disagreement over whether mixing managed and unmanaged languages is “usually unwise”: some say it’s broadly problematic, others say it’s routine and efficient in ecosystems designed for it (.NET, Swift/ARC).
  • Techniques like pinning, GC roots, and finalizers are mentioned as necessary but fiddly.

MLIR and related tangents

  • MLIR is described as a framework for intermediate “dialects” and progressive lowering, not directly relevant to Rust calling conventions.
  • It does have a linalg dialect for common linear algebra ops, but that’s treated as orthogonal.

Site and presentation details

  • Several comments note the blog’s skewed headings and minimap; some find it creative, others think it harms readability or accessibility.