The real realtime preemption end game

Original Article ↗ Hacker News Discussion ↗

Efforts to make the Linux kernel fully preemptible for real-time workloads highlight how hard it is to guarantee bounded latency in a large, general-purpose OS, especially around something as seemingly simple as logging (`printk`). Commenters contrast Linux’s soft real-time ambitions with hard real-time needs in fields like avionics, robotics, and industrial control, arguing that while true safety‑critical systems will still favor small RTOSes or microkernels, mainline real-time Linux can significantly improve latency and jitter for applications such as audio, video, and some embedded control. Many see this work less as a bid to replace dedicated RTOS stacks and more as “good hygiene” that makes Linux behave more predictably under load while preserving its broad hardware and software ecosystem.

Realtime logging & printk

  • Printing from kernel and RT contexts is hard: synchronous logging can block on slow I/O and break realtime guarantees.
  • Common patterns: ring buffers with overwrites or drops, low-priority flusher threads, and accepting data loss under load.
  • There’s debate between dropping newest vs overwriting oldest; both trade data fidelity for forward progress.
  • Linux printk changes for RT are tricky; backports have even introduced deadlocks in some distros.

Logging, CAP theorem & “guaranteed delivery”

  • Many commenters link logging design to CAP-style tradeoffs: you can’t have both guaranteed delivery and zero impact on availability.
  • “Guaranteed delivery” logging is seen as appropriate for audit/billing, but unacceptable if it can halt services.
  • Best-effort schemes: local queues, async network logging, mmap’d files, UDP to local aggregators; all eventually drop logs under extreme conditions.
  • Several point out that “guaranteed” often really means “very unlikely to lose,” bounded by memory, disk, or network partitions.

Hard vs soft realtime, and where RT Linux fits

  • Strong distinction: hard realtime (missed deadlines = system failure/safety risk) vs soft realtime (occasional misses = degradation).
  • Consensus that Linux, even with PREEMPT_RT, is for soft realtime: audio, video, robotics control layers, industrial equipment, CNC, etc.
  • Safety‑critical avionics/medical/automotive often use small RTOSes or MCUs; Linux may sit alongside as a higher-level controller or UI.
  • Some see potential displacement of proprietary RTOS stacks; others argue certification complexity and size make Linux unsuitable for many safety‑critical roles.

Hardware limits to realtime

  • Modern CPUs add unbounded jitter sources: caches, MMUs, complex buses, system management mode, opaque memory controllers.
  • “Realtime” is framed as “bounded time”; if you can’t bound memory access or interrupt latency, you don’t truly have hard realtime.
  • Simpler cores (8051, Cortex‑M/R) and techniques like disabling caches, pinning memory/cores, and avoiding paging are cited as typical hard-RT approaches.

Microkernels, alternative RTOSes & architectures

  • Microkernels like QNX, L4/seL4, and capability-based systems are praised for small, bounded kernels with drivers in userspace.
  • Others note Linux’s vast driver ecosystem as a huge practical advantage; microkernels often end up running Linux as a guest anyway.
  • Xenomai and dual-kernel approaches are reported to deliver near–hard-RT behavior by running Linux as a lower-priority task.

Impact on everyday users

  • Expected benefits for general users are modest but real: lower latency and jitter for audio, gaming input, video conferencing, and better behavior under load.
  • Some hope distros will eventually ship RT-enabled kernels by default, as throughput impact is described as small in many workloads.