Real-time audio programming 101: time waits for nothing (2011)

Original Article ↗ Hacker News Discussion ↗

Real-time audio programming demands not just speed but deterministic, low-jitter execution, because even a rare 10 ms overrun can cause audible glitches that are far more jarring than dropped video frames. Commenters delve into how buffer sizes, OS scheduling, CPU/cache behavior, buses, and plugin chains constrain the true processing budget, why typical desktop systems are poorly suited to hard real-time guarantees, and where extra buffering or latency is acceptable (e.g., games, conferencing) versus unacceptable (live instruments, large shows). They also trade techniques and tools—from lock-free patterns and LUTs to specific DSP frameworks and measurement methods—for reducing latency risk while acknowledging that absolute guarantees are rarely achievable outside dedicated hardware or RTOS environments.

Real‑time audio constraints and shared pipelines

  • In practice, audio code often runs as one “module” in a shared bus / pipeline / graph, not alone with the device.
  • Your 5.6 ms buffer is a global deadline for all processing on that path, so each module must leave time for others.
  • Standalone apps that control the entire input→output chain are simpler but less common than plugin / DAW / game-style graphs.

Determinism, jitter, and deadlines

  • Real-time audio needs deterministically fast execution: consistent worst-case time, not just good averages.
  • Jitter matters because it sets how small buffers can be before glitches; if deadlines are always met, residual jitter is irrelevant.
  • Distinction between hard vs soft real time is emphasized: audio is usually soft RT, but glitches can still be very costly.

Optimizations and modern CPU behavior

  • Debate over micro-optimizations: divisions removal, LUTs, CPU-specific tricks.
  • Some argue these are now mostly for embedded DSP; main problem on desktops is OS scheduling.
  • Others say LUTs are still common and can be faster for complex functions, especially with small, cache-friendly tables plus interpolation.

Latency tolerances across use cases

  • Pro-audio / live instruments need very low latency and especially low jitter; small buffer sizes and strict coding rules apply.
  • Video conferencing shows that clever buffering and time-warping can recover from missed deadlines without user repeats.
  • Games often tolerate 50–100+ ms audio latency via buffering; users rarely notice as much as they would a musical instrument delay.
  • There is disagreement about how sensitive typical users really are to MIDI and instrument latency.

Measuring and engineering for low jitter

  • Main nondeterminism sources cited: OS scheduling, power management, caches, and inter-core effects.
  • Suggested approach: measure end-to-end scheduling jitter, shrink buffers until glitches appear, and use RT priorities / kernels where possible.
  • Kernel tools like cyclictest are mentioned for baseline timing; real applications still need their own instrumentation.

Tools, libraries, and platforms

  • Suggested DSP tools: Pure Data, Max/MSP, SuperCollider, Csound, ChucK, AudioMulch, JSFX, and various Windows DSP / virtual device solutions.
  • Rust’s cpal is praised for cross-platform audio but criticized as cumbersome; wrappers and example projects are shared.
  • WebAudio + WASM can achieve near-native DSP, but browsers’ GC, scheduling, and security mitigations introduce rough edges.

Graphics/UI and broader “real time” discussion

  • Audio programming is seen as more rigorously real-time than graphics/UI; tiny audio glitches are more noticeable than occasional frame drops.
  • Some wish for UI toolkits with real-time-like guarantees, especially for high-refresh displays, but many question the payoff.
  • Participants stress that “real time” means meeting deadlines (hard/soft), not merely “fast,” contrasting audio/RTOS work with typical app development.