Real-time audio programming 101: time waits for nothing (2011)

Original Article ↗ Hacker News Discussion ↗

Real‑time audio constraints and shared pipelines

  • In practice, audio code often runs as one “module” in a shared bus / pipeline / graph, not alone with the device.
  • Your 5.6 ms buffer is a global deadline for all processing on that path, so each module must leave time for others.
  • Standalone apps that control the entire input→output chain are simpler but less common than plugin / DAW / game-style graphs.

Determinism, jitter, and deadlines

  • Real-time audio needs deterministically fast execution: consistent worst-case time, not just good averages.
  • Jitter matters because it sets how small buffers can be before glitches; if deadlines are always met, residual jitter is irrelevant.
  • Distinction between hard vs soft real time is emphasized: audio is usually soft RT, but glitches can still be very costly.

Optimizations and modern CPU behavior

  • Debate over micro-optimizations: divisions removal, LUTs, CPU-specific tricks.
  • Some argue these are now mostly for embedded DSP; main problem on desktops is OS scheduling.
  • Others say LUTs are still common and can be faster for complex functions, especially with small, cache-friendly tables plus interpolation.

Latency tolerances across use cases

  • Pro-audio / live instruments need very low latency and especially low jitter; small buffer sizes and strict coding rules apply.
  • Video conferencing shows that clever buffering and time-warping can recover from missed deadlines without user repeats.
  • Games often tolerate 50–100+ ms audio latency via buffering; users rarely notice as much as they would a musical instrument delay.
  • There is disagreement about how sensitive typical users really are to MIDI and instrument latency.

Measuring and engineering for low jitter

  • Main nondeterminism sources cited: OS scheduling, power management, caches, and inter-core effects.
  • Suggested approach: measure end-to-end scheduling jitter, shrink buffers until glitches appear, and use RT priorities / kernels where possible.
  • Kernel tools like cyclictest are mentioned for baseline timing; real applications still need their own instrumentation.

Tools, libraries, and platforms

  • Suggested DSP tools: Pure Data, Max/MSP, SuperCollider, Csound, ChucK, AudioMulch, JSFX, and various Windows DSP / virtual device solutions.
  • Rust’s cpal is praised for cross-platform audio but criticized as cumbersome; wrappers and example projects are shared.
  • WebAudio + WASM can achieve near-native DSP, but browsers’ GC, scheduling, and security mitigations introduce rough edges.

Graphics/UI and broader “real time” discussion

  • Audio programming is seen as more rigorously real-time than graphics/UI; tiny audio glitches are more noticeable than occasional frame drops.
  • Some wish for UI toolkits with real-time-like guarantees, especially for high-refresh displays, but many question the payoff.
  • Participants stress that “real time” means meeting deadlines (hard/soft), not merely “fast,” contrasting audio/RTOS work with typical app development.