OpenAI’s WebRTC problem

Original Article ↗ Hacker News Discussion ↗

Critiques of OpenAI’s use of WebRTC for voice interactions argue that the protocol’s real‑time, packet‑dropping behavior and complex P2P heritage make it a poor fit for cloud‑based “voice AI,” where capturing every bit of audio can matter more than ultra‑low latency. Others counter that WebRTC’s maturity, built‑in echo cancellation, codec support, and browser ubiquity still outweigh its quirks, and that alternatives like WebSockets or WebTransport simply move the same complexity elsewhere. Underneath the technical details is a broader question of what users value most in voice interfaces with LLMs: conversational fluidity and instant responses, or slightly higher latency in exchange for greater reliability and accuracy.

WebRTC complexity and design trade‑offs

  • Many commenters find WebRTC painful: huge protocol surface (SDP, ICE, STUN/TURN, RTP, many RFCs), tricky handshakes, and over‑engineering for an internet that “doesn’t exist” (heavy P2P/NAT assumptions).
  • Others argue the complexity reflects real‑time media reality: NAT traversal, jitter buffers, congestion control, codecs, encryption, echo cancellation, and browser ubiquity are hard problems WebRTC already solves.
  • There’s frustration that even simple client–server scenarios must still go through the full P2P-style signaling dance.
  • Some note that WebRTC has tuning knobs (FEC, jitter buffer params, PLC, etc.) and that many glitches in OpenAI’s voice seem more like inference/model issues than transport.

Latency vs accuracy in voice AI

  • One camp: for voice AI, it’s more important not to drop audio than to be perfectly real time. LLMs can handle gaps; users might accept ~200–500 ms extra delay for correctness.
  • Opposing camp: in human–AI voice conversations every millisecond counts. Extra 200–500 ms on top of model and tool latencies makes interactions feel broken, increases interruptions, and kills engagement.
  • Several people running large voice systems report that going from ~1.2–1.5 s to ~700 ms turn latency dramatically improved usability; they are spending heavily to shave off another ~100–200 ms.
  • Some call the “latency vs reliability” framing a false dichotomy: TCP/WebSockets can stream audio immediately while still retransmitting losses, with the main challenge moving to jitter buffering (needed only on the human‑output side).

Alternatives: WebTransport, WebSockets, QUIC, MoQ

  • WebTransport + QUIC, RTP-over-QUIC, HTTP/2, and MoQ are discussed as ways to get WebRTC-like behavior with more control and without the P2P baggage.
  • Downsides: WebTransport server setup is non‑trivial, support is still maturing; MoQ is promising but niche; rolling your own over WebSockets forces you to re‑implement jitter, congestion, AEC, etc.
  • Some see a future in QUIC-based media stacks; others doubt MoQ’s alignment with end‑to‑end problems like echo cancellation and device DSP.

Real‑world reports and UX observations

  • Some practitioners report good results with WebRTC-based stacks (e.g., LiveKit, Daily, mesh/managed clouds, pipecat) for large‑scale voice agents, despite setup pain.
  • Others find non‑WebRTC systems (e.g., persistent HTTP/2 connections similar to smart speakers) adequate and simpler for server‑centric voice AI.
  • Several note that users strongly feel latency in voice; some argue users implicitly trade accuracy for responsiveness, while others insist correctness must remain primary, especially for high‑stakes scenarios.
  • There is side discussion about IPv6 potentially simplifying routing, and about browser APIs (including WebRTC) having many undocumented edge cases and weak timestamp semantics.