WebSockets cost us $1M on our AWS bill

Original Article ↗ Hacker News Discussion ↗

A startup that records video meetings via headless Chromium found that piping raw 1080p frames over loopback WebSockets between processes was doubling CPU usage and inflating its AWS compute bill by about $1M per year. Commenters dissect why WebSockets are a poor fit for high‑bandwidth, low‑latency interprocess communication, highlighting overhead from fragmentation, masking, and multiple memory copies, and argue that shared memory or existing IPC mechanisms would have been more appropriate. The exchange also touches on broader themes: the pitfalls of “make it work first” architectures at scale, how easily raw video overwhelms modern systems, and whether cloud platforms like AWS are cost-effective for such workloads compared with dedicated servers or GPU-based approaches.

Overall theme

  • Thread largely agrees the issue was CPU cost from inefficient IPC of raw video over WebSockets on loopback, not AWS data transfer.
  • Many see it as a classic “PoC in production” that only becomes a problem at scale; others argue experienced systems engineers would have avoided it.

Architecture & root cause

  • Recall used headless Chromium bots to join third‑party video calls, capture rendered frames, then send raw 1080p video via WebSockets to another process for encoding and analysis.
  • Profiling showed heavy overhead from WebSocket fragmentation, masking, multiple memcpy operations, and general message framing on huge frames.
  • They replaced this with a shared‑memory ring buffer, significantly reducing CPU usage and thus AWS compute cost.

Debate on design choices

  • Many criticize decoding and compositing in Chromium, then shipping uncompressed frames instead of:
    • Keeping streams compressed longer.
    • Tapping into underlying codecs, WebRTC, or GPU pipelines.
    • Using existing IPC/shm mechanisms (/dev/shm, Mojo, iceoryx2, Redis, etc.).
  • Defenders point out constraints:
    • They’re scraping many meeting platforms that don’t expose compressed streams publicly.
    • Reverse‑engineering or negotiating private APIs for each provider would be brittle and slow.
    • A naïve but robust solution let them validate the business first.

Technical nitpicks & disagreements

  • Some say their discussion of MTU, fragmentation, and bandwidth shows shallow systems knowledge; others argue that since they ended up on shared memory, further TCP tuning is moot.
  • Disagreement over how critical zero‑copy really is at 1080p given typical server memory bandwidth.
  • Some question their use of CPU instead of GPU for rendering/encoding, given million‑dollar scale.
  • There’s debate about lock‑free shared‑memory designs, atomics, and memory ordering, but general agreement that shared memory is the right class of solution.

AWS, cost, and framing

  • Several commenters find the title misleading, expecting AWS egress or API Gateway WebSocket charges; the real issue is CPU hours on EC2.
  • Long side‑discussion compares AWS prices vs dedicated servers/colos (e.g., Hetzner), with claims of large cost deltas but no consensus on trade‑offs.
  • Some praise the transparency and postmortem; others think it highlights a lack of low‑level expertise.