I designed my own fast game streaming video codec – PyroWave

Original Article ↗ Hacker News Discussion ↗

Overall reception & target use cases

  • Many commenters praise the codec’s clarity and speed, seeing it as ideal for local game streaming (e.g., Sunshine/Moonlight, LAN setups) where bandwidth is plentiful but latency is critical.
  • Some readers find it particularly useful for research or as an educational example of codec design focused on known signal characteristics (games).

Latency vs bandwidth trade-offs

  • Strong agreement that for in‑home or private networks, sacrificing bandwidth (e.g., ~100–200 Mbps) to slash encode latency is a good trade.
  • Several argue that in practice, encode/decode and display latency dominate, not network latency; others counter that for internet/cloud streaming, network remains the main bottleneck.
  • Display processing latency (10–100 ms) is called out as a major remaining issue once the pipeline is otherwise optimized.

Game‑engine cooperation & motion vectors

  • Long subthread debates using engine‑provided motion vectors and depth for better motion prediction and/or client‑side reprojection.
  • Some assert most modern 3D games already have motion/vector buffers (for TAA, DLSS, motion blur), others dispute how universal this is.
  • There’s disagreement on how directly 3D engine motion vectors map to codec motion compensation, and whether they meaningfully reduce encoder work versus existing heuristics.
  • Ideas extend to: separate encoding of HUD/overlays, RGBD streaming with client reprojection, VR‑style late reprojection for UI and camera, and sensor‑assisted encoding.

Comparison to existing codecs and hardware

  • People suggest comparisons to H.264 “zero‑latency” modes, AV1 RTC, JPEG‑XS, VC‑2, NDI, QOI‑based video, and HTJ2K.
  • Some emphasize that mainstream codecs already have low‑latency/low‑complexity profiles and hardware, especially NVENC‑style ASICs; others note that even “<10 ms” hardware encode is still large compared to the new approach (~0.13 ms).
  • Discussion clarifies that GPU hardware encoders are typically separate blocks, though there’s confusion about how “dedicated” they are and about driver‑imposed limits.

Patents and commercial concerns

  • JPEG‑XS and similar standards are cited as low‑latency but patent‑encumbered, considered both “safer” (clear licensing) and also a form of protection racket.
  • Some warn that any new codec risks “improvement patents,” suggesting proactive research and publication to fence off the space.

Alternative architectures & speculative ideas

  • Threads explore streaming graphics API command streams or scene data instead of video, but bandwidth and texture streaming make this questionable for thin clients.
  • More speculative/whimsical ideas include foveated encoding based on eye‑tracking and LLM‑based “text-to-video” game streaming.