Jetstream: Shrinking the AT Protocol Firehose by >99%

Original Article ↗ Hacker News Discussion ↗

Jetstream design and compression choices

  • Jetstream is an unofficial optimization layer over the AT Protocol firehose, converting protocol events into simpler JSON for easier consumption.
  • Two modes exist: uncompressed JSON and Zstandard (zstd) with a custom dictionary; compressed mode shrinks bandwidth by >99% according to the article.
  • Some argue there’s no need for an uncompressed stream in a new protocol; others say uncompressed is simpler for local/testing use and for generic tools (websocat, pipes) that expect plain JSON.
  • Compressed mode requires clients to know the custom zstd dictionary, which is seen as a usability barrier for ad‑hoc or language‑agnostic consumers.

WebSockets vs SSE and compression strategy

  • One camp suggests Server-Sent Events plus gzip for simplicity and standard tooling.
  • Others counter that zstd with a shared dictionary gives better compression at this scale and enables “compress once, serve many,” reducing server CPU.
  • WebSocket tooling is described as more mature and ergonomic than SSE in most languages.

Security, verification, and “throwing features away”

  • Critics worry Jetstream trades away important security and integrity properties from the full firehose (signatures and MST proofs) for performance and ease of use.
  • The full firehose allows verifying that repo updates are signed and non‑omitting; Jetstream’s reduced stream omits these guarantees.
  • Some ask whether parts of these security features could be reintroduced while keeping Jetstream’s simplicity.

Federation, centralization, and relays vs ActivityPub

  • A major thread debates whether AT Protocol is “really federated” if large, central relays are practically required for discovery and aggregated views.
  • Defenders say:
    • PDS (personal data servers) are the true sources of truth; relays are “trustless aggregators” and an optimization.
    • Clients can in principle fetch directly from PDSs (pull-based model), similar to ActivityPub, but then inherit ActivityPub-like discovery problems.
  • Critics respond that in practice the main relay is central and critical, and that running without it is unsupported or impractical.

Scale, cost, and performance

  • Pre-surge firehose volume was ~24 GB/day; a surge took it to ~232 GB/day, with relay peak output >8.5 Gbps to 250+ subscribers.
  • Some argue the raw bandwidth (tens of Mbps) is cheap, so dev time on compression seems overkill.
  • Others note:
    • Bandwidth multiplies across many subscribers; compression extends how long a single, unsharded firehose remains viable.
    • Hobbyists can reportedly run full relays for roughly €75–$150/month.
    • Jetstream makes the ecosystem more approachable for small operators and developers.

Naming and terminology confusion

  • Multiple commenters initially misread “AT Protocol” as Hayes modem AT commands and “Jetstream” as NATS JetStream, highlighting naming collisions and jargon assumptions.
  • Some feel the article should have introduced AT Protocol more clearly for a general audience.