Jetstream: Shrinking the AT Protocol Firehose by >99%

Original Article ↗ Hacker News Discussion ↗

Developers are evaluating Jetstream, an unofficial service that dramatically reduces the bandwidth of Bluesky’s AT Protocol “firehose” by converting low-level events into simplified JSON and optionally compressing them with zstd and a custom dictionary. Supporters see it as a practical way to cut costs, ease onboarding for third‑party developers, and delay complex scaling steps like sharding, while critics worry about weakened cryptographic verification, added complexity, and reliance on centralized relays. The exchange also compares AT Protocol’s relay-centric model to ActivityPub’s more peer-to-peer approach, probing whether large‑scale social networks can remain truly federated while still offering powerful global discovery and analytics.

Jetstream design and compression choices

  • Jetstream is an unofficial optimization layer over the AT Protocol firehose, converting protocol events into simpler JSON for easier consumption.
  • Two modes exist: uncompressed JSON and Zstandard (zstd) with a custom dictionary; compressed mode shrinks bandwidth by >99% according to the article.
  • Some argue there’s no need for an uncompressed stream in a new protocol; others say uncompressed is simpler for local/testing use and for generic tools (websocat, pipes) that expect plain JSON.
  • Compressed mode requires clients to know the custom zstd dictionary, which is seen as a usability barrier for ad‑hoc or language‑agnostic consumers.

WebSockets vs SSE and compression strategy

  • One camp suggests Server-Sent Events plus gzip for simplicity and standard tooling.
  • Others counter that zstd with a shared dictionary gives better compression at this scale and enables “compress once, serve many,” reducing server CPU.
  • WebSocket tooling is described as more mature and ergonomic than SSE in most languages.

Security, verification, and “throwing features away”

  • Critics worry Jetstream trades away important security and integrity properties from the full firehose (signatures and MST proofs) for performance and ease of use.
  • The full firehose allows verifying that repo updates are signed and non‑omitting; Jetstream’s reduced stream omits these guarantees.
  • Some ask whether parts of these security features could be reintroduced while keeping Jetstream’s simplicity.

Federation, centralization, and relays vs ActivityPub

  • A major thread debates whether AT Protocol is “really federated” if large, central relays are practically required for discovery and aggregated views.
  • Defenders say:
    • PDS (personal data servers) are the true sources of truth; relays are “trustless aggregators” and an optimization.
    • Clients can in principle fetch directly from PDSs (pull-based model), similar to ActivityPub, but then inherit ActivityPub-like discovery problems.
  • Critics respond that in practice the main relay is central and critical, and that running without it is unsupported or impractical.

Scale, cost, and performance

  • Pre-surge firehose volume was ~24 GB/day; a surge took it to ~232 GB/day, with relay peak output >8.5 Gbps to 250+ subscribers.
  • Some argue the raw bandwidth (tens of Mbps) is cheap, so dev time on compression seems overkill.
  • Others note:
    • Bandwidth multiplies across many subscribers; compression extends how long a single, unsharded firehose remains viable.
    • Hobbyists can reportedly run full relays for roughly €75–$150/month.
    • Jetstream makes the ecosystem more approachable for small operators and developers.

Naming and terminology confusion

  • Multiple commenters initially misread “AT Protocol” as Hayes modem AT commands and “Jetstream” as NATS JetStream, highlighting naming collisions and jargon assumptions.
  • Some feel the article should have introduced AT Protocol more clearly for a general audience.