Progressive JSON

Original Article ↗ Hacker News Discussion ↗

Concept and relation to React Server Components (RSC)

  • Thread converges on the idea that the post is really an explanation of the RSC wire protocol, with “Progressive JSON” as an illustrative device rather than a new format.
  • RSC uses a stream of JSON-like chunks with placeholders (“holes”) that correspond to parts of the UI tree; as data becomes available, later chunks “fill” those holes.
  • Key idea: the data being streamed is the UI itself, so outer JSON corresponds to outer UI, enabling outside‑in, progressive reveal with intentionally placed loading states (e.g. via Suspense), not arbitrary layout jumps.

Alternatives, prior art, and related formats

  • Multiple commenters note existing streaming or incremental formats: JSON Lines / ndjson, JSON Patch, HTTP/2 multiplexing, GraphQL @defer/@stream, JSON API links, HATEOAS, CSV-first pipelines, SAX‑style JSON parsers, protobuf, Cap’n Proto, DER‑like streaming encodings, and Mendoza-style JSON diffs.
  • There’s debate whether ndjson/jsonl are effectively the same; ndjson has a formal spec.
  • Some suggest simpler patterns: line‑delimited JSON + patches, or ordered keys so “big” arrays come last.

Use cases and perceived benefits

  • Latency-sensitive UI: dashboards, comment threads, slow DB queries, poor mobile networks, and systems where cached and uncached data mix.
  • Streaming UI updates from LLMs and AI tool calls; Pydantic‑based streaming validation is mentioned.
  • Tree/graph data: several comment chains explore breadth‑first or parent‑vector encodings and ways to send string tables and node batches for fast, incremental rendering.

Skepticism, complexity, and “overengineering”

  • Many argue most apps shouldn’t need this; better to:
    • Split data across multiple endpoints,
    • Use pagination, cursors, or resource‑level APIs,
    • Improve DB queries, caching, and architecture.
  • Concerns include: breaking JSON semantics (order, validity), debugging partially failed streams, subtle UI bugs from reordering, and leaky abstractions that require deep protocol understanding.
  • Some see this as solving self‑inflicted problems from large SPAs and overgrown frontends, preferring simpler stacks (traditional SSR, small SPAs, LiveView‑style systems).

GraphQL, REST, and BFF discussion

  • GraphQL is cited as addressing under/over‑fetching and having similar streaming semantics; others call it heavy, debt‑creating, or awkward to maintain.
  • REST/HATEOAS and “backend‑for‑frontend” patterns are presented as alternative ways to let the client choose follow‑up fetches instead of one huge, progressively streamed payload.