Stop syncing everything

Original Article ↗ Hacker News Discussion ↗

Project concept & goals

  • Graft is presented as a low-level, page-based, transactional object store with replication, used under SQLSync/SQLite but intended to be more general (other engines like DuckDB or graph DBs are discussed as possible future fits).
  • Aim: “stop syncing everything” by syncing only the changed pages, enabling partial replication, offline operation, and edge-native architectures.

Architecture & comparisons (Cloud-Backed SQLite, Turso, others)

  • Compared to Cloud-Backed SQLite (CBS):
    • Both ship changed pages/blocks and use manifests.
    • Graft inserts a PageStore/MetaStore “middleman” instead of direct client ↔ blob storage:
      • Collates and compacts segments over time (performance, tombstone cleanup).
      • Supports “floating pages” that can be reused on rebased commits (improving concurrency).
      • Can centralize permissions; CBS assumes clients can reach blob storage directly.
    • Graft’s compressed bitset metadata aims to reduce manifest traffic with many clients.
  • Compared to Turso/libsql:
    • Turso uses traditional WAL-based physical replication; Graft’s model is new and designed for trivial partial replication.
    • Graft is not tied to SQLite conceptually; any page-addressable storage could fit if its filesystem hooks allow it.

Consistency model & conflict resolution debate

  • Design claims global strict serializability for server-side commits, with clients optionally committing locally and syncing asynchronously.
  • Several commenters challenge the terminology:
    • Confusion around “local commit” vs “global commit” and what “strictly serializable” means if offline writes can later be rejected.
    • Suggest splitting read vs write guarantees and emphasizing flexibility rather than a single strong model.
  • Conflict handling options listed: reject offline writes, rebase by replaying, automatic merges (accepting snapshot isolation/write skew), CRDTs, forking volumes, or discarding changes.
  • Skepticism:
    • Page-level conflict detection can miss read conflicts; soundness may require tracking read sets.
    • Local “successful” transactions that later vanish are effectively user-visible data loss.
    • Building robust app-level merge semantics is hard; CRDTs help but don’t cover all constraint-heavy domains (e.g., booking constraints).

Use cases, limits, and performance tradeoffs

  • Seen as promising for offline/edge/mobile workloads (e.g., React Native apps with poor connectivity; serverless/edge functions).
  • Not a good fit for high write contention on a single volume; better suited to partitioned workloads or “one volume per document/group”.
  • Page replication is simple but can ship more data than logical replication; 4KB minimum page overhead is acknowledged, with plans to optimize.
  • Concern about very slow links and huge deltas (initial sync); suggestion to fall back to server-side query execution when sync is too slow.

Permissions & security

  • Current implementation has no permissions; full access to PageStore/MetaStore.
  • Planned:
    • Fine-grained write checks in PageStore by inspecting pages.
    • Read security primarily via volume-level partitioning; more advanced “layered volumes” are floated but acknowledged as complex.
  • Commenters note that per-user or per-document volumes get tricky for Notion/Slack-style granular sharing and cross-document references.

Ecosystem, related tools, and alternatives

  • Many adjacent systems mentioned: CouchDB/PouchDB, RxDB, ElectricSQL, Replicache, InstantDB, Tinybase, CRDT frameworks (Automerge, Yjs-based systems, etc.), Turso, Cloud-Backed SQLite, and others.
  • Mixed reactions:
    • Some see a vibrant, important “offline/local-first” space (airlines, ships, field work, low-connectivity regions).
    • Others feel these abstractions are mis-layered; for many apps, simple REST APIs, local caches, “last write wins,” and occasional full reloads are sufficient.

Developer experience, clarity, and licensing

  • Diagrams and technical depth are praised; however, some readers still find it unclear “what exactly this does” and request a concrete, minimal API example.
  • Some confusion between Graft-as-general-store and the specific SQLite/SQLSync use case.
  • Project is alpha-quality; not recommended for production yet.
  • Dual MIT/Apache-2 licensing is used for Rust-ecosystem compatibility and to provide an Apache-style patent grant.