In Search of a Faster SQLite

Original Article ↗ Hacker News Discussion ↗

Efforts to build a faster, cloud-friendly alternative to SQLite—most notably Turso’s Rust-based “Limbo” using async I/O and io_uring—are drawing mixed reactions. Commenters note that benchmark gains mostly appear in extreme tail latencies and specific multi-tenant serverless scenarios, while SQLite already performs extremely well for typical single-host workloads and benefits from a famously rigorous test suite. The conversation also surfaces concerns about security, durability, and incomplete documentation in the new ecosystem, along with broader debates over SQLite’s closed contribution model, alternative test strategies, and practical deployment patterns using tools like AWS Lambda and S3.

Benchmark results and practical relevance

  • Paper shows big improvements only at extreme tail latencies (p999+); p90–p99 similar to SQLite.
  • Workload (fetching 100 rows without filters/sorts) is seen as too trivial and I/O-light to be broadly meaningful.
  • Several commenters conclude the gains mainly matter in heavily contended, multitenant scenarios, not in single-node monoliths.

Serverless / edge patterns with SQLite

  • Multiple experiences using SQLite databases prebuilt and stored in S3 or container images for Lambda/“serverless” functions.
  • Pattern: periodic job parses source data → writes/indices SQLite → uploads to S3 or bakes into image; functions download or read local copy for very fast lookups.
  • Local /tmp and persistent global scope in Lambda are highlighted as useful caches.

Consistency, S3 scaling, and “thundering herd”

  • Clock-based freshness checks can be unsafe for strict consistency; Etags or explicit version files are suggested instead.
  • Some report Lambda-level throttling and S3 rate limits when many functions pull the same DB at once; approaches include bundling DB in images or staggering rollout.
  • Discussion of S3 “prefix” sharding, request limits, and the danger of hot date-based prefixes.

io_uring, async I/O, and safety

  • Question whether major cloud/edge providers enable io_uring due to past vulnerabilities and SELinux policy complexity; currently unclear and may vary by platform.
  • Async I/O mainly helps concurrency and thread count, not individual query latency, though batching via ring buffers can cut syscall overhead.
  • Concern about trading off simplicity and safety; others argue async I/O is overdue on Linux and can be robust (e.g., other databases using it).

Rewriting SQLite in Rust and project politics

  • Limbo (Rust rewrite) and libSQL raise debate:
    • Pro: open-source, MIT-licensed, adding “cloud-capable” features, building in the open, targeting SQLite API/file compatibility and strong testing (Deterministic Simulation Testing + external tools).
    • Con: seen by some as hype-driven, overlapping with SQLite’s domain without matching its test rigor yet, and as marketing itself as a “better SQLite” before parity.
  • There is criticism of how SQLite’s ethics/code-of-conduct framing is contrasted with the new projects’ code-of-conduct; others defend the new projects’ approach.

Testing, correctness, and TH3

  • SQLite’s extensive, partly proprietary test harness (TH3) is viewed as a high bar.
  • Suggestions: license TH3 or rely on open test suites plus DST.
  • Some doubt whether new testing strategies will match SQLite’s long-proven reliability; others are optimistic but note it “remains to be seen.”

SQLite governance and “open source”

  • Debate over whether SQLite is “open source but not open contribution”:
    • It’s public domain and accepts only carefully vetted, public-domain-dedicated contributions, often rewritten.
    • Some see lack of GitHub-style PRs as non-OSS in spirit; others argue open contribution is not required for user freedom and that this model is reasonable for such critical software.

SQL vs key–value stores

  • Several argue KV stores are awkward for relational/graph-like data and are better as internal building blocks; SQL offers better developer experience for table-like data.
  • Counterpoint: SQL makes nested/0NF structures (tables within tables, hierarchical comments) awkward; requires joins, CTEs, views, or JSON aggregation.
  • Others respond that relational modeling intentionally avoids binding to a single access pattern and that such transformations belong in queries/views.

Other databases and async efforts

  • Anecdotes that SQLite can outperform client/server RDBMS (e.g., Postgres, MSSQL) in single-machine setups due to lack of network and IPC overhead and effective page caching.
  • Pointer to past and ongoing work to add async I/O and pluggable storage managers in Postgres, suggesting a broader interest in async storage backends, not just for SQLite.