Redis re-implemented with SQLite

Original Article ↗ Hacker News Discussion ↗

Project Goals and Design

  • Redka re-implements a Redis-like API on top of SQLite, exposing both an in-process Go API and a standalone RESP-compatible server.
  • Stated goals: convenient data-structure API backed by SQL, with small memory footprint for large datasets, ACID transactions, and SQL access for introspection and reporting.
  • Not aiming to beat Redis throughput; uses “reasonable” SQLite defaults (e.g., WAL), with a single connection to avoid write contention.
  • Current coverage is mostly key–value and core structures; advanced Redis features like streams, pub/sub, HyperLogLog, and Lua are missing for now.

Use Cases and Motivations

  • Strong interest from people who like Redis semantics but don’t want in-memory-only behavior or large RAM costs.
  • Example use case: caching tens of millions of mostly static API responses (hundreds of GB) without paying for huge RAM instances or adding separate document DB infrastructure.
  • Others want a simple local or single-node store that can be used both embedded and at small scale, or a drop-in Redis-like backend for local development and debugging.

Alternatives and Comparisons

  • Suggestions for similar goals: Redis-on-RocksDB systems (e.g., kvrocks), ScyllaDB with TTLs, FoundationDB, RocksDB directly, DiskCache, S3 (including S3 Express), Varnish, memcached with disk-backed “extstore,” Garnet, and language-agnostic job queues like Faktory.
  • Some argue that once you give up in-memory performance, you might as well use a regular database (Postgres or other SQL/NoSQL); others counter that Redis’s protocol, connection handling, and rich data structures are not trivially replicated.

Performance, Concurrency, and SQLite Suitability

  • Debate over whether SQLite is appropriate under Redis-like loads: some predict scalability issues and recommend RocksDB/TiKV/FoundationDB; others say “SQLite isn’t production-ready” is a myth.
  • Technical discussion around SQLite in WAL mode, single-writer constraints, connection pooling strategies (single connection vs read/write pools vs dedicated writer), and busy_timeout.
  • Acknowledgment that Redis’s value is low-latency in-memory access, but several note that network overhead often dominates, so a local SQLite-backed Redis-compatible store can still be attractive.