Redis re-implemented with SQLite
A new project, Redka, reimplements key parts of the Redis API on top of SQLite, aiming to provide familiar data-structure operations with an ACID-compliant, disk-backed store and a much smaller memory footprint. Commenters explore use cases such as large, mostly static caches and local development environments, comparing this approach to alternatives like RocksDB, ScyllaDB, and Garnet, and debating whether SQLite can meet production performance and concurrency needs. Many see value in a Redis-like interface that trades raw in‑memory speed for simplicity, introspection via SQL, and easier scaling beyond RAM limits.
Project Goals and Design
- Redka re-implements a Redis-like API on top of SQLite, exposing both an in-process Go API and a standalone RESP-compatible server.
- Stated goals: convenient data-structure API backed by SQL, with small memory footprint for large datasets, ACID transactions, and SQL access for introspection and reporting.
- Not aiming to beat Redis throughput; uses “reasonable” SQLite defaults (e.g., WAL), with a single connection to avoid write contention.
- Current coverage is mostly key–value and core structures; advanced Redis features like streams, pub/sub, HyperLogLog, and Lua are missing for now.
Use Cases and Motivations
- Strong interest from people who like Redis semantics but don’t want in-memory-only behavior or large RAM costs.
- Example use case: caching tens of millions of mostly static API responses (hundreds of GB) without paying for huge RAM instances or adding separate document DB infrastructure.
- Others want a simple local or single-node store that can be used both embedded and at small scale, or a drop-in Redis-like backend for local development and debugging.
Alternatives and Comparisons
- Suggestions for similar goals: Redis-on-RocksDB systems (e.g., kvrocks), ScyllaDB with TTLs, FoundationDB, RocksDB directly, DiskCache, S3 (including S3 Express), Varnish, memcached with disk-backed “extstore,” Garnet, and language-agnostic job queues like Faktory.
- Some argue that once you give up in-memory performance, you might as well use a regular database (Postgres or other SQL/NoSQL); others counter that Redis’s protocol, connection handling, and rich data structures are not trivially replicated.
Performance, Concurrency, and SQLite Suitability
- Debate over whether SQLite is appropriate under Redis-like loads: some predict scalability issues and recommend RocksDB/TiKV/FoundationDB; others say “SQLite isn’t production-ready” is a myth.
- Technical discussion around SQLite in WAL mode, single-writer constraints, connection pooling strategies (single connection vs read/write pools vs dedicated writer), and
busy_timeout. - Acknowledgment that Redis’s value is low-latency in-memory access, but several note that network overhead often dominates, so a local SQLite-backed Redis-compatible store can still be attractive.