I’m leaving Redis for SolidQueue

Original Article ↗ Hacker News Discussion ↗

Operational simplicity vs Redis overhead

  • Many welcome using SolidQueue (or similar DB-backed queues) to remove Redis from the stack, especially for small/medium Rails apps where Redis mainly serves background jobs.
  • Core argument: you already operate Postgres/MySQL; if the database can handle queuing “well enough,” adding Redis is extra cost (deployment, HA, monitoring, auth, networking).
  • Others push back that those Redis “costs” mostly mirror what you must do for Postgres anyway; the real benefit is consolidation, not that Redis is uniquely hard.

Queue DB vs application DB

  • Debate over whether the job queue should share the same database as the main app:
    • Pro-same-DB: simpler architecture, transactional enqueueing with business data, fewer reconciliation headaches when one system fails.
    • Pro-separate-DB: avoids the queue overwhelming the primary DB; easier backup/restore semantics (e.g., not replaying old jobs like emails).
  • Some argue coupling business and queue state tightly (same DB, same transactions) makes later migration to a separate queue harder.
  • SolidQueue can use a separate DB config by default but also supports sharing the app DB.

Scalability and performance of DB-backed queues

  • Several reports of Postgres-based queues hitting performance walls at modest rates (e.g., 2–5k jobs/min) and teams moving to Redis-based systems like BullMQ to stop tuning PG.
  • Others claim such loads should be trivial for a properly tuned database and suspect misconfiguration or lack of batching.
  • Elixir’s Oban benchmark (1M jobs/min) is cited as evidence DB queues can scale, but critics note it relies heavily on batching and is far from typical usage.
  • LISTEN/NOTIFY in Postgres is called out as a bottleneck at high volume; systems work around this by debouncing notifications or relying more on polling.
  • Replication overhead is mentioned as a real scaling limit even if a single node can churn through rows quickly.

Feature set, maturity, and Rails ecosystem

  • SolidQueue praised for making “Rails 8 monolith” setups very simple and integrated (Active Job, unified “Solid” features).
  • Concerns: still “early days,” some concurrency bugs reported, silent failures when DB pools fill, and a relatively weak UI compared to Sidekiq or GoodJob.
  • GoodJob (Postgres-only) is viewed as more mature and feature-rich (e.g., batches, better UI), but its use of PG-specific features and session requirements can conflict with some connection poolers.
  • Rails’ desire to keep SolidQueue SQL portable across databases (e.g., MySQL) limits engine-specific optimizations; some see this as a downside for Postgres-heavy shops.

Payload size and job design

  • Consensus from multiple commenters that large job payloads are a poor fit for DB-backed queues and often for Redis too.
  • Recommended pattern: store large data elsewhere (DB tables or object storage) and enqueue only identifiers; many frameworks’ docs already encourage this.
  • One commenter explicitly benchmarked Redis vs Postgres vs SQLite and found Redis clearly faster for large JSON payload jobs.

When Redis (or other queues) still shines

  • Several argue that if low-latency, high-throughput, or very large-scale workloads are central, a purpose-built queue (Redis, SQS, Kafka, etc.) remains the right tool.
  • Redis is still valued for:
    • Caching (multi-layer caches in large SaaS setups).
    • Rate limiting and high-frequency counters where DB roundtrips are too slow or too costly.
    • Pub/sub or broadcast to many services with simple ops and language-agnostic clients.
  • Others suggest that if you’re already on a major cloud, managed queues like SQS/PubSub are often a better default than either Redis or the primary DB.