River: A fast, robust job queue for Go and Postgres

Original Article ↗ Hacker News Discussion ↗

A new Go library called River proposes using PostgreSQL as a transactional job queue, tying job creation to the same database transactions that modify application data. Supporters argue this “single dependency” approach simplifies architecture, improves correctness through atomic enqueueing, and scales well enough for most workloads, citing similar systems like Oban as proof. Critics counter that relational databases make poor high-scale queues and prefer dedicated services or HTTP-based task systems, raising concerns about performance, long-running jobs, and the trade-offs versus tools like Redis, Kafka, Temporal, or cloud task queues.

Design: Postgres-Backed, Transactional Job Queue

  • Core idea: jobs are enqueued inside the same DB transaction as business data (“transactional outbox” style).
  • This ensures job creation is atomic with domain changes: either both commit or neither.
  • Jobs are worked by separate processes; the transaction is only for enqueueing, not execution.
  • Support for scheduled (delayed) jobs exists via a ScheduledAt option, though docs are still being fleshed out.

RDBMS as Queue: Pros and Cons

  • Proponents:
    • Strong correctness guarantees, simple mental model, and fewer moving parts if Postgres is already in use.
    • Adequate throughput for most systems; examples from other ecosystems show tens of thousands of jobs/sec.
    • Easier operations than adding Redis/RabbitMQ/Kafka for many small/medium apps.
  • Skeptics:
    • “Databases make poor queues” remains a common belief, citing scalability, bloat, and long-running job concerns.
    • Some argue they’d “never” use an RDBMS as a job queue based on prior experience.

Implementation Details and Patterns

  • Typical pattern: SELECT … FOR UPDATE SKIP LOCKED (or variants) to lease jobs safely across workers.
  • Suggestions include:
    • Using FOR NO KEY UPDATE to avoid blocking foreign-key inserts.
    • Partitioning the jobs table, per-tenant sequencing, and bulk-processing via leases for better throughput.
  • Postgres NOTIFY can be used to wake workers, but there’s concern about overhead and incompatibility with poolers like PgBouncer.
  • Some features are under development or requested: UI dashboard, job completion notifications, richer workflow support.

Comparisons to Other Systems

  • Related Postgres job queues in other languages (e.g., well-known Elixir and JS libraries) are cited as prior art and evidence the model works.
  • Alternatives discussed:
    • HTTP-based queues (GCP Tasks, SQS-style) praised for simplicity but criticized for transactional correctness issues and timeout limits.
    • Kafka- or NATS-based approaches, Temporal-like workflow engines, and pure-SQL queues (e.g., PGMQ) mentioned as different trade-off points.

Project Direction, Licensing, and Attribution

  • Some wonder about commercial vs purely open-source goals and whether similar Go queues should “join forces.”
  • The LGPLv3 license for a Go library is questioned due to static linking; maintainers acknowledge needing to clarify or adjust.
  • There is debate about how much design was inspired by existing Postgres-based job systems and calls for clearer attribution.