Show HN: PgQueuer – Transform PostgreSQL into a Job Queue

Original Article ↗ Hacker News Discussion ↗

Developers react to PgQueuer, a Python job queue built on PostgreSQL primitives like LISTEN/NOTIFY and `SELECT FOR UPDATE SKIP LOCKED`, as another entry in the growing trend of “Postgres as your queue.” Supporters emphasize the simplicity and strong transactional guarantees of keeping background jobs in the primary database, especially at low to medium scale and for use cases like “insert row + enqueue job in one transaction.” Critics counter that high-throughput workloads expose PostgreSQL’s limits compared to dedicated systems such as SQS, Redis, or Kafka, and note that the ecosystem already has many similar Postgres-backed queues with richer features like workflows, scheduling, UIs, and multi-language support.

Overall reaction and use cases

  • Many commenters like the idea of a simple PostgreSQL-backed job queue, especially for small to medium systems where avoiding extra infrastructure (Redis, RabbitMQ, SQS) is a major win.
  • Common use cases mentioned: transactional background work (e.g., send emails, update Elasticsearch after a DB write), low/medium throughput queues, and environments already heavily invested in Postgres.

“Postgres is all you need” – pros and cons

  • Pro side: Using only Postgres (often via tools like PostgREST, RLS) can drastically cut development time and complexity; one system to manage, backup, and secure.
  • Con side: Painful once you hit use cases where Postgres is a poor fit (high-throughput queues, time-series, hierarchical data). Dedicated tools (Kafka, SQS, etc.) often have needed features Postgres-based reimplementations lack.
  • Several warn that trying to replace everything with one Postgres instance eventually runs into version, performance, or feature limits.

Queue semantics, LISTEN/NOTIFY, and locking

  • PgQueuer uses LISTEN/NOTIFY only as a signal that the queue table changed; actual job fetching uses SELECT FOR UPDATE SKIP LOCKED for safe concurrent processing.
  • LISTEN/NOTIFY avoids polling overhead but has limits (8k payload, no push through many connection poolers, all listeners see all notifications, potential missed events if the app mismanages pooled connections).
  • Discussion distinguishes “signaling” from “messaging”: Postgres notifications wake workers; jobs still live in tables.

Scaling and performance concerns

  • Low throughput: Postgres queues are considered fine, often ideal.
  • High throughput: issues raised include table bloat from frequent updates, write amplification, planner instability, row deletion complexity, and challenges with partitioning and autovacuum.
  • Some argue that once you need append-only status histories, partitions, and custom maintenance, you’re better off with a dedicated queue system.

Ecosystem and alternatives

  • Many similar Postgres queues already exist (Oban, River, pgmq, Graphile Worker, QueueClassic, GoodJob, SolidQueue, Procrastinate, Minion, etc.).
  • Some advocate backend-agnostic task systems (Celery alternatives, Dramatiq, Redis-based queues) for better local testing and flexibility, possibly with Postgres as one backend.
  • Others value Postgres queues specifically for transactional guarantees: enqueueing jobs and writing data in the same DB transaction.