Postgres as queue

Original Article ↗ Hacker News Discussion ↗

Using PostgreSQL as a job queue divides opinion: some engineers praise the simplicity of reusing an existing, ACID-compliant database to manage tasks, avoid dual-write consistency issues, and reduce infrastructure and DevOps overhead. Others argue that databases are costly and complex to scale as queues grow, and that purpose-built systems (like SQS, Redis, or newer workflow engines) offer better performance, isolation, and architecture. The debate centers on trade-offs between “boring” consolidated stacks versus modular, specialized components, with many noting that PostgreSQL-based queues are fine for modest workloads but may become problematic at larger scales or under strict ordering and latency requirements.

When Postgres-as-queue Makes Sense

  • Many commenters use Postgres as a job queue successfully for:
    • Internal tools and low–to–moderate traffic apps (e.g., ~10k–1M jobs/day).
    • Cases where the database is already required and adding new infra is politically or operationally hard.
    • Work that isn’t ultra-latency-sensitive and doesn’t need “web scale.”
  • Benefits cited:
    • Single source of truth with ACID semantics; queue and business data share transactions.
    • Simpler architecture: fewer services, fewer “moving parts,” easier local dev.
    • Easy evolution using SQL, indexes, and schema changes; no new tech expertise needed.

Arguments Against Postgres-as-queue

  • Some respondents describe it as one of the worst architectural decisions they’ve seen, though details are sparse (sometimes due to regulatory constraints).
  • Main concerns:
    • Mixing concerns: likened to using an IMAP server as an app queue; modularity and separation are considered more robust long-term.
    • Scaling cost and complexity: Postgres can become expensive or cumbersome at high throughput and with many workers.
    • Risk of lock contention, connection limits, and complex failure handling when worker counts grow.

Transactional Consistency & Patterns

  • Strong support for using the database queue to avoid the “dual write” problem (DB + external queue not in one transaction).
  • Transactional outbox / saga patterns are mentioned as alternatives when an external queue is used.
  • Idempotency and leases (lease_expire timestamps, “visible_at” fields) are common patterns for handling crashed workers and retries.

Postgres Features Highlighted

  • SKIP LOCKED for concurrent workers without blocking.
  • Partial indexes and filtered queries for efficient selection of “ready” tasks.
  • LISTEN/NOTIFY (sometimes via triggers or extensions) to:
    • Reduce polling.
    • Achieve low-latency wakeups when new work appears.
    • But they are not durable; consumers must still periodically scan to cover missed notifications.

Alternatives & Ecosystem

  • Other SQL databases (MySQL, SQL Server, MariaDB) offer similar primitives (READPAST, filtered indexes, built-in message brokers).
  • Dedicated queues and workflow systems (SQS-like, Temporal-style, etc.) are suggested once scale or complexity justifies extra infrastructure.
  • Several libraries in different languages implement Postgres-backed job queues, indicating an active ecosystem.