Building durable workflows on Postgres

Original Article ↗ Hacker News Discussion ↗

PostgreSQL is being pushed far beyond traditional OLTP use, with engineers building durable job queues and workflow engines directly on top of it to avoid adding specialized infrastructure like Temporal or Kafka-style systems. Commenters highlight the appeal of reusing an existing, well-understood database for retries, orchestration, and durability, but warn that homegrown solutions often grow into complex, fragile workflow engines that struggle with scale, correctness, and maintenance. The debate centers on where to draw the line between “just use Postgres” and adopting dedicated workflow platforms, balancing simplicity and cost against performance, observability, and long-term reliability.

Postgres as Durable Workflow Engine

  • Many commenters praise how far you can go with “just Postgres,” using it for durable queues, workflows, notifications, even vector search and time‑series.
  • Several teams report success building durable execution and complex notification logic on top of Postgres, highlighting low cost and simplicity when it’s already a core dependency.
  • Some use it specifically to unify data and workflow state in one place, deferring specialized tooling until scale truly demands it.

Locking, Queues, and Scaling Concerns

  • Common patterns include SELECT … FOR UPDATE SKIP LOCKED and advisory locks for distributed workers.
  • There is disagreement on which pattern scales better; newer advice may favor SELECT … SKIP LOCKED, but people call for real benchmarks.
  • Others warn that SKIP LOCKED can cause vacuum issues and table bloat as worker counts grow.
  • LISTEN/NOTIFY is seen as fragile and not scalable on busy instances, though upcoming Postgres versions are expected to improve it.

DBOS, Temporal, and Other Workflow Systems

  • Users compare DBOS, Temporal, Restate, Cloudflare Workflows, and others.
  • DBOS is described as less complex than Temporal but limited by Postgres throughput; its strength is atomic messaging in the same DB transaction as business logic.
  • Temporal draws sharply mixed reviews: SDK and concepts are praised; operational footprint, scaling with Cassandra, and sales tactics are heavily criticized by some, while others question those cost claims.
  • Some prefer to copy Temporal’s ideas and reimplement a simpler variant directly in Postgres.

Build vs Buy and the Complexity of “Rolling Your Own”

  • Multiple comments note that homegrown “simple queues” often accrete retries, backoff, timeouts, cancellation, idempotency, visibility, and tooling, becoming a partial workflow engine.
  • Advocates of specialized systems argue this reliability layer is non‑trivial; skeptics say queues plus idempotent jobs and a status column are enough for most use cases.

Postgres‑Everywhere vs Appropriate Tools

  • One camp argues: start with Postgres for almost everything and only add systems when truly needed.
  • Others call this “religious,” pointing out Postgres cost and operational pain at TB‑scale or very high throughput, and recommending purpose‑built stores (ClickHouse, etc.) for heavy analytics or logs.
  • There’s broad agreement that over‑engineering for hypothetical “web scale” is as harmful as under‑engineering.