Show HN: Hatchet – Open-source distributed task queue

Original Article ↗ Hacker News Discussion ↗

Architecture & durability model

  • Tasks are backed by predefined workflows whose steps and inputs/outputs are persisted in Postgres.
  • Engine components poll Postgres, assign work over long‑lived gRPC connections, and track timeouts and retries in the DB.
  • Durable execution is “at‑least once”: steps can be replayed with stored inputs; idempotent task design is recommended.
  • No need for Redis-based locks; Postgres is the core durability and coordination layer.

Worker behavior & failure semantics

  • Workers heartbeat every few seconds; if inactive for ~60s, tasks are reassigned (if retries remain).
  • Cancellation signals are sent but not guaranteed to be delivered; duplicate executions are possible if workers ignore cancels or come back after reassignment.
  • This is recognized as an edge case needing alerting and better monitoring.

Workflows, DAGs, and limitations

  • Workflows are declared DAGs ahead of time; this simplifies visualization, versioning, and multi-step debugging vs procedural code.
  • Tradeoffs: no try/catch inside a single task, no simple “await futures & gather later” model yet. Procedural workflows are on the roadmap.
  • Child workflows can be triggered from steps; fan-out based on an array return value is being considered.
  • Dynamic DAG mutation at runtime is not currently described as supported.

Comparisons to other tools

  • Positioned as an alternative to Celery/Redis/RabbitMQ with stronger observability and a UI, and as conceptually similar to Temporal/Cadence but simpler to operate.
  • Differentiated from pg-boss/River/Graphile Worker by being a separate service + UI rather than a pure library and by adding workflow semantics and worker pools.
  • Compared to Windmill/Inngest: similar Postgres-based workflow engine; Hatchet is narrower (task queue + workflows) and MIT-licensed.

Dependencies & infra choices

  • Today uses Postgres plus RabbitMQ for some pub/sub; maintainers would like to move to Postgres LISTEN/NOTIFY or similar, but note complications with connection poolers and replay.
  • Some in the thread argue RabbitMQ is cheap and reliable; others prefer minimizing moving parts around Postgres.

Self-hosting, cloud, pricing, licensing

  • Self-hosting guide exists; key is highly available ingestion service and managed Postgres with PITR.
  • Cloud offering is in early access; pricing not yet published.
  • Project is fully MIT; revenue is expected from hosted Hatchet Cloud and possibly enterprise support.
  • There is awareness of hyperscaler “lift-and-serve” risk; they choose MIT despite that.

Scheduling, delayed jobs, and use cases

  • Cron-based scheduling exists via a Go cron library; one-time/delayed scheduling exists in a basic form and is being refactored to be DB-driven and scalable.
  • Some long-horizon reminder/notification use cases (e.g., healthcare follow-ups) aren’t well supported yet but are a target.
  • Hatchet is not AI-specific but is pitched as suitable for GenAI pipelines and other multi-step, resource-bound workflows.

Performance and observability

  • Reported average start latency: ~50 ms for first step, ~25 ms for subsequent steps in load tests, acknowledged as slower than raw message queues but acceptable for DAG/task semantics.
  • Observability/UX is a major selling point vs Celery/Temporal: UI to inspect workflows, worker health, and DAG execution; still improving argument visibility and type/serialization introspection.

Overall reception

  • Many commenters are enthusiastic, saying they’ve waited for a Postgres-backed, multi-language queue with good observability.
  • Skeptics question added complexity vs simpler PG queues, RabbitMQ, or existing workflow engines, and raise concerns about edge cases (cancellation, exactly-once semantics, long-term scheduling, and dependency bloat).