Ask HN: Why do message queue-based architectures seem less popular now?

Hacker News Discussion ↗

Message-queue-based architectures like RabbitMQ, Kafka, and SQS are still widely used, but many engineers report they’ve faded from view as “boring,” commoditized infrastructure rather than something to design entire systems around. Commenters attribute the shift to faster hardware and databases, easier horizontal scaling, and better HTTP/async tooling, which let many applications stay simple monoliths or use lightweight queues embedded in Redis or Postgres. Queues remain essential for high-scale, decoupled, or highly asynchronous workloads, but there is broad agreement that they were previously overused in microservice-heavy, resume-driven designs where their complexity and consistency pitfalls outweighed the benefits.

Perceived Decline vs Reality

  • Many commenters argue queues are still used “everywhere” (SQS, Kafka, RabbitMQ, Pub/Sub, Redis, etc.), but have become “boring infrastructure,” so fewer blog posts and talks.
  • Cloud-hosted and managed offerings (SQS, Kinesis, EventBridge, Kafka-as-a-service, SNS, etc.) hide much of the complexity, so they appear less prominently in architecture diagrams.
  • Hype cycles moved on (microservices → queues → “event-driven” → AI), but queues have largely reached the “mature, boring, productive” phase.

Impact of Microservices and Architecture Trends

  • Earlier microservices enthusiasm drove heavy queue usage; later backlash showed many teams built “distributed monoliths” and unnecessary complexity.
  • Several argue most companies are not at Google/Uber scale and can run comfortably on a monolith plus a strong database.
  • Others maintain microservices still make sense for large organizations and organizational reasons (team boundaries, ownership), with queues as glue.

Why Queues Are Still Used

  • Decoupling and resilience: protect core apps from slow/unreliable third parties, absorb traffic spikes, buffer during outages/maintenance, support retries and dead-lettering.
  • Asynchronous/background work: emails, billing, ETL/ML pipelines, analytics, IoT messaging, hardware/sensor data, workflow engines, embedded/RTOS, and enterprise integration buses.
  • Operational benefits: backpressure, observability of queue depth, backfilling, smoothing load, and simpler scaling of stateless workers.

Reasons to Avoid or Minimize Queues

  • Added complexity: distributed debugging, observability, schema evolution, message loss/duplication, ordering, and consistency with the source of truth.
  • Many “event-driven” designs underestimate sync/consistency issues and the difficulty of guaranteeing delivery; missed or out-of-order events cause subtle bugs.
  • Some see widespread queue/microservice adoption as resume-driven overengineering, especially for small teams and low-volume apps.

Alternatives and Evolutions

  • Databases + Redis: job tables, transactional outbox patterns, NOTIFY/LISTEN, Redis lists/streams often suffice at modest scale with better tooling and simplicity.
  • HTTP + service discovery: lightweight services using HTTP/JSON and async runtimes replace many queue-based RPC patterns.
  • Logs and workflow engines: log-based systems (Kafka, Pulsar) used as stateful event logs; durable execution/workflow platforms (Temporal, Step Functions, etc.) handle orchestration atop queues/streams.