Queues don't fix overload (2014)
Queues are often added to software systems in hopes of handling overload, but engineers here argue they only smooth short, bursty spikes and can hide deeper capacity problems, leading to unbounded latency or catastrophic failure if arrival rates exceed processing rates. They contrast queues with alternatives like load shedding, backpressure, fair or weighted-fair queueing, and autoscaling, stressing that each is a trade-off among latency, availability, complexity, and cost rather than a silver bullet. Several comments point to queueing theory, careful monitoring, and identifying true bottlenecks as essential to designing resilient, performant systems.
Queues, Overload, and Trade‑offs
- Strong consensus: queues don’t “fix” overload; they buffer it. If long‑term arrival rate > service rate, queues must grow or load must be shed.
- Queues are useful for smoothing short spikes and hiding jitter, but harmful if treated as a magical scalability solution.
- Unbounded or very large queues cause “bufferbloat”: huge latency, masked problems, and harder recovery. Bounded queues that quickly fail are preferred.
Load Shedding, Backpressure, and Client Behavior
- Load shedding and backpressure are framed as explicit, honest trade‑offs: fewer requests served vs. more complexity vs. higher latency.
- Some argue “ignoring” excess requests and forcing callers to retry with idempotent APIs is appropriate; others see this as just pushing the queue elsewhere.
- HTTP 429 + exponential backoff is cited as a practical pattern; but naive retries can worsen overload.
Autoscaling and Capacity Limits
- One camp claims autoscaling “solves” overload in practice for most products; another stresses cost, non‑scaling components (DBs, third‑party services), and cloud resource limits.
- Multiple comments note that scaling stateless services without addressing bottlenecked databases or IO can make things worse.
Queueing Theory and Utilization
- Little’s Law and basic queueing results are frequently referenced: near‑100% utilization produces very long queues even if capacity ≈ demand.
- Recommended practice: keep utilization significantly below 100% (often ~80%) to maintain acceptable latency.
- Several resources and books on queueing theory, scheduling, and metastable failures are suggested.
Fair Queuing, Priorities, and Products
- Fair/weighted‑fair queuing is highlighted as beneficial: isolates misbehaving clients, enforces per‑class capacity shares, and can prioritize critical traffic.
- Priority queues can starve low‑priority work under sustained overload; fair queuing helps but cannot defy basic capacity math.
- Some participants promote systems implementing WFQ and concurrency‑based limiting, especially for AI and API workloads.
Monitoring, Design, and Anti‑Patterns
- Queues are not inherently bad; problems come from unbounded buffers, lack of SLAs, and absent monitoring.
- Suggested metrics: queue depth, time in queue, how often queues empty, and worker activity.
- Broader point: many teams add queues, caches, or shims without first measuring or understanding the real bottleneck, leading to fragile systems.