Queues Are Not a Silver Bullet

Queues solve a real problem: they decouple work, absorb bursts, and keep user-facing paths from waiting on slow downstream systems. But they do not solve reliability by themselves. If you add a queue without idempotency, observability, retry policy, and a redrive path, you often move failure into a darker corner of the system.

The useful mental model is simple: a queue is a buffer, not a guarantee. It can improve resilience and throughput, but it also introduces delivery semantics, duplicate processing, ordering ambiguity, poison messages, and operational overhead.

Teams usually add queues because they want to:

1. move slow work off the request path
2. absorb spikes without failing fast
3. retry transient failures automatically
4. isolate expensive or unstable integrations

Those are good reasons. The mistake is assuming the queue removed the failure instead of relocating it.

The first question is what delivery guarantee you are actually buying.

• at-most-once: messages may be lost, but duplicates are rare
• at-least-once: duplicates are expected, but loss is reduced
• exactly-once: usually an application-level effect, not a free transport property

If your side effects cannot tolerate duplicates, the queue is not where the magic happens. You need idempotent consumers.

Amazon SQS documents at-least-once delivery directly, which is useful because it removes the last excuse for pretending duplicate delivery is unusual.

This matrix is not anti-queue. It is anti-magical thinking.

If a worker performs the side effect and crashes before acknowledgement, the message may be redelivered. That is normal at-least-once behavior, not a corner case.

A poison message fails repeatedly and quietly burns retries, time, and worker capacity. If it is not isolated into a DLQ or equivalent path, it can turn one bad payload into a broad incident.

Ordering often breaks once you add retries, parallel consumers, or partitioning. If order matters, define exactly which entity needs order and how out-of-order processing is corrected.

Retries help until they amplify load. A slow dependency triggers more retries, the backlog grows, latency rises, and the queue starts manufacturing its own outage.

Dead-letter queues are not storage. They are a safety valve.

AWS's guide to using dead-letter queues in SQS is a good operational reference because it treats the DLQ as a configured failure path with explicit redrive behavior, not as a permanent resting place for bad messages.

The sane workflow is:

1. move repeated failures to a DLQ after a defined threshold
2. classify the root cause
3. fix the systemic issue if one exists
4. redrive only messages that are safe to replay
5. add a test or alert so the same class of failure becomes visible next time

If DLQ entries accumulate without review, the system is not resilient. It is only hiding work.

The cleanest way to make a queue safe is to design the consumer so that repeated delivery is harmless.

If you need the deeper version of this principle, read Idempotency for webhooks.

The idempotent consumer pattern is the cleanest external reference for this section because it makes the application-level responsibility explicit: delivery can repeat, so the effect must not.

This checklist should exist before the queue becomes important:

• define the queue purpose in one sentence
• document expected delivery guarantees
• define retry policy and maximum attempts
• specify DLQ threshold and redrive process
• record which messages are safe to replay
• define which side effects must be idempotent
• track backlog depth, oldest message age, failure rate, and redrive count
• assign an owner for manual intervention

That runbook should live near the code, not in a forgotten wiki.

Queues help when the work is:

• asynchronous by nature
• safe to retry
• not immediately user-visible
• bounded by a clear failure policy

Examples include notifications, background enrichment, media processing, and unstable third-party integrations.

A queue is usually the wrong answer when:

• the workflow needs immediate consistency
• the side effect cannot be repeated safely
• strict ordering is a business requirement
• the team cannot commit to operating retries and DLQs

If any of those are true, a synchronous flow, stronger transactional boundary, or different workflow design may be the better choice.

If you want the broader async architecture lens, read Event-driven systems without folklore. If you want the dedupe and repeat-processing layer, read Idempotency for webhooks. If you want the operational visibility layer, read Observability for product engineers.