Background Jobs: Idempotency, Retries, and DLQs

A user makes a payment. Notification, email, dashboard update, billing event, fraud check — all parallel work. If you do all that synchronously inside the HTTP request, the user waits 10 seconds. The right approach: enqueue everything, process async.

But "enqueue it" is half the solution. The other half is whether that queue actually behaves reliably.

Three patterns I've leaned on for the last 5+ years in production.

1. Idempotency — the foundation

What happens when a job runs twice?

The scenario you don't want: user paid 100 lira, the "update balance" job ran twice, balance is up by 200. Customer support is now your day.

Solution: every job must be idempotent. Running it twice should produce the same outcome as running it once.

In practice:

• Use an idempotency key. The payment job's payload includes payment_id. Before working, check: has this payment already been processed? If yes, skip.

def process_payment(message):
    payment_id = message['payment_id']
    if already_processed(payment_id):
        return  # idempotent skip
    
    update_balance(...)
    mark_processed(payment_id)

• The way you answer already_processed: either a processed_payments table in your DB, or a 24-hour Redis set. Whichever fits your scale.

• The operation itself can be idempotent by nature. "Set user.email = X" is — running it twice is the same as running it once. "Increment user.balance by 100" is not — that needs an idempotency key.

2. Retry with exponential backoff

A job hit an error. Network timeout, downstream service was slow. What do you do?

Wrong: retry immediately, then again, then again. Result: you DDoS the downstream.

Right: retry with growing delays.

attempt 1: now
attempt 2: 5 seconds
attempt 3: 30 seconds
attempt 4: 2 minutes
attempt 5: 10 minutes

This is exponential backoff. Most queue systems (RabbitMQ, AWS SQS, Sidekiq) support it natively.

Trap: don't retry on every error type. For example:

• Validation errors → won't fix on retry, send to DLQ
• 4xx HTTP responses → retry is pointless
• 5xx HTTP responses → retry makes sense
• Timeouts → retry makes sense

If your job logic can decide, branch in code. Otherwise route to a different queue per error type.

3. Dead Letter Queue (DLQ)

A job tried 5 times, all failed. Now what?

Wrong: drop it. Data loss.

Right: DLQ. Send failed messages to a separate queue. Review periodically.

In our production setup:

• Every main queue has a DLQ (payments.queue → payments.dlq)
• Alarms trigger when DLQ depth > 10
• Weekly DLQ review (usually the on-call's job)

The review process:

1. Find the error pattern. Same error every time?
2. Code bug or data issue?
3. Fixable? → fix + replay (push the message back to the main queue)
4. Not fixable? → manual intervention (CS notification, log, etc.)

Without a DLQ, failed messages silently disappear. Months later you discover users haven't been getting notifications.

At-least-once vs. exactly-once

Most queue systems give you at-least-once delivery: the message will arrive at least once, possibly more. Exactly-once is expensive and limited (Kafka's exactly-once is Kafka-internal only).

Practical rule: at-least-once + idempotency = effective exactly-once.

In other words: assume duplicates from the queue, but make jobs idempotent so duplicates are harmless. Simple and bulletproof.

Job priority and isolation

Payment jobs and email jobs shouldn't share a queue. Why: the email queue gets backed up an hour, payments get stuck behind it.

Practical: separate queues per criticality:

• critical.queue — payments, balance, fraud
• transactional.queue — notifications, email
• bulk.queue — analytics, reports

Each queue gets its own worker pool, rate limit, alarm.

Measurement

Three metrics to watch:

1. Queue depth — messages waiting. Trending up = problem
2. Job duration — average processing time. Climbing = downstream issue
3. Failure rate — % failing. > 1% jump = alarm

Without these three, "we have a queue" doesn't really mean anything operationally — it's flying blind.

Closing

Background jobs aren't a "set and forget" piece. Without idempotency, retries, and DLQs, "we have a queue" is a half-solution that's more dangerous than no solution.

Good news: get them right once and the system runs for years on minimal operational cost. Five years of using this pattern, and queue-related work is at most 2 minutes a week.