Distributed systems · 15 min

Message queues

Let the fast part of your app reply now and the slow part catch up later. Nothing is lost in a rush, but any job may run twice, so make twice harmless.

A TV ad for your app airs at 8 pm. In the next minute, 10,000 people sign up. Each one should get a welcome email, and your email provider lets you send 50 a second.

Do the sum: 10,000 sign-ups in 60 seconds is about 167 a second. You can send 50. So here is the puzzle: how do you not lose a single welcome email, and not take the sign-up page down while it happens?

Sending from the sign-up page breaks it

This is the wrong way, but it is how most apps start. The sign-up handler saves the new user, calls the email provider, waits for its answer, then replies “Welcome!”.

On a quiet day that works. At 167 sign-ups a second, 117 of them every second can’t get their email sent straight away. Either the provider refuses them, and those emails are lost unless your code tries again, or they wait inside your handler.

Waiting is the worse failure. Every waiting request holds a connection open, and a server can only hold so many: say 500. With 117 extra requests piling up each second, 500 are stuck after about 4 seconds, and new visitors get an error page. The slow part (email) has taken down the fast part (saving a user).

Look at what the person signing up actually needs: their account saved, right now. The email can arrive 2 minutes later and nobody minds.

Write a note, reply, send later

So split the job in two. The sign-up handler saves the user, writes a short note, “send a welcome email to sign-up 4182”, and replies at once. Someone else sends the email later.

The sign-up page no longer cares how fast email is. It only needs the queue to accept a message, which takes a few milliseconds. When 167 a second arrive, the line simply gets longer.

Consumers work at their own pace. Each one takes a message, sends the email, and takes the next. Say one consumer sends 10 emails a second. Run 5 of them and together they send 50, exactly the provider’s limit.

The two sides no longer have to run at the same speed, or even be up at the same time. If the email provider is down for 5 minutes, sign-ups keep working and the messages wait in the queue.

How long until the last email?

Walk through the minute of the ad. 10,000 messages go in. The consumers send 50 a second, so 50 × 60 = 3,000 go out during that minute. When the rush ends, 10,000 − 3,000 = 7,000 messages are waiting. That waiting pile is the backlog.

The 5 consumers now work through it at 50 a second: 7,000 ÷ 50 = 140 seconds. The last person to sign up during the rush gets their email about 2 minutes 20 seconds later. (A few new sign-ups keep trickling in, so it takes a little longer in practice.)

The same sum works for any queue:

drain time = backlog ÷ (consumers × emails per second each)
           = 7,000 ÷ (5 × 10)
           = 140 seconds

Done means deleted

A consumer takes the message for sign-up 4182 and starts sending. Then its server loses power. What happens to the email?

If the queue deleted each message the moment it handed it out, that one is gone and person 4182 never hears from you. So queues don’t delete on hand-out. They wait for the consumer to say it finished.

While a consumer works on a message, the queue hides it from the other consumers, so two of them don’t take it at once. The hiding has a time limit, called the visibility timeout: 30 seconds by default in Amazon SQS. If no ack arrives in time, the queue assumes the consumer died and shows the message again. Another consumer picks it up.

So set the timeout well above how long the work normally takes. Too short, and a slow but healthy consumer has its message handed to a second consumer while it is still working.

At least once, sometimes twice

The other choice is to ack first and work second. Then a crash loses the email instead of repeating it: never twice, but sometimes never. For welcome emails, and for most work, losing one is worse than sending two.

Crashes are not the only cause. A consumer that is merely slow, say the provider takes 40 seconds to answer, passes the 30-second timeout while still working. The queue hands the message to someone else, and both finish it. You can’t design duplicates away. You design so they don’t hurt.

Make the second email harmless

Sending an email isn’t idempotent on its own, so the consumer makes it so. It keeps a table of sign-up ids that already got their welcome email. Before sending, it checks the table. If 4182 is already there, it skips the email and just acks.

In the code below, look at the order: check, work, record, ack. If a consumer crashes after recording but before the ack, the message comes back and the check at the top catches the repeat. (One narrow gap is left; the warning below covers it.)

handle(message):
    id = message.signup_id
    if id in welcomed:
        ack(message)            # a repeat: nothing to do
        return
    send_welcome_email(id)
    welcomed.add(id)
    ack(message)

Try it in the demo above: tick “Idempotent consumers” and press “Crash a consumer” again. The message still comes back, but this time “Repeats skipped” goes up and “Duplicate emails” stays where it was.

The sign-up that fails every time

Someone signs up with the address “maya@@example”. Your email library throws an error on it. The consumer catches the error and doesn’t ack. Rather than wait out the whole visibility timeout, it tells the queue to offer the message again after a short wait it picks (in SQS, by shortening that message’s timeout). The message comes back, fails again, and again, forever, using up a consumer each time.

Failures that fix themselves are common too: the provider has a 2-second hiccup, and the next try works. So you want retries, just not forever. The queue counts how many times it has handed each message out. After a limit, say 3, it stops and moves the message aside.

In Amazon SQS the limit is a setting called maxReceiveCount. Put an alert on the dead-letter queue’s size: every message there is a person who never got their email. Many teams also wait longer before each retry (1 s, then 2, then 4), which gives a struggling provider room to recover.

In the demo, press “Send a broken sign-up”. A consumer card shows “Trying sign-up #… (bad address)”, the try fails, and a line under the queue chips shows it waiting 1 s before the next try. After 3 failed tries it lands in the Dead-letter queue box, and the other sign-ups kept flowing past it the whole time.

When a queue is the wrong tool

A queue makes work happen later. That is the point, and also the cost. Don’t use one when the caller needs the result now to decide what happens next.

• Logging in. The page has to know right now whether the password is right. Putting the check on a queue only adds a wait.
• “Is this username taken?” The person is typing and waiting for the answer.
• Saving the new user itself. The sign-up page must not say “Welcome!” until the account really exists.

Also expect messages slightly out of order. Amazon SQS standard queues only try their best to keep the order. If order matters, SQS also has FIFO (first in, first out) queues. They keep the order, but because a message can’t go out until the one before it is done, they handle fewer messages per second.

Back to 8 pm

Now the puzzle has a full answer:

1. The sign-up handler saves the user, puts one message on the queue, and replies in milliseconds, however many people arrive.
2. 5 consumers send 10 emails a second each, matching the provider’s 50, and the backlog clears about 2 minutes after the rush.
3. Each consumer acks only after sending. The 30-second visibility timeout is far longer than one email takes, so repeats are rare.
4. The few repeats hit the welcomed table and turn into skips.
5. A broken address fails 3 times and lands in the dead-letter queue, and the alert on it tells someone to look.

In production, watch two numbers: how many messages are waiting, and how long the oldest one has waited. On a healthy day that wait stays at a few seconds. If it climbs minute after minute, consumers are slower than producers. The drain-time sum tells you how many more to add, up to what the provider allows.