Reliability and Idempotency

This article is about rethinking the use of TCP in your APIs.

Should you be using UDP for your API instead of TCP? That's not a question that gets asked very often. After all, TCP is the default protocol for most APIs, providing ordered and reliable delivery of requests and responses. But is that actually the right choice for your customers?

Even with TCP's reliability guarantees, many APIs still end up building in idempotent safeguards to handle retries and other practical realities. If your API is already designed to be idempotent, you might be paying a significant performance cost for guarantees you don't actually need.

Consider the following story.

A Tale of Two Ticket Windows

I was standing outside the Amphitheater on a busy Saturday afternoon, watching two very different scenes unfold.

To my left, a long line snaked around the building—hundreds of people waiting at what everyone called "the Reliable Window." The sign above it proclaimed in bold letters: GUARANTEED ORDER PROCESSING.

"Why do you call it reliable?" I asked a woman near the front of the line.

"Because you always get your ticket," she explained. "Everyone stays in order, and the system makes sure every transaction completes before moving to the next person. It's very organized."

I watched as the line moved forward in steady increments. When things were going well, it was actually quite efficient. Person after person would step up, state their name, receive their personalized ticket (the Amphitheater had a strict one-ticket-per-person policy—if you already had a ticket and asked again, they'd just hand you a duplicate of the same one), and move on.

But then I saw what happened when things went badly.

A man reached the front of the line and decided to pay entirely in pennies. The cashier had to count each one. The entire line stopped. Two hundred people stood frozen, waiting.

Behind him, a teenager was so engrossed in her phone that when it was finally her turn, she didn't notice for nearly a minute. The cashier called out. Nothing. Called again. Still nothing. Everyone waited.

Then, without warning, the cashier put up a "BE BACK IN A MINUTE" sign and disappeared on a break. The entire line had to disband. When she returned, everyone had to reorganize themselves back into order before service could resume.

"You seem frustrated," I said to a man who had been waiting forty minutes.

"It's reliable, though," he insisted, though his tone suggested he was trying to convince himself.

To my right was a very different scene. A single window with a sign that read: WALK-UP SERVICE. No line at all. People just walked past the window, called out their name, and kept walking.

"How does this work?" I asked a man who was strolling away, glancing at his phone.

"You just walk past and ask." He held up his phone—a ticket had just appeared on the screen. "Tell them your name, and a few seconds later your ticket shows up. You don't even have to stop."

I watched for a while. A woman walked by, called out her name without breaking stride, and checked her phone a moment later. Ticket received. A teenager did the same, glanced at his phone, frowned, circled back to the window and tried again. This time his ticket appeared. Ten seconds, maybe.

"What if you don't get an answer?" I asked.

"You walk past again," a young woman said, overhearing me. "Since each ticket is personalized—you can only get a ticket for yourself—it doesn't matter if you ask twice. If you already have one, they just send the same ticket again. So there's no harm in trying multiple times."

"And nobody waits for anyone else?"

"Never." She smiled. "If someone ahead of me is having a complicated conversation at the window, I just walk past and try anyway. If the clerk is busy, I don't get an answer and I circle back in a few seconds. I never have to stand behind someone counting pennies. And I know within seconds whether my ticket came through—just check my phone—so I always know right away if I need to try again."

I stood there, watching both systems. The Reliable Window had just ground to a halt because someone was arguing with the cashier about their name spelling. Three hundred people stood motionless.

Meanwhile, at the walk-up window, people flowed past continuously. A few seconds each, occasionally circling back if needed, but never waiting. Nobody stood in line. Nobody waited for anyone else's transaction to complete.

"I think I'm starting to understand something about reliability," I muttered.

The man next to me, ticket freshly arrived on his phone, overheard and nodded. "Reliability isn't about guaranteeing order when order doesn't matter. It's about guaranteeing you get what you need. When what you're asking for is idempotent—when asking twice gives the same answer as asking once—you don't need perfect ordering. You just need fast responses and the ability to try again."

He gestured to the massive line at the Reliable Window. "Those people are paying a huge cost for a guarantee they don't actually need. They're all getting personalized tickets—if they asked twice, they'd get the same one. So why make them wait in order? Why make everyone suffer when one person is slow?"

I looked back at the sign: GUARANTEED ORDER PROCESSING.

"Guaranteed order," I said slowly. "But at what cost?"

The Moral

When your API is idempotent (when making the same request multiple times produces the same result) you don't need TCP's ordering guarantees and connection overhead. You can serve your customers better with UDP's simplicity. Fast responses, no queuing, no head-of-line blocking, and if a request gets lost, just try again. The guarantee that retrying is safe means not everything must happen in perfect order.

APIs designed to take advantage of UDP's strengths can often deliver far better performance and reliability in practice. The story above puts a little human context around concepts like head-of-line blocking and connection resets, but there are other issues that come into play as well.

There have been excellent articles written about load shedding, for example, motivated by the way TCP based services fail (hint, badly). An overloaded TCP server ultimately fails to serve any requests at all, because new connections cannot be established and existing connections get stuck waiting for packets that will never arrive. UDP based services, on the other hand, naturally shed load by simply dropping packets when overwhelmed, allowing well behaved clients to retry and recover gracefully. When combined with "fair" server processing that balances work from all clients, UDP based APIs can often remain partially responsive under conditions that would cause TCP based APIs to collapse. The server processes what it can, and clients retry what they need.

The lesson here is that reliability is not a one-size-fits-all property. It's important to understand the specific reliability needs and capabilities of your API, and choose the right protocol accordingly. If your API is idempotent, consider whether you really need TCP's guarantees or if UDP could serve your users better.