Designing a Rate Limiter

"Design a rate limiter" is one of those interview prompts that sounds trivial for the first thirty seconds and then quietly reveals a distributed-systems problem underneath. Everyone knows what a rate limiter does — cap how many requests a client can make in a given period. The interesting part is watching a candidate's first instinct fall apart under a single follow-up question: "what happens when you have more than one server?"

This article walks through that exact progression: the obvious solution, why it breaks, the fix, the subtle bug hiding in the fix, and the refinement that actually gets asked about in a strong interview.

The naive version: a counter in memory

The first idea almost everyone reaches for is a token bucket kept entirely in the application process: a counter per client, incremented on every request, reset by a background timer every window (say, 100 requests per minute). It's a few lines of code, it's fast because there's no network hop, and in a single-process demo it works perfectly.

It also only works because there's exactly one process. The moment this service sits behind a load balancer with three, ten, or a hundred application instances, the counter stops meaning anything global.

Why it breaks with more than one server

Each instance holds its own independent counter, and the load balancer has no idea a limit even exists — it's just routing traffic. A client capped at 100 requests per minute can round-robin across ten instances and get 1,000 requests through, because each instance is happily enforcing its own local limit of 100 with no knowledge of the other nine. Conversely, if a burst of requests happens to concentrate on a single instance (sticky sessions, an unlucky hash, a downed neighbor), that client gets throttled far below the intended limit even though the fleet as a whole has plenty of headroom. Either way, the limit you thought you configured is not the limit that actually applies.

The fix: move the counter somewhere shared

The fix is conceptually simple: stop keeping the counter in each process and instead keep it in a store all instances can read and write — typically Redis, because it's in-memory fast and offers the atomic primitives this problem actually needs. Every instance, regardless of which one a request lands on, checks and updates the same key for a given client. Now the limit is enforced against the fleet's combined traffic, not against whatever slice of traffic one instance happened to see.

The subtlety: "check then increment" still races

This is where a lot of candidates think they're done and aren't. A naive implementation reads the current count with GET, compares it to the limit in application code, and if it's under the limit, writes back the incremented value with SET. That's two round trips with a decision made in between — and under concurrency, that gap is a race. If two requests for the same client arrive at nearly the same instant on two different instances, both can GET the same count, both see it's under the limit, and both proceed to increment — letting one extra request through that should have been rejected. At low traffic this is invisible; at real traffic it's exactly the kind of leak that lets an abusive client slip past a limit that looks correct on paper.

The fix is to make the check-and-increment a single atomic operation instead of two separate ones. Redis's INCR already does this — it atomically increments and returns the new value in one step, so there's no window where two clients can both read the stale value. Pair it with EXPIRE on the first increment of a window so the key cleans itself up, or bundle both into a small Lua script so the whole check-and-increment-and-expire sequence runs as one atomic unit on the Redis server, immune to interleaving from any other request.

Fixed windows have their own bug: boundary bursts

Even with atomic increments, a fixed window (e.g., "100 requests per calendar minute") has a structural weakness: a client can send 100 requests in the last second of one window and another 100 in the first second of the next window, getting 200 requests through in roughly two seconds — double the intended rate — simply by timing requests around the boundary. The limit is technically respected per window, but not over any sliding one-minute span, which is what the limit was actually supposed to mean.

The sliding-window counter approach fixes this without the memory cost of tracking every individual request timestamp (a true sliding log). Instead of trusting the current window's count alone, it also looks at the previous window's count and weights it by how much of that previous window still overlaps the trailing N-second span from right now. Early in the current window, most of the "recent" traffic is still attributable to the previous window, so its count should count almost fully; late in the current window, the previous window barely overlaps the trailing span at all, so its weight fades toward zero. That single proportional adjustment is enough to smooth out the boundary-burst problem while still storing just two integers per client.

Putting it together

The snippet below shows both pieces: a Lua script that performs an atomic fixed-window check-and-increment (the correctness fix for the race condition), and a sliding-window-counter function that blends the current and previous window's counts to approximate a true sliding limit.

-- rate_limit.lua
-- Atomic check-and-increment for a fixed window, run via EVAL so
-- concurrent requests can never race each other.
local key = KEYS[1]
local limit = tonumber(ARGV[1])
local window = tonumber(ARGV[2]) -- seconds

local current = redis.call("INCR", key)
if current == 1 then
  -- first hit in this window: start the expiry clock
  redis.call("EXPIRE", key, window)
end

if current > limit then
  return 0 -- reject
end
return 1 -- allow

-- Application code (Node.js):
async function isAllowed(userId, limit = 100, windowSeconds = 60) {
  const windowStart = Math.floor(Date.now() / 1000 / windowSeconds);
  const key = `rl:${userId}:${windowStart}`;

  const allowed = await redis.eval(
    RATE_LIMIT_SCRIPT,
    1,
    key,
    limit,
    windowSeconds
  );

  return allowed === 1;
}

// Sliding-window-counter refinement: blend current + previous window
async function isAllowedSliding(userId, limit = 100, windowSeconds = 60) {
  const now = Date.now() / 1000;
  const currentWindow = Math.floor(now / windowSeconds);
  const elapsed = now - currentWindow * windowSeconds;
  const weight = 1 - elapsed / windowSeconds; // how much of prev window "bleeds" in

  const currentKey = `rl:${userId}:${currentWindow}`;
  const prevKey = `rl:${userId}:${currentWindow - 1}`;

  const [currentCount, prevCount] = await redis.mget(currentKey, prevKey);
  const weightedCount = (Number(currentCount) || 0) + (Number(prevCount) || 0) * weight;

  if (weightedCount >= limit) return false;

  await redis.multi().incr(currentKey).expire(currentKey, windowSeconds * 2).exec();
  return true;
}

isAllowed enforces a hard fixed-window cap with no race window, because the increment and the expiry decision happen inside a single Redis-side script execution — no other request can observe or mutate the key mid-operation. isAllowedSliding builds on top of that idea but reads two keys (current and previous window) and computes a weighted estimate before deciding whether to allow the request, trading a small amount of approximation for a much smoother rate curve across window boundaries.

Where this goes if the interviewer keeps pushing

A strong candidate doesn't stop at "it works." The natural follow-ups are worth having answers ready for: what key are you actually rate-limiting on — per-user ID for authenticated traffic, per-IP for anonymous traffic, per-API-key for third-party integrations, or some combination, since a single IP can represent thousands of users behind NAT and a single user can rotate across IPs. And what does the client see when they're throttled — a well-behaved API returns HTTP 429 Too Many Requests with a Retry-After header telling the caller exactly how many seconds to back off, rather than letting them guess and hammer the endpoint again immediately.