API Gateways & Rate Limiting

Once a system grows past a handful of services, every one of them needs the same boring plumbing: verify who's calling, decide where the request goes, terminate TLS, maybe reshape the payload, and log what happened. Reimplementing that in every service is how you end up with a dozen slightly different auth bugs. An API gateway exists to do it once, in one place, in front of everything.

What the gateway centralizes

Think of the gateway as the single door every external request walks through before it reaches a backend service. That gives you one spot to handle the concerns that would otherwise be duplicated everywhere:

• Authentication and authorization — validate the API key, JWT, or session token once, and pass a verified identity downstream instead of making every service re-implement token parsing and revocation checks.
• Routing — map a path or hostname to the right backend service, so clients never need to know that /orders and /payments are owned by entirely different teams and deployed independently.
• TLS termination — decrypt HTTPS at the edge once, so internal services can talk plain HTTP (often over a private network) without every one of them managing certificates.
• Request/response transformation — rewrite headers, version an API, or reshape a legacy backend's response into the contract clients actually expect.
• Observability — every request passes through one choke point, which is the natural place to emit consistent access logs, latency metrics, and error rates across the entire API surface.

None of this is exotic — it's the stuff every service needs but nobody wants to own five separate copies of. Pushing it to the edge also means a backend service can stay almost embarrassingly simple: it trusts the identity the gateway attached and gets on with its actual job.

Rate limiting: the algorithms

Centralizing traffic in one place also makes it the natural spot to decide how much traffic any single caller gets to send. There are a few standard algorithms, and they trade off burst tolerance against implementation cost differently.

• Token bucket — a bucket holds up to capacity tokens and refills at a steady rate. Every request costs one token; if the bucket is empty, the request is denied. Because tokens accumulate while idle, a client that's been quiet can burst up to the full capacity before being throttled — a good match for real client behavior, which is rarely perfectly smooth.
• Leaky bucket — requests queue up and drain out at a fixed rate, like water leaking from a hole in a bucket. It smooths bursts into a steady outflow rather than letting them through, which protects downstream services better but adds queuing latency that token bucket doesn't.
• Fixed window — count requests in, say, a one-minute window and reset the counter every minute. Trivial to implement, but it has an edge-case flaw: a client can send its full limit in the last second of one window and its full limit again in the first second of the next, doubling the allowed rate right at the boundary.
• Sliding window — evaluates the limit over a continuously moving window (either by tracking timestamps directly or by weighting two adjacent fixed windows) instead of a hard reset. It closes the boundary-burst problem at the cost of more bookkeeping per request.

function allowRequest(bucket, now):
    # bucket = { tokens, capacity, refillRatePerSec, lastRefill }
    elapsed = now - bucket.lastRefill
    refill = elapsed * bucket.refillRatePerSec

    bucket.tokens = min(bucket.capacity, bucket.tokens + refill)
    bucket.lastRefill = now

    if bucket.tokens >= 1:
        bucket.tokens -= 1
        return ALLOW
    else:
        return DENY  # respond 429, set Retry-After

Token bucket is the default most gateways ship because it's cheap to evaluate, tolerates bursts gracefully, and needs only a counter and a timestamp per key — a good fit for a check that runs on every single request.

Scoping the limit

The algorithm only matters once you've decided what the limit applies to. A single global limit on the whole service is almost never the right granularity — it lets one misbehaving tenant starve everyone else, and it can't reflect that different endpoints cost wildly different amounts of backend work.

• Per-API-key or per-user — the most common scope for authenticated traffic; it maps directly to a pricing tier or fairness contract ("500 requests/minute on the free plan").
• Per-IP — the fallback for unauthenticated traffic, where there's no identity to key on yet. Weaker as a signal — NATs and shared proxies put many real users behind one IP — but still useful against abuse.
• Per-endpoint — a cheap health check and an expensive search query shouldn't share a budget; scoping by route lets you set tighter limits on the handlers that actually hurt when overloaded.

In practice, gateways combine these — a per-user limit layered with a stricter per-endpoint limit on the few routes that do real work — so that fairness and backend protection are enforced independently.

Backpressure and graceful degradation

Rate limiting is only useful if the client on the other end knows what happened. The convention is to reject over-limit requests with HTTP 429 Too Many Requests and a Retry-After header telling the caller how long to wait before trying again — turning an opaque failure into a clear, actionable signal instead of a retry storm.

This is what makes rate limiting a form of backpressure rather than just an abuse filter: it's the mechanism that stops a spike in demand from turning into a cascading failure, where an overloaded service slows down, its callers time out and retry, and the retries pile on load that makes things worse. Rejecting fast and cheap at the edge is far better for the whole system than letting every request limp through to an already-struggling backend.

The gateway is also a risk

Putting one component in front of every request makes it the single best place to enforce policy — and, for exactly the same reason, a single point of failure and an extra network hop on every request. If it goes down, nothing gets through, no matter how healthy the services behind it are.

The standard mitigation is the same one you'd apply to any critical service: run multiple gateway instances behind a load balancer, health-check them so traffic never routes to an unhealthy node, and scale the fleet horizontally as request volume grows. Kept stateless — with rate-limit counters and session data pushed out to a shared store like Redis rather than held in local memory — any instance can handle any request, and losing one is a non-event instead of an outage.