Design a Rate Limiter

Alice’s mobile app retries a checkout three times after a blip. Bob’s scraper hits /search 400 times a second. A launch-day sale sends 50,000 legitimate customers at the same gateway.

If we do nothing, Bob and the sale share one fate: the database falls over and Alice cannot pay. If we cap everyone at 10 requests per minute, the sale looks like an outage.

The main design question is:

How do we allow a fair share of traffic per client, allow short bursts when they are harmless, and enforce the limit across many API nodes without turning the limiter into the next bottleneck?

We will start with a counter in one process. Distributed Redis, token buckets, and fail-open policy appear when that counter is no longer enough.

You can watch token bucket, leaky bucket, and sliding window accept or reject a request stream in the Rate Limiting Visualizer.

1. Clarify the problem

“Design a rate limiter” can mean an nginx module, an API-gateway policy, or a full quota product with billing.

I would ask:

• What is the unit: IP, user id, API key, or tenant?
• Which actions are limited: every HTTP call, or only expensive ones?
• Is a short burst allowed, or must traffic be smooth?
• Do we return 429 or queue the request?
• Is the limit per node or global across the fleet?
• What happens when Redis (or the limiter store) is down — fail open or fail closed?

If the interviewer gives no extra constraints, I would state:

Placement          API gateway + optional in-service checks
Identity           Authenticated: user/API key; anonymous: IP
Default rule       100 requests / minute / user, burst allowed
Response           HTTP 429 + Retry-After
Scope              Global across API nodes
Store              Redis for counters; rules in config or DB
On store failure   Fail open for public reads; fail closed for login/pay

A limiter is not a product database. Counts can be slightly wrong. Double-charging cannot. That distinction will pick the algorithm and the failure mode.

2. Functional requirements

The system must:

1. Decide allow or reject for each request before the expensive work.
2. Apply different limits to different routes and identities.
3. Return a clear 429 with when to retry.
4. Enforce the limit across many stateless API processes.
5. Allow a configured burst for token-bucket rules.
6. Expose remaining quota in headers when useful.

The first version does not include:

• a customer-facing quota dashboard;
• billing on overage;
• bot detection beyond rate;
• shaping (delaying) requests in a server-side queue.

3. Non-functional requirements

Allow/deny decision     fast, slightly stale counters OK
Money / login routes    fail closed if we cannot count
Public catalog GET      fail open if Redis is down
Audit of every reject   optional, sampled

Edge cases

• Two API pods increment the same user at once.
• A NAT gateway makes 1,000 employees share one IP.
• A clock jump breaks a timestamp window.
• Redis is partitioned from half the fleet.
• A viral user is a hot key on one Redis shard.
• Clients retry in lockstep after Retry-After (thundering herd).

4. Estimate the scale

Assume a mid-size API:

Peak request rate              50,000 QPS
Distinct limit keys            2 million active users / hour
Redis ops per request          1–3

Every request that hits the limiter is a Redis command. At 50,000 QPS that is a serious but ordinary Redis cluster, not a science project. The first bottleneck is usually a hot key (one celebrity user, one /health scraper IP), not total QPS.

Memory is small: a counter or a few timestamps per key, plus TTL. Do not store a list of every request forever.

5. Where the limiter lives

Client
  │
  ▼
API Gateway / edge     ← first limiter (IP, key, route)
  │
  ▼
Service                ← second limiter (expensive RPC, per-tenant quota)
  │
  ▼
Database

Put a cheap, coarse limit at the edge so junk dies early. Put a precise limit next to the work that is actually scarce (search, SMS send, password check).

Do not rate-limit only in one application replica. Five pods with “100/min in memory” is 500/min in total.

6. APIs and headers

The limiter is usually not a public CRUD API. It is middleware:

GET /v1/search?q=dal
Authorization: Bearer <token>

Allowed:

HTTP 200 OK
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 63
X-RateLimit-Reset: 1786970060

Rejected:

HTTP 429 Too Many Requests
Retry-After: 12
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1786970060

{
  "error": {
    "code": "RATE_LIMITED",
    "message": "Too many requests"
  }
}

A small admin API can change rules:

PUT /internal/rate-limits
{ "key_template": "user:{id}:search", "algorithm": "token_bucket",
  "rate": 100, "period_seconds": 60, "burst": 20 }

7. Start with a counter — then watch it fail

Fixed window:

key = user:alice:search:2026-08-22T10:01
INCR key
EXPIRE key 60   (if first)
if count > 100: reject

At 10:00:59 Alice sends 100 requests. At 10:01:00 the window resets. She sends 100 more. That is 200 in two seconds, which is not “100 per minute.”

  10:00:00                         10:01:00
      |---------- window A ----------|---------- window B ----------|
                              100 reqs  100 reqs
                                   |    |
                                   v    v
                              allowed   allowed
                              (cap 100) (cap 100)

  two seconds around the boundary: 200 requests

This is the fixed-window boundary burst. It is simple and often good enough for coarse IP limits. It is a weak answer if the interviewer cares about smoothness.

8. Sliding window log

Store each request timestamp:

ZADD user:alice:search <now> <uuid>
ZREMRANGEBYSCORE ... 0 now-60
ZCARD

Accurate, expensive: memory and CPU grow with request rate. A user at the cap keeps 100 timestamps. A 10,000 QPS key is a bad Redis citizen.

  now-60s                          now
     |-------- sliding 60s ---------|
     x  x   x x     x  x x   x   x     ← one timestamp per request
     drop these                        count remaining

Use this only for small, strict quotas.

9. Sliding window counter

Approximate the current minute by weighting the previous window:

estimated = prev_count * (1 - elapsed_fraction) + curr_count

Cheap (two integers). Good enough for most APIs. Still not a perfect continuous window, but it kills the worst fixed-window spike.

  prev window (60s)              current window
  count = 80                     count = 20
  |------------------------------|------------------------------|
                           now at 25% into current
  estimated ≈ 80 * 0.75 + 20 = 80

10. Token bucket (the usual interview pick)

Imagine a bucket that gains tokens at a steady rate and holds at most burst tokens. Each request spends one token. If the bucket is empty, reject.

capacity     20     (burst)
refill       100/60 tokens per second

Alice can send 20 requests immediately after idle, then about 1.67/s. That matches “100 per minute with burst.”

  idle a while          burst of 20           then ~1.67/s
  +----------------+    +----------------+    +----------------+
  | tokens = 20    | -> | tokens = 0     | -> | refill drips   |
  | (full)         |    | 20 allowed     |    | 21st waits     |
  +----------------+    +----------------+    +----------------+
         ▲ refill 100/60 per second, cap 20

State per key:

tokens
last_refill_time

Refill on each request from elapsed time. Do not run a timer per user.

Token bucket is burst-friendly. That is what most product APIs want.

Try it in the visualizer: fire a burst, then a steady stream, and compare with leaky bucket.

11. Leaky bucket

Requests enter a queue; they leak at a constant rate. Excess is dropped (or wait). Outflow is smooth. Bursts are not served immediately.

  incoming burst
       │ │ │ │ │
       v v v v v
    +-------------+
    |  queue /    |   leak 1.67/s   ----->  downstream
    |  waterline  |
    +-------------+
       overflow drops (or 429)

  token bucket: burst hits the API immediately
  leaky bucket: burst is smoothed; the backend never sees a spike

Use this when the downstream must see a smooth QPS (a fragile old service, an SMS provider). Do not use it as the default user-facing API limiter unless you want to queue in memory — that is a different product (buffering).

12. Which algorithm to say out loud

For a generic “design a rate limiter” interview, implement token bucket in Redis and mention the others.

13. One process is not enough

In-memory token buckets on each pod:

Alice → pod A  (has 10 tokens)
Alice → pod B  (has 10 tokens)

The global limit is wrong by a factor of replica count. Sticky sessions do not fix retries or many devices.

Use a shared store. Redis is the default interview answer because INCR / Lua is fast and keys expire.

14. Distributed token bucket without a race

Naive:

GET tokens
if tokens > 0:
  SET tokens-1

Two pods both read 1 and both allow. The limit leaks.

Fix with one atomic script (Lua) or a single INCR on a window counter:

Fixed window (correct, if you accept the algorithm):

n = INCR user:alice:min:10:01
if n == 1: EXPIRE 60
if n > 100: deny

Token bucket: Lua that reads tokens + last_refill, writes both, returns allow/deny. One round trip. Document that you do not do get-then-set in the application.

Alternatives:

Local limiter (fast) + Redis (global)
  allow if local AND global

Local protects the pod; Redis protects the fleet. Local can be a small token bucket so Redis blips do not melt one box.

15. Key design

rl:{scope}:{id}:{route}

Examples:

rl:user:u_alice:search
rl:ip:203.0.113.4:login
rl:key:ak_live_…:sms.send

Login and SMS should be stricter and fail closed. Search can be looser.

Do not rate-limit only on IP for logged-in users behind a company NAT.

Rules belong in config or a small table, cached in the gateway:

route /v1/login     5 / 15m / IP      fail closed
route /v1/search  100 / 1m / user     fail open
route /v1/sms       10 / 1d / user    fail closed

16. Hot keys and sharding

One scraper IP or one popular mobile app install can hit one Redis key at 20,000 QPS. That shard melts.

Mitigations:

Local token bucket in front of Redis
Shard the counter:  rl:user:alice:{0..15}  and sum (weaker fairness)
Isolate abusive keys to a dedicated Redis
Return 429 earlier at the CDN / WAF for obvious junk

Same hot-key story as a viral short URL or a celebrity cache key. Say it before you are asked.

17. Fail open vs fail closed

If Redis times out:

Fail open     allow the request     availability; abuse possible
Fail closed   503 or 429            safety; you become an outage

A Senior answer is per route:

checkout, login, OTP     fail closed
homepage, public GET     fail open + local limiter

Never fail open on password-guessing endpoints.

18. Clients and Retry-After

A correct 429 without jitter creates a second stampede at second 12.

Document for API consumers: exponential backoff with jitter. You already have a visualizer for that pattern: Exponential Backoff and Jitter.

The limiter should not sleep the HTTP worker for the retry. Reject quickly.

19. Consistency and storage

Source of truth     not required for counts; Redis + TTL is enough
Durable rules       config or PostgreSQL
Derived             remaining quota headers
Ephemeral           in-flight token state

Do not put per-request increments in PostgreSQL. That is the view-counter mistake from the YouTube design.

If you need “10,000 SMS per month” for billing, keep a durable monthly aggregate flushed asynchronously, and a real-time Redis cap for abuse. Billing can lag; the cap cannot.

20. Failure scenarios

21. Observability, security, scale

Log sampled rejects with key, route, algorithm — not every allow at 50k QPS.

Metrics: allow/deny, Redis latency, hot keys, fail-open count.

Security: rate-limit by authenticated identity after login; treat X-Forwarded-For as untrusted unless the edge sets it. Hide whether a user exists on login (same 429 shape).

Scale: add Redis shards by key hash; keep the gateway stateless; push obvious junk to WAF/CDN. Multi-region: a limit is usually per region unless you pay for a global counter. Global 100/min across continents adds latency and a new SPOF. State that trade-off.

22. Low-level sketch

Allow(ctx, identity, route) -> (allowed, remaining, retryAfter)

1. load rule (cached)
2. local bucket (optional)
3. Redis Lua token_bucket or INCR window
4. write headers
5. if deny: 429

Gateway middleware calls Allow before the handler. Services call it again for SMS or search fan-out.

Tests: burst of 20 then reject; two concurrent INCR only allow 100; Redis error on /login is deny.

23. Where logic lives

internal/ratelimit
  Allow(identity, route) -> decision
  rules cache
  local token bucket
  redis lua / INCR

The gateway calls Allow before routing. Login and SMS call it again next to the scarce work. Handlers do not increment counters themselves.

24. Final architecture

 Client
   │
   ▼
 ┌─────────────────┐
 │ Edge / WAF      │  coarse IP / ASN limits
 └────────┬────────┘
          │
          ▼
 ┌─────────────────┐
 │ API Gateway     │
 │  1. load rule   │
 │  2. local bucket│  protects this pod
 │  3. Redis Allow │  protects the fleet
 └────────┬────────┘
          │
     ┌────┴────┐
     ▼         ▼
  allowed    429 + Retry-After
     │
     ▼
 Service (search / login / pay)
     │
     └── second Allow() on expensive routes
           fail closed on login / OTP / pay
           fail open on public GET if Redis is down

 Redis cluster
   rl:{identity}:{route}
   atomic INCR or Lua token_bucket

Need a global cap
  → shared Redis, atomic INCR/Lua
Need bursts
  → token bucket
Need smooth outflow
  → leaky bucket
Need simplicity
  → fixed window at the edge
Need to survive Redis
  → per-route fail open/closed + local bucket

25. Interview-ready summary

How to walk through in 10–15 minutes

0–2 min. Alice retries, Bob scrapes, a sale looks like an attack.
2–5 min. Identity, 429, global vs per-pod, fail open vs closed.
5–9 min. Fixed-window burst, then token bucket in Redis (Lua).
9–12 min. Hot keys, Retry-After jitter, billing vs abuse.
12–15 min. Multi-region (usually per-region), local + global.

Key decisions to remember

1. Limit by user/API key, not only IP.
2. Enforce in a shared store; in-memory per pod is not global.
3. Prefer token bucket for product APIs; mention the fixed-window burst bug.
4. Use atomic Redis ops (INCR or Lua), not get-then-set.
5. Return 429 + Retry-After; encourage jittered backoff.
6. Fail closed on login/pay; fail open on cheap public reads.
7. Plan for a hot key.
8. Do not use PostgreSQL for per-request counts.
9. Put a cheap limit at the edge and a precise one next to scarce work.
10. Counts may be approximate; that is acceptable.

Likely interviewer follow-up questions

• Token bucket vs leaky bucket vs sliding window?
• How do two pods not both allow the last token?
• What if Redis is down?
• How do you limit expensive search separately from /health?
• How do you handle a company NAT (many users, one IP)?
• How would you implement 10,000 SMS/month for billing?
• Where do you put the limiter — library, sidecar, or gateway?
• How do you change limits without restarting?
• How does this work in two regions?

Senior-level points that differentiate the answer

• Call out the fixed-window boundary burst before being prompted.
• Separate abuse protection from billing-grade quotas.
• Per-route fail-open / fail-closed, not one global policy.
• Local + global limiting so Redis latency cannot become every request’s p99.
• Hot-key isolation for scrapers and celebrities.
• Never trust client-supplied IP headers blindly.

A 1–2 minute verbal answer

I would place a rate limiter at the API gateway, keyed by authenticated user or API key, with IP as a fallback for anonymous traffic. A typical rule is 100 requests per minute with a short burst, implemented as a token bucket so idle clients can spike briefly. Fixed windows are simpler but allow a double burst at the window edge.

Limits must be global across pods, so state lives in Redis. Each decision is an atomic INCR or a Lua refill-and-consume so two nodes cannot both spend the last token. Denied requests get 429 and Retry-After. If Redis is down I fail closed on login and payment and fail open on public reads, still protected by a small in-process bucket. Hot keys get a local limiter and, if needed, sharded counters. I would not increment PostgreSQL on every request.

You can compare the three algorithms live in the Rate Limiting Visualizer. For the broader interview framework, see the System Design Interview Complete Guide.