Tonight's interview shape is common: protect an API from one client, one IP, or one feature suddenly sending too much traffic. A rate limiter is the bouncer before expensive work. It lets normal traffic through, rejects excess traffic early, and gives the interviewer a clean way to test your trade-off thinking.
The invariant
A rate limiter answers “has this key spent its budget in this window?” — token bucket and sliding window are just different bookkeeping for the same budget.
Keep the word key concrete. It might be a user id, API key, IP address, route, tenant, or a combination like user:123:/payments. The whole design is built around checking and updating that key's budget before the request reaches the service.
The script
Use the same system-design script every time: requirements → estimate scale → API → data model → high-level boxes → deep-dive the bottleneck → failure modes.
Requirements. Limit requests per key, return a clear rejection, allow short bursts, and let operators change limits safely.
Estimate scale. If the API sees 10,000 requests per second, the limiter also sees 10,000 checks per second. It must be cheap and close to the API edge.
API. Internally, the service asks allow(key, cost=1). Externally, rejected HTTP requests usually become 429 Too Many Requests, often with Retry-After.
Data model. Store a policy and state per active key: limit, window, remaining budget, and the timestamp needed by the chosen algorithm.
High-level boxes. Put the limiter before the application service. Keep budget state in a fast shared store so all API servers spend from the same budget.
Deep-dive bottleneck. The interesting part is the atomic budget check: check remaining budget and spend one unit as one operation.
Failure modes. Decide fail-open versus fail-closed, watch hot keys, handle clock skew, expose metrics, and make limits configurable.
The architecture diagram
The diagram's key move is the fork after the limiter. Allowed requests continue to the service. Denied requests never consume database, queue, or worker capacity.
The bottleneck deep-dive
The bottleneck is not drawing boxes. It is making every API server agree on the same remaining budget for the same key. That is why the check and update live in one shared, atomic operation. Redis increments counters in O(1), and Redis Lua scripts can run the multi-step check-and-spend logic atomically on the server.
Fixed window counter. Count requests in key:minute. Cheapest to explain and implement, but a client can burst at the edge of two windows.
Token bucket. Store current tokens and last refill time. This is a strong default: steady rate, controlled burst, O(1) state per active key.
Sliding window. Count only the last rolling window. Fairer near boundaries, but exact logs store more data; approximations are cheaper but less exact.
In an interview, say the trade-off plainly: fixed window is simple, token bucket is smooth with bursts, sliding window is fairer but costs more bookkeeping.
Numbers that matter
Figure
Why it matters
1 check per request
The limiter sits on the hot path, so the budget decision must be fast and close to the API edge.
O(1) counter update
Simple Redis counters use INCR, documented as O(1), which is why they are common for fixed-window limits.
O(1) bucket state
Token bucket keeps tokens plus last refill time per active key; that stays small even with many users.
O(r) sliding log
An exact sliding log stores request timestamps in the window, where r is requests by that key during the window.
429 rejection
HTTP 429 means Too Many Requests; the RFC says the response may include Retry-After.
Practice — answer before you scroll past
1. Where should the limiter usually sit?
The limiter protects scarce capacity by rejecting excess traffic before the application spends database, queue, or worker resources.
2. Why must check-and-spend be atomic?
If two servers read the same remaining budget before either writes, both can allow requests that should not both pass. Atomic logic prevents that race.
3. Which bookkeeping fits smooth refill plus short bursts?
Token bucket refills gradually and lets unused tokens accumulate up to a cap, so it supports a steady average rate plus controlled bursts.
4. When Redis is unreachable, what is the interview answer?
Name the policy and its trade-off. Fail open protects availability but may overload the backend; fail closed protects the backend but may reject valid users.
Your move
Redraw the architecture diagram from memory on paper. Then explain the whole design out loud in 5 minutes as if to an interviewer:
State the invariant: budget, key, window.
Walk the 7-step script in order.
Choose token bucket as your default and explain why.
Name two failure modes: Redis down and hot keys.
For the encode step, put one line in your Notion tracker: what the key is, where the budget lives, and which algorithm you chose.
One primary source
After the redraw attempt, read Stripe — Scaling your API with rate limiters. It is a production write-up, and the useful compare step is to notice how they separate request limiting from broader load shedding.
Stuck or curious? Ask your teacher (the agent) — that's what it's for.