An interviewer can turn a tiny service into a scaling question with one sentence: “What happens when one server is not enough?” Load balancing is the first answer. It is not magic capacity; it is a routing layer that decides which healthy, interchangeable server should receive each request.
The invariant
A load balancer spreads requests across interchangeable servers; consistent hashing keeps most keys in place when a server joins or leaves the pool.
Keep those two halves separate. Plain load balancing is for interchangeable servers. Consistent hashing is for keyed work where moving every key would cause cache misses, session loss, or noisy data movement.
The script
Use the same seven-step system-design script every time: requirements → estimate scale → API → data model → high-level boxes → deep-dive the bottleneck → failure modes.
Requirements. Route web requests to healthy app servers. Keep latency low. Survive one server failing. Allow adding servers without a risky deploy.
Estimate scale. Pick a round number: 20,000 requests per second at peak, 10 app servers, mostly stateless reads. That gives a first-pass average of 2,000 requests per second per server before headroom.
API. The client still calls normal HTTP endpoints like GET /feed or POST /checkout. The load balancer is invisible to the API contract.
Data model. Keep app servers stateless when possible. If a request must stick to a cache shard or session owner, choose a stable key such as user_id, session_id, or cache_key.
High-level boxes. Client → DNS → load balancer → healthy server pool. A health checker continuously removes bad targets from rotation.
Deep-dive the bottleneck. The interesting question is selection: round robin, least connections, or consistent hash?
Failure modes. Servers fail, health checks lie, one key becomes hot, one load balancer becomes a single point of failure, and deploys can drain too slowly.
The architecture diagram
Read the diagram left to right first, then study the ring. The left side handles ordinary stateless requests. The right side shows the special case: keyed routing where most keys should stay put as the pool changes.
The bottleneck deep-dive
The bottleneck is the load balancer's selection policy. NGINX documents several real policies, including round robin, least connections, IP hash, and generic hash with a consistent option. In an interview, name the policy and the trade-off.
Round robin. Simple and fair when servers are equal and requests cost about the same. Weakness: one slow request can leave a server busy while the next turn still arrives.
Least connections. Better when request duration varies, because the balancer prefers the server with fewer active connections. Weakness: it needs fresh connection counts and can chase short-lived spikes.
Consistent hash. Better for caches, sessions, and sharded state because the same key usually lands on the same server. Weakness: hot keys remain hot, so large systems add virtual nodes, replicas, or a fallback for overloaded keys.
The clean interview sentence: stateless traffic starts with round robin or least connections; stateful keyed traffic needs consistent hashing or another sticky-routing strategy.
Numbers that matter
Figure
Why it matters
Source
20,000 RPS / 10 servers = 2,000 RPS each
First-pass capacity math before headroom, uneven load, or failover.
Interview arithmetic
One server lost from 10 means survivors take about 11% more
Failover needs spare capacity, not just healthy routing.
Interview arithmetic
ALB health checks default to 30s for instance or IP targets
Bad targets are not removed instantly; detection time affects errors.
1. A service is stateless and every server is identical. Which first policy is easiest to explain?
Round robin is the clean default for interchangeable servers. It spreads requests without needing a key, session, or per-request state.
2. When a cache server joins a keyed pool, what does consistent hashing protect?
Consistent hashing protects key placement. A joining or leaving server should disturb only the neighboring slice of the ring, not the whole cache.
3. Long file uploads make connections pile up, but CPU looks fine. Which policy reacts better than plain round robin?
Least connections uses active connection count as a signal, so a server tied up by long uploads is less likely to receive the next request.
4. A load balancer marks slow servers healthy because /health only checks the process. What should change?
A health check should match the promise users need: the server can answer real traffic quickly enough, including critical dependencies.
Your move (tonight)
Close this page and redraw the architecture diagram from memory on paper: clients, DNS, load balancer, healthy server pool, health checker, and consistent-hash ring. Then explain the whole design out loud in 5 minutes as if to an interviewer, using the seven-step script.
One primary source
Read NGINX — HTTP Load Balancing after your redraw, as the compare step. Focus on how each policy changes the answer to “which server gets this request?”
Stuck or curious? Ask your teacher (the agent) — that's what it's for.