Why one server stops being enough

Four things can run out#

A request uses four resources on a server. For each one, divide what the machine has by what one request costs. The smallest result is the request rate where this server stops, and that resource is the bottleneck.

For the launch-day server:

• CPU: 32 × 5,000 = 160,000 requests a second.
• Memory: the 40 GB working set (the posts and profiles read often) fills 31% of 128 GB, so reads come from memory.
• Disk: nearly idle, because those reads never reach it.
• Network: 25,000 requests a second, the lowest, so it is the bottleneck.

Memory is the odd one out. It doesn't cap a rate by itself. When the working set outgrows RAM, reads that used to come from memory go to disk, and disk becomes the bottleneck.

Fixing a resource that isn't the bottleneck costs money and changes nothing, which is what the extra cores did in the clip. The fixes that help here make each response smaller: compress it, drop fields the client doesn't use, or return fewer posts per page. The worked example below does that arithmetic.

A bigger machine comes first#

Vertical scaling means moving to a bigger machine. It is usually the right first fix. It is a configuration change, the code stays the same, and it adds no new ways for the system to fail. The largest cloud instances are many times the size of this 32-core server, which covers most products for years.

It stops being the answer for one of three reasons:

1. There is no bigger machine. You are on the largest instance the provider sells.
2. The price stops scaling linearly. The top sizes cost more per core, so two mid-size machines cost less than one large one.
3. One machine is one failure domain. Whenever it restarts, the product is down.

The third reason is the one that usually forces a second server, long before the first two. An uptime target of 99.99% allows 0.01% of a year down: 525,600 minutes × 0.0001 ≈ 52.6 minutes. Say you deploy five times a week and each deploy restarts the process for 20 seconds. That is 5 × 52 × 20 = 5,200 seconds, about 87 minutes a year, before any hardware fails. One server misses the target even with throughput to spare. Two servers behind a load balancer let you restart one at a time.

When you explain a design, keep the two reasons apart. A second server added for uptime is a spare, and the arithmetic for it is different from a second server added because the first ran out of capacity.

Stateless servers copy, stateful ones don't#

An application server is stateless when it keeps nothing between requests that another server would need. Any request can go to any copy, so ten copies behind a load balancer serve about ten times the traffic, and losing three of them loses capacity but no data.

A database is stateful. A second database is no use until it has the same data, and keeping it the same is replication, which raises a new question: what does a reader see on a copy that hasn't caught up yet? Sharding and partitioning and replication each get their own concept later in the course.

So the usual design keeps the application tier stateless and puts state in a few systems built to hold it, such as the database, a cache and a session store. You scale the stateless part by adding machines and spend your design effort on the stateful part.

What breaks with two servers#

Going from one server to two is a bigger change than going from two to twenty. Three things that worked without any design on one machine now need a decision: where a user's session lives, how requests reach a server that can actually serve them, and what happens when two servers change the same row at once. The next three sections take one each.

The session is on the other server#

On one server, keeping sessions in a dictionary inside the process works, because every request reaches the process that stored them. Now put two servers behind a load balancer using round robin, which sends each new request to the next server in turn.

There are three common ways out:

• A shared session store such as Redis. Every server reads and writes sessions there. It is one more system to run and one more network hop per request.
• Signed tokens. The session (user id, expiry) travels with each request inside a token the server signed, so any server can check it without a lookup. The cost is logging someone out early: a token stays valid until it expires unless you also keep a list of revoked tokens.
• Sticky sessions. The load balancer sends a user to the same server every time. It is the quickest change and the weakest. When that server dies or is redeployed, its users lose their sessions.

Any cache kept inside the process has the same problem. With two servers there are two caches, and they can disagree.

A health check that only says "running"#

A load balancer needs to know which servers can take traffic. It calls a health check endpoint on each server every few seconds and stops sending requests to one that fails.

A useful health check tests what a real request needs from that server, such as getting a connection from its own pool, with a short timeout. Keep it to things one server can get wrong on its own. If every server's check runs a query against the same database and the database is slow for a few seconds, every server fails at once and the load balancer has nowhere left to send traffic. Many setups split the check in two: a liveness check (is the process alive; restart it if not) and a readiness check (can it serve right now; take it out of rotation if not).

Two servers, one counter#

On one server, a lock inside the process could make these requests take turns. Two servers share no memory, so the fix has to live where the data is:

• Atomic update. Put likes = likes + 1 in the UPDATE itself, in place of a number the server read earlier. The database applies the increments one at a time.
• Conditional write, also called optimistic locking. Keep a version column, write only WHERE version = <the one I read>, and retry if no row matched.
• Row lock. SELECT ... FOR UPDATE holds the row until the transaction commits. Simple, but every other writer to that row waits.

Time has the same problem. Each server has its own clock and the clocks drift apart, so ordering events by timestamps written on two different machines can put them in the wrong order. Several later problems in this course, like the duplicate run in a job scheduler, come from two machines each acting on their own view of "now".

Worked example: how many API servers?#

Back to the hypothetical Instagram. Assume 20 million daily active users, each loading the feed 30 times a day, with peak traffic at 3× the daily average. Every server is the launch-day machine: 32 cores, a 10 Gb/s card, 0.2 ms of CPU and 50 KB per feed response. Plan to run each server at 60% of its limit to leave headroom.

Load

• 20M × 30 = 600M feed loads a day.
• 600M ÷ 86,400 s ≈ 6,944 a second on average, or about 20,833 at peak.

Servers as they are

• Network limit 25,000 a second, CPU limit 160,000. Network is the bottleneck.
• At 60% that is 15,000 a second per server. 20,833 ÷ 15,000 ≈ 1.39, so 2 servers for the load, plus 1 spare so a deploy or a failure doesn't drop you below peak. 3 servers.

Servers with compressed responses

• Assume gzip shrinks the JSON 4×, to 12.5 KB, and costs another 0.2 ms of CPU per response.
• Network: 1.25 GB ÷ 12.5 KB = 100,000 a second. CPU: 32 × (1,000 ÷ 0.4) = 80,000 a second. CPU is now the bottleneck.
• At 60% that is 48,000 a second per server. 20,833 ÷ 48,000 ≈ 0.43, so 1 server carries peak.

Answer

2 servers: one for the load and one spare. If one fails, the other carries all of peak at 20,833 ÷ 80,000 ≈ 26% of its limit. Finding the bottleneck saved a machine, and the second server is there for uptime. On throughput alone, one server lasts until peak reaches 48,000 a second, which is 48,000 ÷ 3 × 86,400 ÷ 30 ≈ 46 million daily active users.

Those 2 servers bring the three problems above with them: sessions go in a shared store or a signed token, the health check asks for a database connection, and like counts use an atomic increment.

How this comes up in an interview#

"How many servers do you need?" Work it out per resource, as in the worked example, and name the one that runs out. "Network-bound at about 25,000 requests a second per server" is a stronger answer than a round number of machines.

"Why not just add more servers?" For a stateless tier that is often right, once you have the numbers. For a database it means replication or sharding, so a bigger machine, better indexes and a cache come first.

"Where do sessions live?" A shared store or signed tokens, with the cost of each. If you mention sticky sessions, say what happens when that server goes away.

"How does the load balancer know a server is healthy?" A readiness check that tests the server's own dependencies with a short timeout. Interviewers who know this area will ask what happens when a shared dependency makes every check fail at once.

"Two users like the same post at the same moment. What happens?" Name the lost update, then the atomic increment. If they push on a post with millions of likes, all landing on one row, keep the count in a cache and write it to the database periodically, as on the sharding page.

"Do you need a second server at all?" Often not for throughput. Usually yes for uptime, and naming which reason you are using shows you know the difference.

Next: the arithmetic#

Every number on this page came from a handful of inputs: users, actions per user, and the bytes and milliseconds each request costs. The next concept, back-of-the-envelope math, is how to come up with those inputs for a system you have never seen, quickly and on a whiteboard.