Why one server stops being enough
One machine is bigger than you think#
Most system design answers reach for a fleet of servers in the first thirty seconds. That instinct is wrong often enough that interviewers use it as a filter. A single modern server — say 32 cores and 128GB of RAM — will serve tens of thousands of simple HTTP requests per second, hold a hundred million small rows comfortably in memory, and saturate a 10Gb network card long before it runs out of CPU.
So the useful question is not "how do I scale this?" It is "which resource runs out first, and at what number?" Everything in this track follows from being able to answer that.
There are only four candidates.
| Resource | Runs out when | Typical symptom |
|---|---|---|
| CPU | Request handling is compute-bound — parsing, serialisation, compression, crypto | High load average, latency climbs with traffic, cores pinned |
| Memory | The working set no longer fits | Swapping, cache hit rate collapsing, OOM kills |
| Disk I/O | Reads miss memory and hit the device; writes outpace flush | High queue depth, iowait, p99 latency spikes with a flat average |
| Network | Response bytes × requests per second exceeds the link | Throughput plateaus while CPU stays low |
The failure that matters is naming the wrong one. A service that is network-bound does not get faster on a machine with more cores, and a team that adds application servers to fix a database's disk contention has bought themselves more connections competing for the same spindles.
When you are asked to scale something, say which resource you expect to bind and roughly at what traffic level. "I'd expect this to be read-heavy and memory-bound — it stops fitting in RAM somewhere around 50 million users" is a far stronger opening than "we'll add caching and horizontal scaling."
Vertical first, and say so out loud#
Vertical scaling — a bigger machine — has an unfashionable reputation and is almost always the right first move. It costs one line of configuration. It introduces no new failure modes. It does not change your consistency story. Cloud instances go up to hundreds of cores and terabytes of RAM, which covers the entire lifetime of most products.
It stops working for three specific reasons, and being able to name them is the point:
- You hit the largest instance that exists. A real ceiling, but a distant one.
- The price stops being linear. The top instance sizes cost disproportionately more per core, so at some point two medium machines are cheaper than one large one.
- One machine is one failure domain. This is usually the real reason. A single server cannot be restarted, patched or lost without downtime. Availability, not throughput, is what forces most teams to a second machine.
That third reason is worth sitting with. Teams often describe a horizontal architecture as a performance decision when it was really an uptime decision. Both are legitimate; conflating them makes it impossible to reason about the trade-off.
Stateless and stateful scale differently#
Here is the distinction that the rest of this track hangs on.
An application server typically holds nothing that matters between requests. Two of them are interchangeable. You can put ten behind a load balancer, lose three, and the system keeps working. Scaling is close to free: double the machines, roughly double the throughput.
A database holds state. A second one is not interchangeable with the first, because it does not have the same data — and the moment you try to give it the same data, you have invented replication, and with it the question of what a client sees when it reads from a replica that has not caught up yet.
This is why the usual advice is to push state out of the application tier and concentrate it in as few stateful systems as you can. Not because state is bad, but because state is the expensive thing to scale, and you want to be paying that cost in one place where you can think about it carefully.
What breaks the moment there are two#
Going from one server to two is a bigger jump than going from two to twenty. Three things that were free suddenly are not:
State stops being shared. An in-process session map or local cache worked because every request hit the same machine. With two machines, a user's second request lands somewhere that has never heard of them. You now need sticky sessions (fragile), a shared session store (a new dependency), or genuinely stateless requests carrying their own auth.
Requests need routing. Something has to decide which machine gets a request, notice when one is unhealthy, and stop sending traffic there. That is a load balancer, and it needs a health check that actually reflects health — an endpoint returning 200 while the database connection pool is exhausted is worse than no health check.
"Now" stops being well-defined. Two machines have two clocks, drifting apart by milliseconds. Two machines reading the same counter can both see the old value and both write the new one. Every distributed race condition you will meet later in this track — the double-spend in a rate limiter, the duplicate charge in a job scheduler — is a consequence of this one fact.
A single server until it genuinely cannot cope. Then a bigger one. Then a second, accepting that you have traded a performance problem for a coordination problem. Each step should be forced by a named constraint, not assumed at the start.
The number that decides it#
Everything above is qualitative, and interviews are not won with qualitative answers. The next concept is the arithmetic: how to take "design Twitter" and produce requests per second, bytes per day and a machine count, in about three minutes and out loud. Without those numbers, "we'll need to shard" is a guess. With them, it is a conclusion.
Where this shows up
This page is the mechanism on its own. Each problem below bends it to a constraint that page has and this one does not.