Latency numbers, and where the time actually goes
Orders of magnitude, not exact figures#
Nobody will ask you to recite a table. What they will do is watch whether you notice that a design puts a 150ms cross-region call inside a 100ms SLO. That only requires knowing the rough scale of things, and the scale is remarkably easy to remember because it moves in steps of roughly a thousand.
| Operation | Time | In human terms |
|---|---|---|
| L1 cache reference | 1 ns | 1 second |
| Main memory reference | 100 ns | 2 minutes |
| Compress 1KB | 2 µs | 30 minutes |
| SSD random read | 100 µs | 1 day |
| Read 1MB sequentially from memory | 250 µs | 3 days |
| Round trip within a datacentre | 500 µs | 6 days |
| Read 1MB sequentially from SSD | 1 ms | 12 days |
| Disk seek (spinning) | 10 ms | 4 months |
| Read 1MB from network | 10 ms | 4 months |
| Round trip, California to Netherlands | 150 ms | 5 years |
The right-hand column is the one to internalise. If a memory reference took a second, going to disk would take four months and crossing an ocean would take five years. That intuition is what makes a bad design feel wrong before you have done any arithmetic.
Three consequences fall straight out:
Memory is effectively free; the network is not. This is why caching works at all, and why the entire caching concept later in this track is really about avoiding hops rather than avoiding computation.
Sequential beats random by a lot. Reading 1MB sequentially from SSD costs about the same as ten random reads. Data layout — clustering rows you read together — is a performance decision, not a tidiness one.
Geography is a hard floor. Light in fibre does about 200,000 km/s. London to New York and back is roughly 11,000 km, so about 55ms of pure physics before any processing. No engineering removes it. You can only avoid making the trip.
Build the budget backwards#
Given a target — say, 200ms at p99 for a page load — work backwards and assign every hop a slice. A realistic budget for a cached read:
| Hop | Budget | Notes |
|---|---|---|
| Client → edge (TLS established) | 20 ms | Depends entirely on user geography |
| Edge → origin region | 30 ms | Zero if the edge serves it |
| Load balancer → app server | 1 ms | Same datacentre |
| App → cache (hit) | 1 ms | Redis round trip in-region |
| App → database (on miss) | 5–20 ms | Indexed query, memory-resident |
| Serialisation and app logic | 10 ms | Usually underestimated |
| Response back to client | 20 ms | Symmetric with the request |
That totals something like 60–90ms on a cache hit, leaving genuine headroom. The same budget with one cross-region database call is dead on arrival: 150ms of the 200ms goes to a single round trip.
When someone proposes a component, ask where it sits and how many round trips it adds. A design with four sequential service hops inside one datacentre has spent 2ms on network — fine. The same four hops across regions is 600ms, and no amount of query tuning fixes it.
Sequential hops add; parallel hops take the max#
A request that calls A, then B, then C, waits for all three in turn. If each is 20ms, the request is 60ms. A request that calls all three at once and waits is about 20ms — assuming they are genuinely independent.
This is one of the few places where a design change buys a large, easy win, and it is worth naming explicitly when the dependency graph allows it. It is also where tail latency stops being an academic concern.
Why p99 is the number, not the average#
Averages hide the experience. A service averaging 10ms with a 400ms p99 is not a fast service — it is a service that is slow for one user in a hundred, and those are disproportionately your heaviest users, because the more requests you make the more likely you are to hit the tail.
Fan-out makes this much worse. Suppose a page assembles itself from 20 parallel backend calls and cannot render until all 20 return. Each call independently has a 1% chance of landing in its slow tail. The chance that none of them does is 0.99²⁰ ≈ 82%, so about 18% of page loads contain at least one tail-latency call — and the page is only as fast as its slowest component.
The general shape: your p99 becomes the user's median far faster than intuition suggests. Systems with heavy fan-out have to attack the tail directly, usually with hedged requests (send to two replicas, take the first answer) or by capping the fan-out.
First, quote a latency target as a percentile, never as an average — "p99 under 200ms" is a requirement, "fast" is not. Second, when you fan out, say what happens to the tail, because the interviewer is waiting to see whether you noticed.
Where the time usually actually goes#
In practice, the dominant cost in most real requests is not the one people optimise:
- TLS handshakes and connection setup, when connections are not pooled. A new TLS connection is two extra round trips; at 50ms each, that is 100ms before a byte of payload moves.
- Serialisation, especially JSON over large payloads. Often larger than the query it wraps.
- Queueing delay under load. A service at 80% utilisation has queueing time comparable to its service time — which is why latency curves bend sharply upward near saturation rather than degrading linearly. This is the single most common reason a system that looked fine in a load test falls over in production.
- The slowest shard. Any scatter-gather query is bounded by its worst participant.
Next#
You can now size a system and budget its latency. The next stage is the machinery that acts on those numbers: how a request finds a healthy machine, and how caching removes the hops you just learned to count.
Where this shows up
This page is the mechanism on its own. Each problem below bends it to a constraint that page has and this one does not.