# Design WhatsApp - Source: https://intervueclub.com/problems/realtime/design-whatsapp - Author: Dhananjay Aggarwal - Category: Real-time Systems - Difficulty: hard - Asked at: Meta, Google, Microsoft, Apple, Telegram, Signal - Last updated: 2026-07-02 A complete system design walkthrough for designing a chat system like WhatsApp. Covers WebSocket connection management, delivery receipts, offline queueing, group fan-out, message ordering, Kafka routing, failure scenarios, and everything an interviewer actually evaluates. ## Problem Statement > "Design a chat system like WhatsApp. Users should be able to send messages to each other and to groups, in real time." That's it. One sentence. The rest is yours to figure out. --- ## Requirements Gathering This is where the interview actually begins. A strong candidate doesn't start drawing boxes. They ask questions. Not to stall, but because the answers fundamentally change the design. Here's how a good requirements conversation looks: --- **Candidate:** "Are we designing 1:1 messaging, group messaging, or both? And what's the max group size?" **Interviewer:** "Both. Groups up to about 1,000 members." --- **Candidate:** "Do we need delivery and read receipts, the sent, delivered, read distinction?" **Interviewer:** "Yes. The double-tick and blue-tick behavior is in scope." --- **Candidate:** "What scale? Daily active users and message volume?" **Interviewer:** "500 million DAU, 100 billion messages per day." --- **Candidate:** "How fast must delivery be when both users are online?" **Interviewer:** "Sub-second. Under 500 milliseconds end to end." --- **Candidate:** "Do we persist history? If someone reinstalls the app, do they get their messages back?" **Interviewer:** "Yes. Messages are durable and retrievable. Never lose a confirmed message." --- **Candidate:** "Text only, or media too, images, video, documents?" **Interviewer:** "Text, images, video, documents. But you can treat media as an upload-then-reference flow." --- **Candidate:** "Out of scope: end-to-end encryption, voice and video calls, reactions and threaded replies?" **Interviewer:** "Correct, out of scope. Just reliable, ordered message delivery." --- ## Functional and Non-Functional Requirements ### Functional Requirements 1. Users can send and receive text messages in real time (1:1 conversations) 2. Messages show delivery status: sent, delivered, read (the famous double tick) 3. Users can message groups of up to ~1,000 members 4. Messages are durable and retrievable, so a reinstalled client can load history 5. Users can send media: images, video, documents 6. Online/offline presence indicator for contacts ### Non-Functional Requirements | Requirement | Target | |---|---| | Delivery latency (both online) | under 500ms p99 | | Durability | Never lose a message once the sender is ACKed | | Ordering | In-order delivery within a conversation | | Availability | 99.99% | | Scale | 500M DAU, 100B messages/day | **Out of scope:** End-to-end encryption (Signal Protocol), voice/video calls, reactions and threaded replies, WhatsApp Business API ### Capacity Estimates Work through the math. Don't just state the numbers. ``` Message throughput: 100B messages/day ÷ 86,400 sec = ~1.16M messages/sec Concurrent connections (assume 20% of DAU online at once): 500M × 0.20 = 100M concurrent WebSocket connections Chat servers (each holds ~100k persistent connections): 100M ÷ 100k = ~1,000 chat servers Storage (text only, avg 100 bytes/message): 100B × 100B = 10 TB/day of message data At replication factor 3: ~30 TB/day written Group fan-out amplification: A 1,000-member group message = up to 1,000 delivery attempts This is the write-amplification number to watch ``` The number that should make you pause: **100M concurrent persistent connections.** You can't poll your way to that: the connection model *is* the core of this design. --- ## High-Level Design Two things drive everything: holding millions of live connections, and routing a message from one connection to another reliably. Keep them separate in your head. ``` [Client A] ⇄ WebSocket ⇄ [Chat Server A] → [Kafka] → [Chat Server B] ⇄ WebSocket ⇄ [Client B] ↓ ↓ [Message Store: Cassandra] [Presence: Redis] ↓ [Media Service → S3] ``` **Write path:** Client A sends over its WebSocket to Chat Server A → server persists to Cassandra (status: sent) → publishes to Kafka → ACKs Client A immediately (single tick). A router stage consumes from Kafka and, for each recipient, asks the **presence service** which server currently holds that recipient's socket (`user:{id} → server_id` in Redis) and forwards the message *only* to that one server, which pushes it over the recipient's WebSocket. The forward is directed, not broadcast: you never fan a message out to all 1,000 chat servers to find the one that owns the connection. If presence says the recipient is offline, delivery is skipped and the reconnect scan handles it. **Read path (history / offline):** On reconnect, a client sends its last-seen message ID; the server range-scans Cassandra for anything newer across its conversations and delivers it over the fresh connection. The key insight: **the ACK to the sender is decoupled from delivery to the recipient.** The sender learns the server durably has the message (single tick) *before* the recipient is even reachable. Kafka is only the real-time routing layer; Cassandra is the source of truth, so a message dropped from the queue is never lost. --- ## Deep Dives ### 1. Connection Model: WebSocket vs Long Polling vs SSE This is the first fork in the road. You need bidirectional, low-latency, persistent connections for 100M clients. **HTTP long polling:** client requests, server holds it open until a message arrives, client immediately re-requests. Wastes connections, high overhead per message, not truly real-time. A fallback, not a design. **Server-Sent Events (SSE):** server pushes to client over one persistent HTTP stream, but it's one-directional. The client can't send over the same channel, so you'd need a second channel for outbound messages. Wrong shape for chat. **WebSocket (recommended):** full-duplex persistent connection. One handshake, then low-overhead frames in both directions. Client and server both send at will. This is what every serious chat app uses (WhatsApp historically used a customized XMPP over the same idea). The connection stays open while the app is foregrounded. Chat servers are therefore **stateful** (each holds ~100k live sockets) but the per-connection state is tiny, so they scale horizontally behind a connection-aware load balancer. ### **For senior interviews ↓** connection management at 100M sockets Holding 100M sockets is an infrastructure problem in its own right. **Heartbeats and dead connections.** TCP won't tell you promptly that a mobile client walked into a tunnel. Application-level ping/pong every ~30s detects dead sockets so you can free resources and mark the user offline. Too aggressive and you drain phone batteries; too slow and presence lies. **Reconnection storms.** When a chat server crashes, its ~100k clients all reconnect at once. Without jittered backoff they stampede the remaining servers and cascade the failure. Clients must reconnect with randomized exponential backoff, and the LB must spread them. **Sticky routing.** A message for user B must reach *the specific server* holding B's socket. That's the presence service's job (`user:{id} → server_id` in Redis), and it must be updated on every connect/disconnect and consulted (or bypassed via Kafka broadcast) on every delivery. **Graceful drain.** On deploy, a server stops accepting new connections, tells its clients to reconnect elsewhere, and drains, rather than dropping 100k sockets at once. --- ### 2. Message Flow and Delivery Receipts This is the most nuanced part of the design and where strong candidates separate themselves. WhatsApp has four tick states: clock (sending), single tick (server received), double tick (delivered to device), and blue tick (read). Each requires a precise acknowledgement traveling back up the chain. **1:1 message flow:** ``` 1. Client A → Chat Server A over WebSocket 2. Chat Server A persists to Cassandra (status: sent) 3. Chat Server A publishes to Kafka topic "messages" 4. Chat Server A → ACK to Client A → single tick ✓ 5. Chat Server B consumes from Kafka 6. Chat Server B → delivers to Client B over WebSocket 7. Client B → delivery ACK to Chat Server B 8. Chat Server B updates status: delivered in Cassandra 9. Chat Server B → status event back via Kafka 10. Chat Server A → pushes status to Client A → double tick ✓✓ 11. Client B opens the chat → sends read receipt 12. Read receipt propagates to Client A → blue ticks ``` The key insight: **step 4 happens before step 6.** The sender gets immediate confirmation the server durably holds the message, independent of whether the recipient is reachable. Delivery and read are later, separate hops. ### **For senior interviews ↓** exactly-once feel over at-least-once plumbing Kafka gives at-least-once delivery, and WebSocket pushes can be retried after a missed ACK, so the recipient can receive the same message twice. The user must never see a duplicate. **Dedup on the client-generated `temp_id`, not on `message_id`.** The subtlety people get wrong: a retry generates a *fresh* `message_id` on the server, so a Cassandra `IF NOT EXISTS` on `message_id` catches nothing. Worse, `IF NOT EXISTS` is a lightweight transaction (Paxos, ~4 round-trips per write), which you cannot afford on a 1.16M-writes/sec hot path. Instead the *client* stamps every outbound message with a stable `temp_id`. The server keeps a short-TTL `temp_id → message_id` map (Redis); on a retry it finds the existing mapping and returns the *same* `message_id` instead of writing a second row. The Cassandra insert itself is then a plain, cheap write, idempotent because re-inserting the same `(conversation_id, message_id)` is a harmless overwrite. **Client-side render dedup.** Because Kafka is at-least-once and WebSocket pushes can be retried, the recipient may still receive the same `message_id` twice. The client dedupes on `message_id` before rendering, turning at-least-once delivery into an exactly-once *experience*. **Receipt batching.** Read receipts for an actively-used chat can be extremely chatty. Batch them ("read up to message_id X") instead of one receipt per message, collapsing N receipts into one. --- ### 3. Offline Message Queueing When the recipient is offline, Chat Server B can't deliver over a socket that doesn't exist. The message already sits durably in Cassandra with status `sent`. When Client B reconnects: 1. Client B connects to *any* chat server (load balanced) 2. Client B sends its last-seen message ID 3. Server range-scans Cassandra for messages after that ID across all of Client B's conversations 4. Server delivers the backlog over the new WebSocket, in order 5. Client B ACKs, statuses flip to `delivered`, senders see double ticks The key: messages are **never** stored only in memory or only in Kafka. Cassandra is the durable source of truth; the queue is just the fast path for online delivery. If a message is never consumed from Kafka, it's still safely in Cassandra waiting for the reconnect scan. --- ### 4. Group Messaging and Fan-Out A group message to 500 people is fundamentally different from a 1:1 message. First, separate two things people conflate, **storage fan-out** and **delivery fan-out**, because they get different answers here. **Storage: one shared log, not per-member copies.** The message is written *once* to the conversation's partition (`messages` keyed by `conversation_id`, per the data model). All members read that same partition; per-member read state lives in `conversation_members.last_read_id`. That's storage fan-out on *read*: storing 1,000 copies of every group message would be wasteful and it's not what the schema does. **Delivery: fan out to each online member.** Real-time push *is* per-member: the router resolves each member's server via presence and forwards to whoever is currently connected. This is bounded work because group size is capped, so it stays tractable. ``` # STORAGE: always one write to the shared conversation log. append_to_conversation_log(conversation_id, message_id) # sender ACKed here # DELIVERY: async, per online member, bounded by the ~1,000 cap. enqueue_delivery_fanout(message_id, member_ids) # router → presence → push ``` Do the delivery fan-out **asynchronously**: the sender gets its single tick after the one durable log write, and a background job drives the up-to-1,000 directed pushes. For unbounded broadcast channels (millions of subscribers) even delivery fan-out on write breaks, and you flip fully to pull: subscribers fetch the shared log on read, and you drop per-subscriber receipts. ### **For senior interviews ↓** why the cap is the whole trick The reason WhatsApp can push where a social feed must go hybrid is the hard **1,024-member cap**. Worst-case fan-out is a known, small constant, so you never face the celebrity problem: no single message explodes into 10M writes. Contrast with [Design a News Feed](/problems/databases/design-a-news-feed), where follows are unbounded and one celebrity post would fan out to tens of millions, forcing a pull path for high-follower accounts. For broadcast-style channels (WhatsApp Channels, Telegram broadcast) the audience *is* unbounded, and there the design flips back to fan-out on read with a shared log plus a heavy read-side cache, exactly the feed problem again. Naming this boundary explicitly is what signals senior-level judgment. --- ### 5. Message Ordering Within a conversation, messages must arrive in order. Two hazards: clock skew between servers, and concurrent messages from both participants. Be precise about what solves what here, because a common answer ("just use TIMEUUID") is only half right. A TIMEUUID's high bits *are* a wall-clock reading (100ns ticks since the Gregorian epoch), generated on the same potentially-skewed server, so TIMEUUID does **not** by itself immunize you against clock skew. What it buys is **global uniqueness and a deterministic tiebreak**, and, critically, a **single, consistent total order** that every reader of the partition agrees on. ``` conversation_id (partition key) message_id (clustering column, TIMEUUID / UUIDv7, DESC) sender_id content status ``` Two levers give you *correct* ordering, not just consistent ordering: 1. **Stamp the ID at one authoritative point**: the recipient-facing server (or a per-conversation sequencer) that owns the partition, rather than trusting each client/sender's clock. That removes cross-sender skew from the ordering decision. 2. **For strict correctness** (financial-grade "message N always precedes N+1"), assign a **per-conversation monotonic sequence number** from that single owner. It costs a counter but eliminates skew entirely; TIMEUUID is the pragmatic default when "consistent order everyone agrees on" is good enough. Partitioning by `conversation_id` keeps a whole conversation on one partition, so a range scan returns it already ordered. --- ## Data Model ```sql -- Messages (Cassandra, partitioned by conversation for ordered range scans) messages ( conversation_id UUID, -- PARTITION KEY message_id TIMEUUID, -- CLUSTERING KEY DESC (time-ordered) sender_id UUID, content TEXT, -- null for media media_url TEXT, -- null for text message_type VARCHAR, -- text | image | video | doc status VARCHAR, -- sent | delivered | read created_at TIMESTAMPTZ, PRIMARY KEY (conversation_id, message_id) ) -- Conversations (Postgres) conversations ( id UUID PRIMARY KEY, type VARCHAR, -- direct | group created_at TIMESTAMPTZ ) -- Membership + read tracking (Postgres) conversation_members ( conversation_id UUID, user_id UUID, joined_at TIMESTAMPTZ, last_read_id TIMEUUID, -- drives read receipts PRIMARY KEY (conversation_id, user_id) ) -- Presence (Redis) -- Key: user:{id} → server_id (which chat server holds the socket) -- TTL refreshed by heartbeat; expiry = offline ``` --- ## API Design ``` WebSocket: wss://chat.example.com/ws ───────────────────────────────────────────── Client → Server (send): { "type": "message", "temp_id": "client-uuid", // idempotency for retries "conversation_id": "conv-uuid", "content": "Hey!", "message_type": "text" } Server → Client (ACK, single tick): { "type": "ack", "temp_id": "client-uuid", "message_id": "server-timeuuid", "status": "sent" } Server → Client (incoming message): { "type": "message", "message_id": "server-timeuuid", "conversation_id": "conv-uuid", "sender_id": "user-uuid", "content": "Hey!", "created_at": "2026-07-02T10:00:00Z" } REST (history / reconnect): GET /api/v1/conversations/:id/messages?before=&limit=50 Authorization: Bearer Response 200: { "messages": [ ... ], "has_more": true, "next_cursor": "" } ``` Note: `sender_id` and identity always come from the auth token server-side, never from a client-supplied field. The client's `temp_id` is only an idempotency hint; the server assigns the authoritative `message_id`. --- ## Failure Scenarios and Edge Cases ### **For senior interviews ↓** what breaks and how the system recovers **Chat server crashes mid-delivery** If a server dies after persisting a message but before ACKing the client, the client retries with the same `temp_id`. Cassandra writes are conditional on `message_id`, so the retry doesn't create a duplicate, and the server returns the original `message_id`. No lost message, no double message. **Recipient offline for days** The message stays durably in Cassandra with status `sent`. On reconnect, the last-seen-ID range scan delivers the whole backlog in order. Retention policy (e.g. 30 days of undelivered) bounds storage; beyond that, undelivered messages expire. **Presence lies (stale online status)** A phone drops off the network without a clean disconnect. Until the heartbeat times out, presence says "online" and delivery is attempted to a dead socket. The delivery fails, the message falls back to the offline path, and presence flips to offline on heartbeat expiry. Keep the heartbeat window short enough that the lie is brief. **Kafka consumer lag during a spike** A viral moment spikes message volume and consumer lag grows: delivery is delayed but nothing is lost, because Cassandra already has every message. Add consumers horizontally (partitions sized ahead of time), and clients still get everything on their next reconnect scan. Monitor consumer lag as a key SLO. **Duplicate delivery from at-least-once routing** Kafka may redeliver and WebSocket pushes may be retried. The client dedupes on `message_id` before rendering, so at-least-once plumbing yields exactly-once *user experience*. **Group delivery fan-out job fails partway** The message is already durably in the shared conversation log, so nothing is lost regardless. A background delivery job dies after pushing to 300 of 1,000 members; it's idempotent and resumes from its committed offset, and members who already got the push dedupe on `message_id`. The 700 not yet pushed either receive the resumed push or pick the message up on their next reconnect scan. Nobody misses it, nobody sees it twice. **Message deleted after delivery** Sender deletes a message already on recipients' devices. Mark deleted in Cassandra and push a tombstone event to online recipients; offline recipients get the tombstone on reconnect. Clients that already rendered it remove it. Eventual consistency is accepted here. --- ## Code ### Message Send: Persist, Route, and ACK **Java** ```java public class ChatServer { private final CassandraSession cassandra; private final KafkaProducer kafka; private final ConnectionRegistry connections; // sockets held on THIS server private final PresenceService presence; // user:{id} → server_id (Redis) private final IdempotencyCache idCache; // temp_id → message_id (Redis, TTL) private final Router router; // directed forward to a server // Runs on the SENDER's chat server: persist + hand off for routing. public void onSend(Session session, InboundMessage in) { // 1. Dedup on the client's temp_id, NOT on message_id, and no LWT. UUID messageId = idCache.resolveOrCreate(in.tempId(), UUIDs::timeBased); if (idCache.wasExisting()) { // a retry we already handled session.send(new Ack(in.tempId(), messageId, "sent")); return; } // 2. Persist: a plain write. Re-inserting the same (conv, msg) is a // harmless overwrite, so we never pay for IF NOT EXISTS (Paxos). cassandra.execute(insert("messages") .value("conversation_id", in.conversationId()) .value("message_id", messageId) .value("sender_id", session.userId()) .value("content", in.content()) .value("status", "sent")); // 3. Hand off to the router. Kafka decouples ingest from delivery. kafka.send(new ProducerRecord<>("messages", in.conversationId().toString(), new MessageEvent(messageId, in.conversationId(), session.userId(), in.content()))); // 4. ACK the sender immediately → single tick. session.send(new Ack(in.tempId(), messageId, "sent")); } // ROUTER (separate consumer group): for each recipient, look up the ONE // server holding their socket and forward only there, never broadcast to // all 1,000 servers. public void onKafkaMessage(MessageEvent event) { for (UUID recipient : membersOf(event.conversationId())) { String serverId = presence.serverFor(recipient); // null = offline if (serverId != null) { router.forward(serverId, recipient, event); // directed } // offline recipients get it via the reconnect range-scan } } // On the OWNING server: deliver to the local socket. public void onDirectedDelivery(UUID recipient, MessageEvent event) { Session s = connections.get(recipient); if (s != null) s.send(event.toOutbound()); } } ``` **Python** ```python # Runs on the SENDER's chat server: persist + hand off for routing. def on_send(session, inbound, cassandra, kafka, id_cache): # 1. Dedup on the client's temp_id, NOT on message_id, and no LWT. message_id, existing = id_cache.resolve_or_create(inbound.temp_id, uuid6) if existing: # a retry we already handled session.send({"type": "ack", "temp_id": inbound.temp_id, "message_id": str(message_id), "status": "sent"}) return # 2. Persist: a plain write. Re-inserting the same (conv, msg) is a # harmless overwrite, so we never pay for IF NOT EXISTS (Paxos). cassandra.execute( """INSERT INTO messages (conversation_id, message_id, sender_id, content, status) VALUES (%s, %s, %s, %s, 'sent')""", (inbound.conversation_id, message_id, session.user_id, inbound.content), ) # 3. Hand off to the router. Kafka decouples ingest from delivery. kafka.send("messages", key=str(inbound.conversation_id), value={ "message_id": str(message_id), "conversation_id": str(inbound.conversation_id), "sender_id": str(session.user_id), "content": inbound.content, }) # 4. ACK the sender immediately → single tick. session.send({"type": "ack", "temp_id": inbound.temp_id, "message_id": str(message_id), "status": "sent"}) # ROUTER (separate consumer group): forward only to the ONE server holding # each recipient's socket, never broadcast to all 1,000 servers. def route(event, presence, router, members_of): for recipient in members_of(event["conversation_id"]): server_id = presence.server_for(recipient) # None = offline if server_id is not None: router.forward(server_id, recipient, event) # directed # offline recipients get it via the reconnect range-scan # On the OWNING server: deliver to the local socket. def on_directed_delivery(recipient, event, connections): sock = connections.get(recipient) if sock is not None: sock.send(to_outbound(event)) ``` **C++** ```cpp // Runs on the SENDER's chat server: persist + hand off for routing. void ChatServer::onSend(Session& session, const InboundMessage& in) { // 1. Dedup on the client's tempId, NOT on messageId, and no LWT. auto [messageId, existing] = idCache_.resolveOrCreate(in.tempId, TimeUuid::now); if (existing) { // a retry we already handled session.send(Ack{in.tempId, messageId, "sent"}); return; } // 2. Persist: a plain write. Re-inserting the same (conv, msg) is a // harmless overwrite, so we never pay for IF NOT EXISTS (Paxos). cassandra_.execute( "INSERT INTO messages (conversation_id, message_id, sender_id, " "content, status) VALUES (?, ?, ?, ?, 'sent')", in.conversationId, messageId, session.userId(), in.content); // 3. Hand off to the router. Kafka decouples ingest from delivery. kafka_.send("messages", in.conversationId.str(), MessageEvent{messageId, in.conversationId, session.userId(), in.content}); // 4. ACK the sender immediately → single tick. session.send(Ack{in.tempId, messageId, "sent"}); } // ROUTER (separate consumer group): forward only to the ONE server holding // each recipient's socket, never broadcast to all 1,000 servers. void ChatServer::onKafkaMessage(const MessageEvent& event) { for (const auto& recipient : membersOf(event.conversationId)) { auto serverId = presence_.serverFor(recipient); // empty = offline if (!serverId.empty()) { router_.forward(serverId, recipient, event); // directed } // offline recipients get it via the reconnect range-scan } } // On the OWNING server: deliver to the local socket. void ChatServer::onDirectedDelivery(const Uuid& recipient, const MessageEvent& event) { Session* s = connections_.get(recipient); if (s != nullptr) s->send(event.toOutbound()); } ``` --- ## Trade-off Summary | Decision | Chosen | Alternative | Why | |---|---|---|---| | Connection protocol | WebSocket | Long polling / SSE | Full-duplex, low-overhead, true real-time; SSE is one-directional | | Message store | Cassandra | PostgreSQL | 1.16M writes/sec and time-series range scans fit Cassandra's model | | Routing | Kafka to decouple + presence-directed forward | Broadcast to all chat servers | Kafka handles backpressure/replay; presence forwards to the one server holding the socket instead of every server scanning membership | | Sender ACK | Decoupled from delivery | ACK only after recipient gets it | Sender gets instant durable confirmation even when recipient is offline | | Group storage vs delivery | One shared conversation log; async per-member *delivery* fan-out | Per-member storage inboxes | Storing 1,000 copies is wasteful; the cap keeps *delivery* fan-out bounded | | Retry dedup | `temp_id → message_id` cache | Cassandra `IF NOT EXISTS` (LWT) | LWT is Paxos (unaffordable at 1.16M writes/sec) and wouldn't catch a retry's fresh ID | | Ordering | TIMEUUID clustering (single stamping point) | Wall-clock timestamp | Consistent total order + uniqueness; skew handled by stamping at one owner, not by TIMEUUID itself | | Duplicate handling | Idempotent on message_id | Trust the queue | At-least-once plumbing → exactly-once user experience | --- ## Follow-up Questions **Basic** **Q: Why WebSocket instead of HTTP polling?** Chat is bidirectional and latency-sensitive at massive connection counts. Long polling re-establishes a request per message, huge overhead and not truly real-time. SSE is push-only, so the client can't send over the same channel. WebSocket is one handshake then cheap full-duplex frames, which is exactly the shape of a conversation. **Q: Why does the sender get a single tick before the recipient receives anything?** Because the ACK to the sender is decoupled from delivery. The single tick means "the server durably has your message": it fires as soon as Cassandra write plus Kafka publish succeed. Delivery to the recipient's device (double tick) and read (blue tick) are later, separate acknowledgements that depend on the recipient being reachable. **Q: What happens when the recipient is offline?** Nothing is lost. The message is already durable in Cassandra with status `sent`. When the recipient reconnects, they send their last-seen ID and the server range-scans for everything newer and delivers it in order. Kafka is only the online fast path; Cassandra is the source of truth. **Q: How is group messaging different from 1:1?** A group send fans out to every member. WhatsApp uses async fan-out on write: the sender is ACKed after one durable write, and a background job delivers to the other members' inboxes. It works because group size is capped, so worst-case fan-out is a bounded constant, unlike an unbounded social feed. ### **Senior follow-ups ↓** **Q: A user sends a message, sees a single tick, then their app crashes and retries. Walk me through avoiding a duplicate.** The client retries with the same `temp_id` it generated originally. The server looks up `temp_id` in a short-TTL Redis map; finding an existing entry, it returns the *same* `message_id` instead of minting a new one and writing a second row. (Don't reach for Cassandra `IF NOT EXISTS` here: it's a Paxos lightweight transaction you can't afford at 1.16M writes/sec, and it wouldn't help anyway since the retry's fresh `message_id` wouldn't collide.) Downstream, recipients also dedupe on `message_id` before rendering. At-least-once plumbing, exactly-once user experience. **Q: How do you keep 100M WebSocket connections healthy, and what happens when a chat server dies?** Application-level heartbeats (~30s ping/pong) detect dead sockets and drive presence. When a server dies, its ~100k clients reconnect (with randomized exponential backoff so they don't stampede the survivors) and land on other servers via the load balancer. The presence entry (`user:{id} → server_id`) is updated on the new connect, and any messages missed during the gap are delivered by the reconnect range-scan. **Q: The interviewer changes group size to unbounded broadcast channels with millions of subscribers. What breaks and how do you fix it?** Fan-out on write breaks: one message would become millions of inbox writes, the celebrity problem. Flip to fan-out on read: store the message once in a shared channel log, and have subscribers pull-and-merge on read, fronted by a heavy read cache. Per-subscriber delivery receipts become impractical at that scale, so you drop them for broadcast (which is why channels don't show per-recipient ticks). This is exactly the hybrid trade-off from the news feed problem. **Q: How do you guarantee ordering when both participants send at the same instant?** Order by a time-ordered ID (TIMEUUID/UUIDv7) that's globally unique, not by wall clock. Within a conversation, both messages land in the same Cassandra partition and the clustering column sorts them deterministically. Two truly simultaneous messages get a total order from the ID's tiebreak bits, and every client that range-scans the partition sees the identical order. --- ## Minute-by-Minute Interview Playbook **0 to 5 min: Requirements** Ask the questions above. Scope 1:1 first, then groups. Nail down receipts, scale, durability, and out-of-scope (no E2E encryption, no calls). Write FR/NFR on the board. **5 to 8 min: Capacity estimates** Do the math out loud. The 100M concurrent connections number justifies the entire connection-model discussion; the 1.16M writes/sec justifies Cassandra over Postgres. **8 to 18 min: HLD** Draw the WebSocket connection model first, then the path Client → Chat Server → Kafka → Chat Server → Client, with Cassandra hanging off the write. Explain why WebSocket over polling. Get the happy path right before edge cases. **18 to 38 min: Deep dives** Lead with the delivery-receipt flow: draw all three tick states step by step. This is the highest-value section. Then offline queueing ("what if the recipient is offline?"), then group fan-out with a concrete threshold, then ordering (TIMEUUID, not wall clock). **38 to 45 min: Failure scenarios and wrap-up** Cover the server-crash retry (idempotent on message_id) and Kafka lag (nothing lost, Cassandra is truth). Summarize in one sentence: "WebSocket connections, Kafka routing, Cassandra as the durable source of truth, with the sender ACK decoupled from delivery so nothing is lost and the sender always gets instant feedback." **Green flags** - Reaches for WebSocket immediately and can say why, not just that - Walks the delivery-receipt flow with all three tick states - Decouples the sender ACK from recipient delivery deliberately - Treats group fan-out as a distinct, bounded problem - Handles the crash/retry duplicate with idempotency keys **Red flags** - Uses HTTP polling for real-time delivery - Never addresses offline users - Treats group messages identically to 1:1 - Uses a relational DB as the primary message store at 100B/day - Orders by wall-clock timestamp and ignores clock skew --- ## Further Reading - [The WhatsApp Architecture Facebook Bought for $19 Billion](https://highscalability.com/the-whatsapp-architecture-facebook-bought-for-19-billion/): How a tiny team ran a service for hundreds of millions of users: connection handling, Erlang, and the design decisions behind the scale. - [1 Million Connections Per Server](https://blog.whatsapp.com/1-million-is-so-2011): WhatsApp engineering on pushing a single server past a million concurrent connections: the connection-density problem at the heart of this design. - [Uber's Real-Time Push Platform](https://www.uber.com/blog/real-time-push-platform/): A production WebSocket delivery system: connection management, message acknowledgement, and reliability patterns that map directly onto chat. - [Related: Design Ticketmaster](https://intervueclub.com/problems/realtime/design-ticketmaster): Also demands strict ordering and consistency, but adds hard correctness constraints (no double-booking) on top of real-time delivery. - [Related: Design a News Feed](https://intervueclub.com/problems/databases/design-a-news-feed): The fan-out decision transfers directly, but follows are unbounded there, which forces the hybrid push/pull the capped group size lets WhatsApp avoid.