Designing a Chat System

"Design a chat system" is one of the most common system design interview prompts, and for good reason — it touches almost every hard problem in distributed systems at once: real-time delivery, fan-out to many recipients, ordering guarantees, durable storage, and graceful handling of clients that are offline half the time. None of the individual pieces are exotic, but the way they compose is where the interesting design decisions live.

This isn't about building the next messaging app — it's about the handful of concepts that show up whenever two or more people need to exchange messages in something close to real time.

Delivery: why persistent connections, not polling

The naive approach — have the client ask "any new messages?" every few seconds — works, but it's a bad trade of latency against load. Poll too slowly and messages feel delayed; poll too fast and you're hammering the server with mostly-empty responses from millions of idle clients. Chat systems instead keep a persistent, bidirectional connection open between client and server — a WebSocket, or an equivalent like long-lived HTTP/2 streaming or server-sent events paired with a write channel. The server can then push a message the instant it arrives, with no polling interval standing between send and receive.

For a 1:1 conversation this is almost the whole story: sender writes a message, the server looks up whether the recipient has an open connection, and if so, pushes it directly down that socket. Group chat is where it gets more interesting.

The fan-out problem in group chats

A message sent to a group of 50 people needs to reach up to 50 open connections — and because a chat service runs many server instances behind a load balancer, those 50 connections are almost certainly spread across several different machines. The instance that received the incoming message has no direct way to reach a socket held open on a different instance.

The standard fix is a layer of indirection: rather than the receiving instance trying to locate every recipient's socket itself, it publishes the message to a shared pub/sub layer keyed by conversation (or by each recipient's user ID). Every server instance subscribes to the channels for the users it currently has connected, so when a message is published, each instance holding a relevant connection receives it and pushes it down that specific socket. This decouples where a message originates from where a recipient's connection happens to live, and it lets the fleet scale horizontally — adding more server instances just means more subscribers, not a rearchitecture of how messages get routed.

Ordering and idempotent delivery

Two problems tend to get tangled together here, and it's worth separating them. First, ordering: participants in a conversation expect to see messages in a consistent sequence, even if the underlying delivery infrastructure doesn't guarantee that a message sent first arrives first. The usual fix is to assign each message a monotonically increasing sequence number scoped to its conversation — the client renders by sequence number, not by arrival time, so a message that arrives out of order still gets placed correctly.

Second, duplicate delivery. If a client sends a message and the connection drops before the server's acknowledgment makes it back, the client can't tell whether the message was actually received. Its only safe move is to retry — but a naive retry risks the message being stored twice. The fix is exactly the idempotency pattern used across API design generally: the client attaches a unique ID it generated itself (a client_msg_id) to the message, and the server treats an insert with an already-seen ID as a no-op rather than a new message. The retry becomes safe to send as many times as needed, because only the first successful write actually counts.

{
  "message_id": "01HZY3...",      // ULID, sortable by time
  "conversation_id": "conv_88f2",
  "sender_id": "user_42",
  "client_msg_id": "c-9f3a1e",    // set by the client, used for dedup
  "body": "see you at 6?",
  "created_at": 1737033600123,
  "seq": 481                       // monotonic per-conversation sequence
}

Persisting history: an append-only log per conversation

Once a message is accepted, it needs to be stored durably — delivery over a socket is best-effort, and the source of truth for "what was actually said" has to be a database write that happened before (or atomically with) the push. Because messages within a conversation are only ever appended and essentially never updated in place, an append-only log per conversation is a natural fit — physically or logically partitioned by conversation ID, ordered by the sequence number described above.

That storage shape also drives how history gets read back. Returning an entire conversation's history on every load doesn't scale — a conversation with years of messages would dwarf anything the client actually needs to render. Instead, the client fetches only the most recent page (say, the last 50 messages), and paginates backward — requesting the next older page keyed off the earliest sequence number it already has — as the user scrolls up. This keeps both the initial load fast and the client's memory footprint bounded, regardless of how long the conversation has existed.

Presence and read receipts: a lighter, separate concern

It's tempting to route presence (online/offline), typing indicators, and read receipts through the same durable path as messages, but they have fundamentally different requirements. Losing a "typing…" event or an occasional presence update is harmless — it'll be superseded by the next one within seconds. Losing a message is not. Treating them the same way means paying message-durability costs for data that doesn't need it.

These signals are typically handled by a much simpler, ephemeral pub/sub channel: a user's client publishes "online", "typing in conversation X", or "last read message ID Y", and interested peers subscribe and receive the latest value. There's no append-only log, no retry-with-idempotency dance, and no pagination — just a best-effort broadcast of current state that's allowed to be slightly stale or occasionally dropped.

Delivering to someone who isn't connected

A recipient is frequently not connected at all — phone locked, app closed, laptop asleep. The message still has to reach them eventually, just through a different route. Two mechanisms cover this, usually together: a push notification sent through the platform's notification service (which doesn't require an open connection at all), and the fact that the message was already durably written to that conversation's log — so the moment the recipient's client reconnects, its normal "fetch recent messages" request picks it up as part of ordinary history, with no special "offline queue" logic required.

That's the quiet payoff of treating message storage as the source of truth rather than the live socket connection as the source of truth: online delivery becomes an optimization for latency, not a requirement for correctness. A client that was offline for a week and one that never disconnected end up in exactly the same state once it catches up.