Designing a Low‑Latency Multi‑Region Real‑Time Chat System

#Problem statement and requirements

We need to build a real‑time chat platform that powers a consumer‑facing mobile/web app. The service must feel instantaneous, survive network partitions, and keep costs predictable as the user base grows. Below are the functional and non‑functional requirements, together with realistic scale assumptions that guide design decisions.

#Functional requirements

#Non‑functional requirements

#Assumptions (scale numbers)

• Active users: 5 M concurrent, 50 M total monthly.
• Average messages per active user: 20 msg / day → ≈ 100 M messages / day.
• Peak concurrency: 2 M users online during global events.
• Geography: Three primary regions – US‑East, EU‑West, AP‑SouthEast – each with its own data center.

These numbers are deliberately generous to stress‑test the design while staying realistic for a fast‑growing consumer app.

#High-level architecture

graph TD
    subgraph Edge[Edge Layer]
        LB[Global Load Balancer] -->|TLS| API[API Gateway]
        WS[WebSocket Frontend] -->|WS| API
    end

    subgraph Region[Region (e.g., US‑East)]
        API -->|REST| Auth[Auth Service]
        API -->|WS| ConnMgr[Connection Manager]
        ConnMgr -->|Pub/Sub| MsgBroker[Kafka Cluster]
        MsgBroker -->|Topic: chat.{conv_id}| MsgProc[Message Processor]
        MsgProc -->|Write| ChatStore[Sharded Cassandra]
        MsgProc -->|Write| PresenceStore[Redis Cluster]
        MsgProc -->|Emit| Notif[Push Notification Service]
        MediaSrv[Media Service] -->|S3| ObjectStore[Object Storage]
    end

    subgraph Global[Cross‑Region Sync]
        MsgBroker -- Replicate --> MsgBroker_EU
        MsgBroker -- Replicate --> MsgBroker_AP
        PresenceStore -- Sync --> PresenceStore_EU
        PresenceStore -- Sync --> PresenceStore_AP
    end

    subgraph Client[Client]
        MobileApp -->|WS| WS
        WebApp -->|WS| WS
    end

Key flow

1. Authentication – JWT issued by Auth Service, verified at the edge.
2. WebSocket connection – Handled by Connection Manager; each client gets a logical session ID.
3. Message publish – Client sends a JSON payload over WS; Connection Manager forwards to Kafka topic chat.{conversation_id}.
4. Processing – Message Processor consumes, assigns a Lamport timestamp, writes to Cassandra, updates read‑receipt state in Redis, and pushes to the recipient’s active connections.
5. Cross‑region – Kafka’s MirrorMaker replicates topics to other regions; read‑receipt updates are eventually synced via Redis CRDTs.
6. Media – Attachments are uploaded via a pre‑signed URL to Object Storage; message payload contains a reference URL.

#Data model

#1. Message schema (Cassandra table)

CREATE TABLE chat.messages (
    conversation_id uuid,
    partition_key timeuuid,          // Lamport timestamp + UUID for ordering
    message_id uuid,
    sender_id uuid,
    payload text,                    // JSON with text or media reference
    created_at timestamp,
    edited_at timestamp,
    is_deleted boolean,
    PRIMARY KEY (conversation_id, partition_key)
) WITH CLUSTERING ORDER BY (partition_key ASC);

• partition_key combines a Lamport timestamp (ensuring total order per conversation) with a UUID to avoid collisions.
• Time‑series partitioning: each conversation is a partition; Cassandra’s wide‑row model handles millions of rows efficiently.

#2. Presence & read receipts (Redis)

Redis stores the latest read pointer per user per conversation. Updates are idempotent and replicated across regions using Redis‑CRDT for eventual consistency.

#3. Media metadata (SQL)

A lightweight relational table (PostgreSQL) tracks media URLs, size, MIME type, and expiration for cleanup jobs.

CREATE TABLE media (
    media_id UUID PRIMARY KEY,
    conversation_id UUID,
    uploader_id UUID,
    object_key TEXT,
    mime_type TEXT,
    size_bytes BIGINT,
    created_at TIMESTAMP WITH TIME ZONE
);

#Key components

#Scaling & bottlenecks

#1. WebSocket connection density

• Bottleneck: Each Connection Manager instance can handle ~30 k concurrent sockets (limited by file descriptors and event loop).
• Mitigation: Horizontal autoscaling behind a TCP load balancer; use sticky sessions based on user ID to keep a user’s socket on a single manager, reducing cross‑instance fan‑out.

#2. Kafka throughput

• Bottleneck: Topic partitions per conversation; too many partitions can overwhelm the controller.
• Mitigation: Partition by conversation hash (e.g., first 2 characters of UUID) to achieve ~10 k partitions per cluster, enough to parallelize consumption while keeping per‑partition ordering.

#3. Cassandra write amplification

• Bottleneck: High write rate can increase compaction latency.
• Mitigation:
  • Use TimeWindowCompactionStrategy tuned for time‑series data.
  • Set replication factor = 3 (one per region) with NetworkTopologyStrategy.
  • Enable batch writes only for atomicity within a single conversation (avoid large multi‑row batches).

#4. Cross‑region replication latency

• Bottleneck: MirrorMaker introduces ~30‑50 ms lag, which can affect read receipts visible to remote participants.
• Mitigation: Accept eventual consistency for read receipts; surface a “Seen by X (syncing…)” UI hint until the remote region acknowledges.

#5. Media upload bandwidth

• Bottleneck: Large video files can saturate edge bandwidth.
• Mitigation:
  • Enforce client‑side compression and size limits.
  • Use multipart upload with CDN edge nodes (e.g., CloudFront) to offload traffic from core region.

#6. Autoscaling thresholds

• Metric: messages_per_sec per region, ws_conn_count per manager, cpu_utilization.
• Policy: Scale out when any metric exceeds 70 % for 2 minutes; scale in when below 30 % for 5 minutes.

#Trade‑offs

Choosing the right balance depends on product priorities: if ultra‑low latency is non‑negotiable, we accept higher operational complexity; if cost predictability is paramount, we may relax ordering guarantees for low‑traffic groups.

#Interview tips

1. Start with requirements – Interviewers love a structured approach. List functional vs. non‑functional items, then add realistic scale numbers.
2. Explain ordering guarantees – Show you understand why per‑conversation ordering matters and how Kafka partitions enforce it.
3. Talk about CAP – Emphasize that chat is CP for message ordering but AP for read receipts, and justify the trade‑off.
4. Use diagrams – A quick mermaid sketch demonstrates clarity and helps the interviewer follow your mental model.
5. Address failure modes – Discuss how the system recovers from a Kafka broker loss, a Redis partition outage, or a network split.
6. Mention observability – Mention metrics (latency, queue depth, WS connections) and tracing (OpenTelemetry) as part of a production‑ready design.
7. Quantify – When you say “scale to 10 k msg/s”, back it up with a rough calculation (e.g., each Kafka broker can handle ~500 k msg/s, so a 5‑node cluster is safe).

#Lessons from production

• Heartbeat matters – Without a 5‑second WS ping, idle connections linger, exhausting file descriptors. Implement a server‑side timeout and client‑side reconnection logic.
• Back‑pressure is essential – During spikes, the Connection Manager should buffer at most 100 msg per socket; excess messages are dropped with a “retry later” flag to avoid OOM.
• Schema evolution – Use Cassandra’s flexible columns for adding new fields (e.g., reactions) without downtime. Keep a version field in the message payload for forward compatibility.
• Media cleanup – Schedule a nightly job that deletes objects older than 30 days and not referenced in the media table. This prevents orphaned blobs from inflating storage costs.
• Testing latency – Simulate cross‑region traffic with a tool like tcpcopy or chaos‑mesh to verify the 100 ms SLA under realistic network jitter.

#Key takeaways