Skip to content

Message Queue — Detailed (Kafka-flavoured)#

flowchart TB
  subgraph Producers
    P1[App A]
    P2[App B]
    P3[CDC tail]
  end

  subgraph Brokers[Broker Cluster]
    direction TB
    META[Controller / KRaft<br/>metadata + leader election]
    B1[[Broker 1]]
    B2[[Broker 2]]
    B3[[Broker N]]
  end

  subgraph Topic
    P0[Partition 0]
    P1p[Partition 1]
    P2p[Partition 2]
    ISR[In-sync Replicas]
    LEAD[Leader / Followers]
  end

  subgraph Storage[Storage Layer]
    SEG[Segments + index]
    PAGE[Page cache]
    TIER[Tiered storage to S3]
    COMPACT[Log compaction by key]
    RETN[Retention by time/size]
  end

  subgraph Consumers
    G1[Group billing]
    G2[Group analytics]
    OFFSETS[(Committed offsets)]
    DLQ[[Dead-letter topic]]
  end

  subgraph Semantics
    AL[at-least-once]
    EOS[Exactly-once via tx]
    ORD[Per-partition order]
    KEY[Partition by key]
  end

  Producers --> Brokers
  Brokers --> Topic
  Topic --> Storage
  Topic --> Consumers
  Consumers --> OFFSETS
  Consumers --> DLQ
  Semantics --- Topic

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class P1,P2,P3,META,P0,P1p,P2p,ISR,LEAD,SEG,COMPACT,RETN,G1,G2,AL,EOS,ORD,KEY service;
    class OFFSETS datastore;
    class PAGE cache;
    class B1,B2,B3,DLQ queue;
    class TIER storage;

Replication & ISR#

  • Each partition has one leader, N followers.
  • Producers write to leader; followers fetch.
  • ISR = followers caught up within replica.lag.time.max.ms.
  • Producer acks=all waits for ISR commit; acks=1 waits for leader only.

Consumer groups#

  • Each partition is consumed by exactly one consumer in a group.
  • Rebalance on member add/remove (causes brief pause).
  • Offsets stored in __consumer_offsets; consumers commit periodically.

Exactly-once semantics#

  • Idempotent producer (sequence number per partition).
  • Transactional producer for write-and-commit-offset atomically.
  • Consumer reads read_committed isolation.

Tiered storage#

  • Hot segments local; older segments moved to S3.
  • Brokers read transparently from S3 when needed.

Glossary & fundamentals#

Concepts referenced in this design. Each row links to its canonical page; the tag column shows whether it is a high-level (HLD) or low-level (LLD) concept.

Tag Concept What it is Page
HLD Pub/Sub & message brokers topics, consumer groups, delivery semantics pub-sub-pattern
HLD Raft / Paxos consensus replicated state machine via majority quorum consensus-raft-paxos
HLD Leader/follower replication sync/semi-sync/async replication, failover replication-leader-follower
HLD LSM vs B-Tree engines WAL, memtable, SSTables, compaction storage-engines-lsm-btree
HLD Change Data Capture WAL/binlog tailing, outbox publishing change-data-capture
LLD Testing strategy pyramid, doubles, TDD, contracts testing-strategy
LLD Concurrency primitives mutex, semaphore, RW lock, atomic, CAS concurrency-primitives