Pub/Sub Pattern — Detailed

flowchart TB
  subgraph Producers
    P1[Service A]
    P2[Service B]
    P3([CDC / Debezium])
  end

  subgraph Broker[Broker Cluster]
    direction TB
    subgraph Topic_orders[Topic: orders]
      P0[Partition 0]
      P1p[Partition 1]
      P2p[Partition 2]
      P3p[Partition 3]
    end
    META[[Controller / Metadata<br/>Kafka KRaft, Zookeeper]]
    REPL[Replication ISR<br/>RF=3]
  end

  subgraph Storage
    LOG[Append-only log /<br/>segment files]
    IDX[Index per partition]
    TIER[Tiered storage S3<br/>cold segments]
  end

  subgraph Consumers
    subgraph CG_billing[Consumer Group: billing]
      CB1[c1 partitions 0,1]
      CB2[c2 partitions 2,3]
    end
    subgraph CG_analytics[Consumer Group: analytics]
      CA1[c1 partitions 0-3]
    end
    DLQ[[(Dead Letter Topic)]]
  end

  subgraph Semantics
    AL[At-Least-Once]
    AM[At-Most-Once]
    EO[Exactly-Once<br/>idempotent + tx]
    ORD[Per-partition Order]
    RT[Retention by time/size]
  end

  P1 --> Topic_orders
  P2 --> Topic_orders
  P3 --> Topic_orders
  Topic_orders --> LOG
  LOG --> IDX
  LOG -. age out .-> TIER
  META -. assign leaders .-> Topic_orders
  REPL --- Topic_orders
  Topic_orders --> CG_billing
  Topic_orders --> CG_analytics
  CG_billing -. poison msg .-> DLQ

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class P1,P2,P0,P1p,P2p,P3p,REPL,LOG,IDX,CB1,CB2,CA1,AL,AM,EO,ORD,RT service;
    class DLQ datastore;
    class META queue;
    class P3 compute;
    class TIER storage;

Routing models

• Topic (Kafka, NATS, Pub/Sub): publisher writes once, multiple consumer groups read independently.
• Queue / Work (SQS, RabbitMQ direct): each message goes to exactly one consumer of a group.
• Topic + filters (NATS subjects, SNS attributes): selective subscribe.

Delivery semantics

• At-most-once: fire and forget.
• At-least-once: ack after processing; duplicates possible — make consumers idempotent.
• Exactly-once: transactional producer + idempotent consumer + read-process-write tx (Kafka EOS).

Ordering

• Per-partition / per-key only. To preserve order, use stable partition key (e.g., user_id).

Backpressure & DLQ

• Pull (Kafka) vs push (RabbitMQ); pull naturally backpressures.
• Retries with exponential backoff; after N failures → DLQ.

Glossary & fundamentals

Concepts referenced in this design. Each row links to its canonical page; the tag column shows whether it is a high-level (HLD) or low-level (LLD) concept.

Tag	Concept	What it is	Page
HLD	Pub/Sub & message brokers	topics, consumer groups, delivery semantics	pub-sub-pattern
HLD	Leader/follower replication	sync/semi-sync/async replication, failover	replication-leader-follower
HLD	Idempotency & retries	safe re-execution, backoff + jitter	idempotency-retries
HLD	Resilience patterns	timeout, retry, breaker, bulkhead, backpressure	resilience-patterns
HLD	Change Data Capture	WAL/binlog tailing, outbox publishing	change-data-capture