Skip to content

Message Queue — Notes#

Functional#

  • Persistent log of records by topic.
  • Partitioned for parallelism.
  • Multiple consumer groups read independently.
  • Ordering per partition; retention by time/size.

Non-functional#

  • 1M+ msg/s/broker possible (Kafka, with batching).
  • p99 produce < 10 ms with batched producer.
  • 99.99% durability with RF=3 + acks=all.

Capacity#

  • Disk = throughput × retention; e.g., 100 MB/s × 7 days × 3 replicas = 180 TB cluster.
  • Network often the bottleneck.

API#

Produce(topic, key?, value, headers)
Consume(topic, group, partition, offset)
CommitOffset(group, partition, offset)

Trade-offs#

  • Push (RabbitMQ) vs Pull (Kafka): pull self-regulates, push tightens latency.
  • EOS has cost (transactional overhead); use only where needed.
  • Tiered storage trades S3 latency for far longer retention.

Refs#

  • Kafka design doc (Confluent), "I Heart Logs" Jay Kreps, RabbitMQ docs, NATS JetStream, Pulsar architecture.