Webhooks System — Detailed

flowchart TB
  subgraph Producers
    APP[App services]
    OUTBOX[[Outbox events]]
  end

  subgraph Pipeline
    BUS[[Event bus]]
    SUB([Subscription registry<br/>customer endpoints])
    FILTER[Event filter / type]
    TRANS[Payload templater + transform]
  end

  subgraph Delivery
    POOL([Delivery worker pool])
    SIGN[HMAC signing]
    RETRY[Exponential backoff + jitter]
    BUDGET[Per-endpoint retry budget]
    CB[Circuit breaker per endpoint]
    DLQ[[(DLQ)]]
    REPLAY[Manual replay UI]
  end

  subgraph Customer
    EP([Customer endpoint])
    IDEM([Customer idempotency])
  end

  subgraph Ops
    OBS[Metrics: delivered, retry, latency]
    AUDIT[Audit log]
    SECRETS([Per-customer signing secret])
  end

  Producers --> BUS --> FILTER --> TRANS --> POOL --> EP
  SUB --- FILTER
  POOL --> RETRY --> POOL
  POOL --> CB
  POOL --> DLQ --> REPLAY
  POOL --> SIGN
  SECRETS --- SIGN
  Ops --- POOL

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class SUB,EP,IDEM,SECRETS client;
    class APP,FILTER,TRANS,SIGN,RETRY,BUDGET,CB,REPLAY service;
    class DLQ datastore;
    class OUTBOX,BUS queue;
    class POOL compute;
    class OBS,AUDIT obs;

Delivery semantics

• At-least-once with HMAC-signed bodies and Idempotency-Key header.
• Customer side: dedupe by (event_id, type).

Retry policy

• 5xx / timeout → backoff sequence (1s, 2s, 4s, … up to days).
• 4xx (auth, validation) → don't retry; alert customer.
• Per-endpoint circuit breaker: open after high failure rate to protect the system.

Glossary & fundamentals

Concepts referenced in this design. Each row links to its canonical page; the tag column shows whether it is a high-level (HLD) or low-level (LLD) concept.

Tag	Concept	What it is	Page
HLD	Pub/Sub & message brokers	topics, consumer groups, delivery semantics	pub-sub-pattern
HLD	Idempotency & retries	safe re-execution, backoff + jitter	idempotency-retries
HLD	Resilience patterns	timeout, retry, breaker, bulkhead, backpressure	resilience-patterns
HLD	Observability	metrics, logs, traces, SLOs	observability
LLD	REST API design	verbs, statuses, pagination, errors	rest-api-design