Skip to content

News Feed — Detailed (Hybrid Push/Pull)#

flowchart TB
  subgraph Write[Write Path]
    POST[Compose / Post API]
    MQ[[Kafka post events]]
    FAN[[Fan-out workers]]
    FG[Follower graph<br/>cached]
    HOT([Hot-user detector<br/>celebrity threshold])
  end

  subgraph Stores
    POSTS[(Posts DB<br/>Cassandra: post_id PK)]
    GRAPH[(Social Graph<br/>FlockDB / TAO)]
    TL[(Timeline cache<br/>Redis ZSET per user)]
    MEDIA[(Object store + CDN)]
  end

  subgraph Read[Read Path]
    FEED[Feed API]
    MERGE([Merger<br/>cached TL + pull from celebs])
    RANK[Ranking model<br/>ML scoring]
    HYD([Hydrator<br/>fetch post + author + media])
    PAG[Cursor pagination]
  end

  subgraph ML[Ranking & Signals]
    SIG[Signals<br/>recency, affinity, engagement]
    MOD([Ranker - GBDT / DNN])
    EXP[Experiments / A-B]
  end

  subgraph Ops
    DEDUP([Dedup<br/>seen posts per user])
    SHIELD[Spam / safety filter]
    NOTIF([Notification trigger])
    OBS[Metrics / SLO]
  end

  USER[Reader] --> FEED --> MERGE
  TL --> MERGE
  POSTS --> HYD
  GRAPH --> MERGE
  MERGE --> RANK --> HYD --> PAG --> USER
  POSTER[Author] --> POST --> MQ --> FAN
  FG -.->|fan-out push| TL
  HOT -.->|skip push| MQ
  HOT -.->|read-time pull| MERGE
  POSTS --- POST
  MEDIA --- POST
  SIG --> MOD --> RANK
  EXP --> RANK
  DEDUP --> PAG
  SHIELD --> RANK
  NOTIF --- MQ
  OBS --- FEED

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class HOT,DEDUP,USER client;
    class POST,FG,FEED,RANK,PAG,SIG,EXP,SHIELD,POSTER service;
    class POSTS datastore;
    class TL cache;
    class MQ,FAN queue;
    class MERGE,HYD,MOD,NOTIF compute;
    class MEDIA storage;
    class OBS obs;

Strategies#

  • Push (fan-out-on-write): when author posts, write to each follower's timeline list. Cheap reads, expensive for celebs.
  • Pull (fan-out-on-read): at read time, query latest posts from each followee. Cheap writes, expensive for big followee sets.
  • Hybrid (most prod systems): push for normal users; for celeb authors, skip push and pull at read.

Timeline cache layout#

  • Redis ZSET per user: score=created_ts, member=post_id, capped at e.g. 1k entries (LTRIM).
  • Cold readers: rebuild from Posts DB on demand.

Ranking#

  • Features: post age, author affinity, prior engagement, content quality, freshness.
  • Online ranker (low-latency) on top of cached candidates.
  • Personalization via embeddings + ANN candidate gen (for explore).

Edge cases#

  • Celebrity 100M+ followers — fan-out IO storm. Treat as pull-only.
  • Multi-DC: writes regional, async replicate timelines + posts.
  • Deletes / edits: tombstones + read-time filter; eventually rewritten.

Glossary & fundamentals#

Concepts referenced in this design. Each row links to its canonical page; the tag column shows whether it is a high-level (HLD) or low-level (LLD) concept.

Tag Concept What it is Page
HLD CDN edge caching for static assets cdn
HLD Pub/Sub & message brokers topics, consumer groups, delivery semantics pub-sub-pattern
HLD Leader/follower replication sync/semi-sync/async replication, failover replication-leader-follower
HLD Observability metrics, logs, traces, SLOs observability
HLD Search internals inverted index, BM25, embeddings, ANN search-internals
LLD REST API design verbs, statuses, pagination, errors rest-api-design