Distributed Logging (ELK / EFK) — Simple

Problem statement (interviewer prompt)

Design a centralised logging platform (ELK / EFK): collect structured logs from thousands of services, parse + enrich + redact PII, index for search (last 30 days hot, 1 year cold), and serve dashboards + alerts at 1M+ events/sec.

flowchart LR
  APP[Apps]
  AG[Agents<br/>Fluent Bit / Filebeat]
  BUS[[Kafka buffer]]
  PROC([Ingest / parse])
  ES[(Elasticsearch / OpenSearch)]
  K[Kibana / Grafana]
  APP --> AG --> BUS --> PROC --> ES --> K

    classDef client fill:#dbeafe,stroke:#1e40af,stroke-width:1px,color:#0f172a;
    classDef edge fill:#cffafe,stroke:#0e7490,stroke-width:1px,color:#0f172a;
    classDef service fill:#fef3c7,stroke:#92400e,stroke-width:1px,color:#0f172a;
    classDef datastore fill:#fee2e2,stroke:#991b1b,stroke-width:1px,color:#0f172a;
    classDef cache fill:#fed7aa,stroke:#9a3412,stroke-width:1px,color:#0f172a;
    classDef queue fill:#ede9fe,stroke:#5b21b6,stroke-width:1px,color:#0f172a;
    classDef compute fill:#d1fae5,stroke:#065f46,stroke-width:1px,color:#0f172a;
    classDef storage fill:#e5e7eb,stroke:#374151,stroke-width:1px,color:#0f172a;
    classDef external fill:#fce7f3,stroke:#9d174d,stroke-width:1px,color:#0f172a;
    classDef obs fill:#f3e8ff,stroke:#6b21a8,stroke-width:1px,color:#0f172a;
    class APP,AG service;
    class ES datastore;
    class BUS queue;
    class PROC compute;
    class K obs;