What Kafka is
Apache Kafka is a distributed event streaming platform: a durable, append-only log that many programs can write to and read from at the same time. It was built at LinkedIn around 2010 for clickstream data and became an Apache project in 2011. Think of it as a post office that never throws letters away: senders drop messages into named boxes, and any number of readers pick them up when ready — minutes or months later.
Core concepts in one minute
An event is one immutable fact: a key, a value, and a timestamp.
A topic is a named stream of events, split into partitions:
ordered logs spread across machines, which is where the throughput
comes from. Order is guaranteed only inside a single partition, never
across the whole topic.
Brokers are the servers forming a cluster; modern clusters
coordinate themselves with the built-in KRaft protocol
(ZooKeeper was removed in Kafka 4.0, so new setups do not need it).
Producers append events, consumers read them at their own
pace and commit offsets — bookmarks of how far they got.
Consumers sharing a group id split a topic's partitions
between themselves, so adding workers scales reading linearly.
Where it fits, and where it does not
Kafka fits high-throughput event flows between decoupled services:
logs and metrics collection, event-driven microservices, and realtime
stream processing. Replay is the killer feature: a new consumer can
re-read history from the beginning without asking producers to resend
anything.
It does not fit request/reply calls (use HTTP or RPC there), tiny
loads a single queue could carry, or data that needs strict global
ordering across many writers. A one-broker Kafka for three messages
a day is operational theater — the setups below assume real traffic.
Scenarios and setups
Concrete setups, from simplest to fullest, live on the scenarios page:
- Centralized log aggregation — every service ships logs to Kafka instead of local files.
- Event-driven microservices — an order service emits events; stock and mail react independently.
- Realtime stream processing — counting and windowing over a live event flow.
- Multi-container Docker Compose setup — producer, consumer, and Kafka wired as containers talking through topics.