Where Kafka is used: scenarios and setups

Four setups

The same Kafka core — append-only topics, consumer groups, replay — serves very different architectures. Below are four setups in growing order of machinery, each with the smallest configuration that works. The last one is a complete multi-container project you can run.

Log aggregation

The original Kafka workload: every service ships its logs to a central place instead of keeping local files nobody reads. One topic per service (or per severity), tiny producers with fire-and-forget sends, and a handful of consumers fanning out to storage and search:

# every service appends to its own topic; retention does the rotation web-1 -> topic logs-web api-2 -> topic logs-api worker-3 -> topic logs-worker # consumers, each in its own group so each sees every event: cold storage <- all log topics search index <- all log topics # pager: only severity >= error alerts <- topic logs-critical

Create a topic with a week of retention for this shape of data:

Create the topic

kafka-topics.sh --create --topic logs-web --bootstrap-server kafka:9092

Event-driven microservices

Services stop calling each other and start announcing facts. An order service appends order.placed to one topic and moves on; stock reservation, e-mail, and analytics each consume that topic at their own pace. Adding a fourth reaction — say, fraud scoring — means deploying one more consumer group, with full history available for replay from day one. No changes to the order service, no downtime.

The contract that makes this work is the event schema: keep events small, versioned, and backward compatible, because old events outlive every consumer deployment.

Realtime stream processing

When the question is about the flow itself — how many checkouts per minute, which pages errors cluster on — a stream processor reads topics and writes derived topics back. A word-count over a click topic is the canonical five-minute example; production jobs add tumbling windows (per-minute aggregates) and joins against slowly changing tables. Kafka Streams and ksqlDB both run this pattern without a separate processing cluster for moderate loads.

Multi-container Docker Compose setup

The full picture: three containers on one Compose network — a shop API that takes HTTP orders, a stock worker, and Kafka itself — with every word between the services traveling through a topic. The API never calls the worker; the worker never polls the API.

Project layout

# shop/ — everything below lives in one directory compose.yaml producer/Dockerfile producer/requirements.txt producer/app.py consumer/Dockerfile consumer/requirements.txt consumer/worker.py

compose.yaml — one KRaft broker plus the two app services

# compose.yaml — single-broker KRaft setup, no ZooKeeper anywhere services: kafka: image: apache/kafka:latest container_name: shop-kafka ports: - "9092:9092" environment: KAFKA_NODE_ID: 1 CLUSTER_ID: MkU3OEVBNTcwNTJENDM2Qk KAFKA_PROCESS_ROLES: broker,controller KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092 KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true" # single broker: replication factors must be 1, or topic creation hangs KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 producer: build: ./producer container_name: shop-producer ports: - "8000:8000" depends_on: - kafka consumer: build: ./consumer container_name: shop-consumer depends_on: - kafka

The replication-factor lines are the ones everyone forgets: their defaults assume three brokers, and on a single broker topic creation hangs forever. The advertised listener uses the Compose service name kafka, which is exactly the hostname the two Python services must dial.

producer/Dockerfile and requirements

# producer/Dockerfile (consumer/Dockerfile is identical, with worker.py) FROM python:3.12-slim WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY app.py . CMD ["python", "app.py"] # requirements.txt — one line: kafka-python

producer/app.py — HTTP in, Kafka out

import json import time from http.server import BaseHTTPRequestHandler, HTTPServer from kafka import KafkaProducer producer = KafkaProducer( bootstrap_servers="kafka:9092", value_serializer=lambda item: json.dumps(item).encode("utf-8"), ) class OrderHandler(BaseHTTPRequestHandler): def do_POST(self): length = int(self.headers.get("Content-Length", 0)) payload = self.rfile.read(length).decode("utf-8") producer.send("orders", {"raw": payload, "at": time.time()}) producer.flush() body = b"accepted" self.send_response(200) self.send_header("Content-Length", str(len(body))) self.end_headers() self.wfile.write(body) HTTPServer(("0.0.0.0", 8000), OrderHandler).serve_forever()

consumer/worker.py — Kafka in, stock reservation out

import json from kafka import KafkaConsumer consumer = KafkaConsumer( "orders", bootstrap_servers="kafka:9092", group_id="stock-keepers", value_deserializer=lambda blob: json.loads(blob.decode("utf-8")), auto_offset_reset="earliest", ) for message in consumer: order = message.value print("reserve stock for", order, flush=True)

The group id is the scaling knob: run three workers with --scale consumer=3 and Kafka spreads the topic's partitions across them. Restart a worker mid-flow and it resumes from its committed offset — no order processed twice, none lost.

Run it: broker first, apps second

docker compose up -d kafka

Wait for docker compose logs kafka to show the broker started, then bring up the applications (two steps because depends_on orders startup but does not wait for readiness — a consumer starting against a cold broker exits instead of retrying):

docker compose up -d --build

Send an order, watch the worker

curl -X POST localhost:8000 -d "{"sku":"book","qty":2}"

docker compose logs -f consumer

The log line reserve stock for … proves the full path: HTTP → producer → topic orders → consumer. Tear everything down with docker compose down; add -v to also drop the broker data volume.

When not to use Kafka

Three honest no-go cases. First, request/reply: if the caller waits for an answer, an HTTP call or RPC beats a topic pair every time. Second, tiny throughput: a few messages a minute fit a lightweight queue (or a database table) with far less machinery than even one broker. Third, strict global ordering across many writers: Kafka orders inside a partition only, so designs needing one worldwide sequence fight the tool instead of using it.

Article author: Arthur Isaev