The request that starts every real-time analytics project sounds modest: the nightly report is not fast enough, can we see it live. What that actually asks for is a different system, because a batch job and a stream processor disagree on something fundamental — a batch job knows when its input is complete, and a stream processor never does.
At Way.com I built real-time analytics on Kafka Streams, Elasticsearch and Kibana over the traffic from a gateway handling 5M+ requests a day. Everything difficult about it came from that one asymmetry.
The split: what each tool is genuinely good at
Kafka Streams and Elasticsearch overlap enough to argue about and are built for opposite access patterns. Streams is a library for continuous, stateful computation over an unbounded log — it excels at “maintain this aggregate as events arrive” and is a poor fit for “answer an arbitrary question somebody just thought of.” Elasticsearch is the inverse: excellent at ad-hoc slicing and filtering of documents, and a bad place to maintain a running aggregate transactionally.
So the rule we settled on: Streams computes anything that must be correct and continuously maintained, Elasticsearch stores the results plus enough raw detail to explore, and Kibana is where people ask questions nobody anticipated. Getting that boundary wrong in either direction is the usual failure — recomputing aggregates in Elasticsearch on every dashboard refresh, or trying to serve exploratory queries from a Streams state store.
Windows, and the event that arrives late
“Requests per minute per client” needs a definition of “minute” and there are two candidates: when the event happened, or when we processed it. Processing time is trivially easy and produces numbers that are wrong in a specific, embarrassing way — a consumer lag spike makes traffic appear to move in time. Event time is correct and forces you to confront the question processing time lets you ignore: what do you do with an event that arrives after its window closed?
There is no free answer. A grace period buys correctness with latency; no grace period buys latency with silently dropped data. What matters is choosing deliberately and telling the dashboard's users which one they are looking at, because a dashboard that quietly revises last hour's number without saying so destroys trust in every number on it.
The related trap is emitting a result per update. A windowed aggregation in Kafka Streams will, by default, produce an updated value every time the aggregate changes, so a one-minute window over a busy client emits thousands of intermediate rows — all of which are correct, all of which your downstream index has to absorb, and only the last of which anyone wanted. Suppressing until the window closes converts that into one emission per window, and it is the single change that most reduces downstream load.
State, and why it needs a changelog
The part of Kafka Streams that surprises people is that it is genuinely stateful. Aggregations live in a local store — RocksDB on the instance's own disk — which is what makes them fast and also means an instance holds data that is not in the topic it is reading.
Streams handles that by backing every state store with a changelog topic in Kafka, so the local store is a cache of durable state rather than the state itself. This is the detail worth internalising, because it determines the operational behaviour you will actually experience: when an instance dies, a replacement rebuilds its store by replaying the changelog, and how long that takes is a function of store size. Discovering that during your first rebalance under load is unpleasant. Sizing windows and retention with it in mind is not.
It also explains why over-partitioning hurts here in a way it does not for a stateless consumer. State is partitioned along with the input, so every partition is a store to rebuild.
Indexing: the mapping is the whole game
Elasticsearch will happily index anything and infer a mapping for you, and dynamic mapping is how a real-time analytics index becomes unqueryable. A high-cardinality identifier indexed as full text, a numeric field inferred as a keyword from one bad document, a nested object whose shape varies — each is fine on day one and expensive to unwind on day ninety, because a mapping cannot be changed in place.
Two things that made this manageable: define the mapping explicitly up front and disable dynamic inference, and write through an index alias that points at date-based indices. The alias is what makes retention a delete rather than a migration, and it is what lets you reindex into a new mapping and swap without the dashboards noticing.
On identifiers: use the rollup's natural key — client plus window start — as the document id, so re-delivery from Kafka overwrites rather than duplicates. That single choice makes the indexer idempotent for free, which matters because the delivery guarantee between Kafka and any external sink is at-least-once whatever you do.
Real-time analytics and monitoring are not the same thing
The mistake I would warn hardest against: using this pipeline as your operational monitoring. The temptation is obvious — the gateway's request events contain status codes and latencies, so the Elasticsearch index can render an error-rate dashboard, and now you have one system instead of two.
The problem is dependency direction. This pipeline runs through the gateway, Kafka, a stream processor and an index. Any of those failing degrades your visibility at exactly the moment you need it, and you cannot diagnose a Kafka outage on a dashboard whose data flows through Kafka. Operational monitoring belongs on a path that shares as little as possible with the thing it watches.
Keep them separate and each is straightforward. The business question — which partners drive volume, where are conversions falling off, what is happening in the last five minutes — belongs in this pipeline. The question of whether the platform is healthy belongs somewhere that does not depend on the platform being healthy.