A mechanism-level guide to designing and operating real streaming, messaging, and caching systems — for backend engineers who need more than buzzword familiarity.
This course is written for engineers who will be on-call for the systems they build. It covers Kafka's core log abstraction, producer/consumer delivery semantics, replication and reliability internals, schema evolution, Kafka Streams, and the architecture patterns (outbox, event sourcing/CQRS, choreographed sagas) that show up repeatedly in event-driven backends. It also covers the actor model and Akka as a genuinely different concurrency paradigm — not a Kafka substitute — and Redis in depth: as a cache, and as a set of deliberate data structures with their own durability trade-offs.
Every section pairs precise, no-hand-waving explanation with a working interactive simulator that behaves the way the real system behaves — partition rebalances actually reassign partitions, the LRU/LFU simulator actually evicts the correct key, the idempotent-producer demo actually deduplicates. Six of them in total, scattered through the material where the mechanism is easiest to see in motion.
From synchronous request/response and disposable point-to-point queues to a durable, replayable commit log — and why partitioning is both Kafka's unit of parallelism and its unit of ordering.
Kafka is not merely a larger queue; it replaces an architecture of temporary, point-to-point handoffs with a durable, replayable integration substrate. A conventional queue couples sender and receiver to availability, capacity, and consumption rate. Once a consumer acknowledges a message, the broker normally discards it, so another consumer cannot independently recover the same event. Kafka retains records in an ordered commit log for a retention policy or until a size quota is met. Consumers track positions independently, so replay, incident recovery, and new downstream use cases do not require republishing or changing producers.
Request/response remains appropriate for immediate authorization or a synchronous decision, but it is a poor integration boundary. The caller must know the responder's identity and availability; latency propagates directly; retries create duplicate work; and a shared operation cannot evolve independently. A producer publishes an event once, while each consumer interprets it at its own pace. This provides temporal and operational decoupling, though it shifts contract, schema, and semantic compatibility into the event stream itself.
A topic is the logical event category and API namespace. It is physically divided into ordered partitions, which are append-only log segments replicated across brokers. A partition leader accepts writes and serves reads; replicas provide fault tolerance, and an elected replica can become leader after failure. Partitioning therefore also distributes storage and network load across the cluster.
Each appended record receives a monotonically increasing offset within its partition, starting at zero. The offset identifies position, not global event time or production sequence. Consumers store their next desired offset, usually in an internal offset topic. Committing an offset is metadata, not message acknowledgement: it records progress but does not delete data. Committed position and application side effects must be coordinated to define delivery semantics — a topic we return to in the next section. The broker's durable commit log is the authoritative append sequence; replication copies it to followers, while consumer offsets are separate, per-group state.
Partitioning is Kafka's parallelism unit because partitions can be stored, replicated, and consumed independently. Multiple consumers in a group can process different partitions concurrently, subject to the assignment and rebalance protocol. More partitions can increase potential throughput, but only when downstream processing and storage also scale to match.
The same boundary is Kafka's ordering unit. Records in one partition have one total order, exposed through offsets. Kafka does not promise one global order across partitions: two records in different partitions may be written concurrently, observed in different orders by consumers, and processed on separate threads or even separate machines. Consequently, all causally related events that require ordered handling must use a stable partition key that maps them to the same partition. Changing the key changes the ordering domain. If an application genuinely needs cross-partition ordering, it must add a sequencer or an application-level reconciliation mechanism — Kafka itself will not provide it.
The append-only log records, organized by topic and partition — not consumer state or acknowledgements. Retention is a broker-side policy independent of who has read what.
The record's zero-based position within one partition; it is not a timestamp or a global sequence number across the topic.
Consumers retain independent positions while Kafka keeps the underlying records according to the retention policy — rewinding an offset is enough to reprocess history.
Only among records assigned to the same partition. Cross-partition ordering requires an explicit application-level mechanism — Kafka provides none by default.
Partition Assignment & Rebalance Visualizer
A topic has a fixed number of partitions. Add or remove consumers from the group and watch how partition ownership is reassigned round-robin — and, critically, which partitions actually had to move.