A field guide to the duplicate events, stale state, rebalances, and deployment traps that quietly distort real-time decisions — told through one decision that should have fired and didn’t.
Note: This piece assumes working familiarity with Kafka and event-streaming fundamentals — topics, partitions, offsets, consumer groups, and delivery semantics. Terms are expanded on first use where it helps, but the focus is on how these behaviors affect real-time decisions, not on introducing Kafka itself. The scenario, systems, and incident described throughout are an illustrative composite created for this article — they do not describe, and are not traceable to, any specific organization, production system, or real-world event.
The failure that doesn’t alert you
Some failures announce themselves. A broker goes down, a consumer crashes, lag spikes, an alert goes off, someone starts a bridge call. Those are the easy ones.
This is about the other kind.
Picture a real-time risk detection system doing its job: consuming events, scoring them, taking action. Dashboards green. Consumer lag normal. No pods restarting. No alerts. By every signal a team usually trusts, the system is healthy.
And yet, somewhere in that healthy-looking pipeline, a decision that should have fired didn’t. An action the system was built to catch went through anyway. Nothing crashed. Nothing logged an error. The pipeline reported itself perfectly fine the entire time.
That is the uncomfortable truth about Kafka in real-time decision systems: the scariest failures aren’t the ones that page you. They’re the ones that let the system look healthy while a decision quietly goes wrong.
This article walks through one such illustrative case — what the symptom looks like, the family of quiet failures that could produce it, the one that actually does, and how it gets fixed. Along the way, every Kafka concept you need to reason about real-time pipelines shows up where it naturally belongs: as a possible explanation for a system that was green and wrong at the same time.
First, what is a real-time risk detection system assessing?
Every large digital platform has decisions that cannot wait
Figure 1. A real-time risk decisioning pipeline. Kafka is not just transport between the boxes — it carries the ordering, delivery, and freshness guarantees every downstream decision depends on. (This diagram depicts a generic, industry-standard streaming architecture for illustration only. It is author-created and does not represent any specific organization’s proprietary architecture or system design.)Should this transaction go through? Should this request be trusted, allowed, held for a closer look, delayed, or turned down?
These decisions happen in milliseconds, with incomplete information, under shifting traffic, and sometimes against people actively trying to beat them. The stakes are different from analytics. In an analytics pipeline, a late or duplicated event makes a dashboard temporarily inaccurate, and you revise it later. In a real-time decision, that same event changes the action taken on a customer, an account, a transaction, or a request — in the moment, with no second draft.
Analytical systems can be wrong and reconcile it later. Real-time risk systems can’t — at least, not without real impact or loss.
That’s why a Kafka-backed decision system needs a stricter mental model than ordinary event transport. Here, Kafka is not just plumbing. Kafka is part of the correctness model. Every arrow on the architecture diagram hides a constraint:
- Did related events stay in order?
- Did every event arrive once — or maybe more than once?
- Did the consumer process fresh state or stale state?
- Did a failed event stop, retry, vanish, or land in a dead letter queue (DLQ)?
- Did two versions of the consumer interpret the same event differently mid-deployment?
- Did the system commit the offset before or after the side effect actually completed?
When one of those assumptions is wrong, you usually don’t get an exception. You get a slightly wrong decision, repeatedly, quietly, at scale. That is far harder to find than a crash — and it’s exactly the kind of thing hiding behind a green dashboard.
So when a decision silently goes missing, where do you even start looking?
The partition key: a decision made before any code runs
Start with the oldest decision in the system, because it’s the one most people forget they made: the partition key.
Kafka guarantees ordering within a partition. It guarantees nothing across partitions.
Partition by a stable entity — account ID, user ID, or another durable key — and every event for that entity lands on the same partition, preserving its order. At any moment one consumer in the group owns that partition, though ownership can move during a rebalance. Partition randomly or round-robin and you get prettier load distribution, but you lose per-entity ordering — which makes anything like “the most recent state for this entity” or “the sequence of changes” much harder to reason about.
There’s no free option that gives you perfect per-entity ordering and perfectly even load. You can mitigate the tradeoff; you can’t wish it away.
So the rule is simple: pick the partition key deliberately, write down why, and treat any later change to it as a correctness change — not a harmless throughput tune-up. A surprising number of “events processed out of order” mysteries trace straight back to a partition key that was chosen for throughput and quietly broke an ordering assumption nobody had written down.
That’s the first place to look. It’s rarely the only one.
The family of quiet failures
When a pipeline is green but a decision is wrong, the cause is almost always one of a small family of failure modes. They share a defining trait: they preserve the appearance of health. The pipeline keeps consuming records, committing offsets, and reporting itself fine — while violating an assumption its decision logic depends on.
It helps to know the whole family, because the symptom rarely tells you which member you’re dealing with. Here are the usual suspects
Figure 2. The family of quiet failures and the guardrail that contains each one. Any of them can keep the pipeline green while a decision goes wrong.Duplicate events inflate reality
Kafka commonly gives you at-least-once delivery: a consumer may see the same message more than once. Fine for logs. Risky the moment events feed counters, thresholds, model features, or enforcement logic. A duplicated event can nudge a running total over a threshold; a duplicated transaction can inflate a velocity feature.
The defense isn’t just producer idempotence — it is consumer-side idempotency keyed on a durable event ID, idempotency key, or business key, so the consumer can say “I’ve already applied this event, don’t apply it again.” In a decision system, idempotency isn’t an optimization. It’s part of the contract.
Out-of-order events corrupt time-based features
The order in which events sit in your Kafka log is not the order they happened in the real world. Kafka preserves order within a partition; it knows nothing about when an event actually occurred — only when it received it. Two different clocks:
- Event time — when the action actually happened.
- Processing time — when your consumer happened to read it.
Plenty of things pull those clocks apart: a mobile client buffering on bad Wi-Fi, a producer retrying a failed send, two services writing about the same entity from machines with skewed clocks, or a backfill replaying yesterday’s data through today’s pipeline.
This matters the instant your logic depends on a time window. A “count of events in the last five minutes” feature, fed three events that were buffered on a slow mobile connection and arrived ten minutes late, gives you the wrong count, on the wrong window, for the wrong moment. The feature is still called five-minute velocity; what it actually measures is “whatever this consumer happened to process in the last five minutes.” Those are not the same thing.
The defense is to stop trusting arrival order and start trusting event time:
- Stamp the real event time at the source, inside the payload, the moment the action happens — and window on that, not the broker’s append time.
- Use version numbers or sequence IDs so a consumer can recognize an older update arriving after a newer one, and ignore or reconcile it.
- Use watermarks (from streaming systems like Flink and Kafka Streams) to mark the point after which earlier events are assumed to have arrived.
- Define a lateness policy — drop, side-channel, reopen the window, or trigger a correction. Pick on purpose.
- Build a correction path for when a late event would have changed a decision already made.
The question teams skip is the important one: what should happen when a late event arrives after a decision has already been made? If the answer is “we don’t know,” then in production the answer becomes “whatever the default is” — and that default is rarely what you’d choose on purpose.
Consumer lag creates stale decisions
The metric most teams watch is record lag — how many records are waiting. The metric a decision system actually cares about is time lag — how stale the processed state may be. Fifty thousand records might mean seconds of staleness on one topic and hours on another. A consumer that’s “only a few minutes behind” can still be dangerously stale if the decision needs current state. The better question is always: how old is the freshest event this decision could have used?
Silent data loss makes risk invisible
The most dangerous event isn’t the one that fails loudly. It’s the one that gets skipped, swallowed, expired, filtered out, or committed before processing actually succeeded. Silent loss comes from weak producer durability, ignored serialization errors, early offset commits, aggressive retention, unmonitored dead letter queues, or replay filters that drop more than intended.
Infrastructure metrics rarely catch it, because records are still flowing. The defense is end-to-end reconciliation: compare what producers sent, what consumers processed, what landed in retries or DLQs, and what reached downstream stores, over explicit windows. That’s the difference between a pipeline that’s merely active and one that’s actually complete.
Two consumers can see two different realities
Two services can consume the same topic and decide about the same entity at the same wall-clock moment. One is current; one is lagging. One has processed a correction event; one hasn’t. One runs new logic; one runs old. Both are healthy. Both are consuming. And their decisions can still contradict each other.
That’s not a Kafka bug — it’s a distributed-systems reality. If multiple services need the same view of state, give them a shared state store, feature store, or materialized view with documented freshness guarantees. If eventual consistency is acceptable, document the window and design downstream behavior to tolerate it. Assuming every consumer sees the same reality because they read the same topic is how non-reproducible incidents are born.
The one that actually bites
Knowing the family is what makes the real cause findable. In our illustrative case, the symptom — a decision that quietly didn’t fire while everything stayed green — isn’t duplicates, or lag, or silent loss. It’s something subtler, and easy to miss, because it only exists for a few minutes during an ordinary, healthy-looking deployment.
Picture a fix that has shipped, with a rolling deploy underway. The service is healthy. Pods are running. The lag looks normal. No alarms.
During such a rollout, the old pods (without the fix) and the new pods (with it) can both be alive, both healthy, and both in the same Kafka consumer group. Kafka does exactly what it’s designed to do: it spreads partitions across all available consumers. Some partitions get handled by the old version, some by the new. Some events are processed with the corrected logic; others with the old logic. Which version handles which event comes down to partition assignment during the deploy window.
From the outside, the fix appears to be deploying successfully. Internally, for a brief window, the pipeline is running two versions of reality at once — and a decision can slip through the gap.
Here’s the thing worth internalizing: a Kafka consumer group is a load-balancing pool. Every consumer that joins with the same group.id is treated as interchangeable. Kafka has no idea one pod runs old logic and another runs new. So:
A consumer group is a statement of code identity. If two consumers share a group, you’re telling Kafka they behave interchangeably. If they don’t, you’ve quietly opted into inconsistent processing.
No one has to do anything reckless. A perfectly normal rolling deployment can briefly turn a single consumer group into two different behaviors — and that’s enough.
Closing the gap: deploying without splitting reality
Once the cause is clear, the fix is mostly about not letting a deploy mean two things at once.
Decouple deployment from behavior change. Ship new code dark behind a feature flag. Let the rolling deploy change the binary, not the behavior. Once every pod is on the new version, flip the behavior intentionally. Often the simplest and most effective option.
Use versioned consumer groups for semantic changes. When processing behavior changes meaningfully, use a new group ID — risk-features-v3 → risk-features-v4 — and run both in parallel during cutover. Be deliberate about where the new group starts: earliest can cause a replay storm, latest can leave a data gap. Neither is automatically right.
Use blue/green consumer groups. Run the new group in parallel, compare outputs, then cut over once you trust the result. More operational machinery, but the deployment becomes observable and reversible.
Reduce unnecessary rebalances. Static group membership via group.instance.id cuts avoidable rebalances during normal restarts. It doesn’t fix mixed-code semantics on its own, but it shrinks the windows where partition-ownership churn creates surprises.
A Kafka consumer deployment is not just a compute rollout. It’s a temporary change to how partitions, offsets, schemas, state, and code versions interact. Treat it that way.
The configs that bound the blast radius
Configuration doesn’t make a system correct — it moves failure modes into shapes you can reason about. A reasonable baseline separates producer, topic, and consumer responsibilities:
# Producer
acks=all
enable.idempotence=true
max.in.flight.requests.per.connection=5
# Topic / broker durability
replication.factor=3
min.insync.replicas=2
# Consumer
enable.auto.commit=false
isolation.level=read_committed
A few caveats matter:
- `min.insync.replicas=2` assumes a replication factor of three or higher. With acks=all, it forces writes to fail rather than silently accepting under-replicated ones.
- `isolation.level=read_committed` only helps when upstream producers use Kafka transactions; it stops consumers from reading aborted transactional writes. It is not a general duplicate-prevention switch.
- `enable.auto.commit=false` only helps if you commit offsets after successful processing and after required side effects are safely done.
These aren’t magic settings. They’re guardrails. The goal is to make failure visible, bounded, and recoverable.
Three patterns worth adopting
None of these are novel, but they’re routinely skipped — and routinely missed when they’re absent.
Treat the DLQ as a signal, not a graveyard. Every Kafka shop ends up with a dead letter queue, and most treat it as a parking lot for “later.” A growing DLQ isn’t a backlog — it’s a leading indicator of a problem nobody has diagnosed yet. Alert on its rate of growth, not its size, and assign someone to drain it on a cadence. A DLQ nobody looks at is just a slower form of data loss.
Back the real-time pipeline with a batch correction path. Real-time pipelines are fast and approximate; batch pipelines are slow and accurate. Run both: the streaming layer makes the live decision, and a daily (or hourly) batch job recomputes the same features over a longer window to catch what streaming missed. They don’t compete — they reconcile. When the two layers disagree, that disagreement is one of the most valuable signals in the system.
Make retries boring. Aggressive retries during a partial outage don’t recover the system — they amplify it. The defaults that hold up: exponential backoff with jitter, a hard retry cap, and a separate retry topic so retries can’t starve the main flow. The strategies that look “more responsive” on paper are usually the ones that turn a five-minute hiccup into a thirty-minute incident.
Build for visible, bounded, intentional failure
The goal isn’t a Kafka pipeline that never fails. It will fail. A producer will retry. A consumer will crash between a side effect and an offset commit. A schema will evolve. A partition will get hot. A replay will surprise someone. A rolling deploy will, for a moment, put two versions of reality in the same system.
The goal is to make those failures visible, bounded, and intentional.
- Visible, so you know the moment a correctness assumption is being violated — instead of finding out later.
- Bounded, so one bad event, deployment, or consumer doesn’t silently contaminate everything downstream.
- Intentional, so recovery reflects a tradeoff someone chose — not whatever default happened to be convenient when the service was first written.
Kafka is powerful, widely used infrastructure. But in a real-time decision system it is more than infrastructure. It’s the courier carrying the evidence, the timeline that orders the story, and the contract between what happened and what the system believes happened.
Treat it as part of the decision model.
Because the pipeline can be green while the decisions are wrong — and those are the failures that rarely page you first
Kafka Inconsistencies and the Silent Failures Behind Real-Time Decision Platforms was originally published in Walmart Global Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
Introduction to Malware Binary Triage (IMBT) Course
Looking to level up your skills? Get 10% off using coupon code: MWNEWS10 for any flavor.
Enroll Now and Save 10%: Coupon Code MWNEWS10
Note: Affiliate link – your enrollment helps support this platform at no extra cost to you.