Event-Driven Architecture Is Powerful — Until It Becomes an Operational Nightmare
A practical architecture deep dive into why event-driven systems look elegant on diagrams but become much harder once production reality shows up.
Every backend team that adopts event-driven architecture gets the theory right: loose coupling, independent scaling, services that communicate through events without knowing about each other. The diagrams look clean. The pattern makes sense.
Then you deploy it.
Events arrive out of order. A consumer crashes mid-processing — and nobody knows whether the message was handled. A schema change silently breaks three downstream services. Your dead-letter queue fills up with unprocessed business events while dashboards show green. Retry logic that worked in staging is now causing duplicate writes in production.
The gap between “this works locally” and “this runs reliably at scale” is where most teams get hurt.
This article goes into that gap — the real failure modes, the patterns that quietly degrade, and the operational discipline required to run event-driven systems in production without constant firefighting.
Support My Content
Consider supporting me and my content by leaving a like and following me on Substack, Medium, and LinkedIn.
Why Teams Love Event-Driven Architecture
The benefits of EDA are not theoretical. They show up in real systems solving real problems, and they’re worth stating clearly before we get into where things break.
The most immediate win is loose coupling:
In a synchronous service mesh, Service A calls Service B, which calls Service C. Every service knows about the next one. Change B’s API, and A breaks
With events, the producer doesn’t know or care who’s listening. It publishes an `
OrderPlaced` event and moves on. The billing service picks it up. So does inventory. So does the notification pipeline. The producer is aware of none of them. You can add a fifth consumer next quarter without touching a single line of producer code.
Fanout is where this gets powerful. One event, many reactions — all independent, all deployable separately, all owned by different teams. The order service doesn’t coordinate billing and analytics. It just says what happened. Everyone else decides what to do about it.
The following diagram illustrates how a single event fans out to multiple independent consumers:
One event, four independent reactions — the Order Service doesn’t know any of these consumers exist, and adding a fifth one next quarter requires zero changes to the producer.
Independent scaling follows naturally. Your analytics consumer is slow and bursty? Give it more instances and a bigger queue depth. Your notification service handles low volume? Keep it lean. None of this requires the producer to change, redeploy, or even know about it. Each consumer scales on its own terms, based on its own load profile.
Async workflows change the user experience too. A synchronous checkout that blocks while it processes payment, updates inventory, and sends a confirmation email is fragile and slow. An event-driven version accepts the order, returns a response, and lets downstream processes handle the rest in the background. The user gets a fast response. The system gets breathing room.
There’s also a modeling argument that doesn’t get enough attention. Some domains are inherently reactive — fraud detection, audit logging, real-time personalization. These systems don’t initiate actions. They respond to things that already happened. Events are the natural representation. Trying to force these into synchronous request-response patterns creates awkward, tightly coupled designs that fight the domain instead of reflecting it.
None of this is niche anymore. AWS built entire managed services around these patterns:
EventBridgefor event routingSNS+SQSfor pub/sub with durable queuesKinesisfor high-throughput streaming.
These aren’t experimental offerings. They exist because event-driven architecture works, and enough teams run it in production that the infrastructure has matured around it.
So yes — the architecture earns its popularity. The question is what happens after you adopt it.
The Happy-Path Architecture: Most Diagrams Stop At
Every event-driven system, no matter how complex it eventually becomes, starts from three building blocks.
The producer detects a state change — an order placed, a payment confirmed, a user updated — emits an event describing what happened, and moves on. It doesn’t wait for a response. It doesn’t care who’s listening.
The broker sits in the middle. It receives events, routes them based on rules or topics, and delivers them to the right downstream consumers. It’s the postal system of the architecture: accept, sort, deliver.
The consumer picks up events from its queue and processes them. It might update a database, trigger a workflow, or feed an analytics pipeline. It works at its own pace, decoupled from whatever produced the event in the first place.
Here’s how these three building blocks map to AWS services:
The happy path maps cleanly to AWS — SNS or EventBridge routes events, SQS queues buffer them, and Lambda or ECS processes them. It’s clean until production traffic hits it.
In AWS terms, this maps cleanly. An Order Service publishes an `OrderPlaced` event to Amazon `SNS` or `EventBridge`. The broker routes it to one or more `SQS` queues based on subscription rules.
A `Lambda` function processes the event for fulfillment.
An ECS service handles inventory updates.
An analytics consumer writes to a data lake.
Each piece independent, each piece replaceable.
The elegance is real. Services don’t need to know about each other. You can add a new consumer without touching the producer. You can scale each component independently. On a whiteboard, the arrows flow in one direction, the boxes are clean, and the whole thing fits on a single slide.
This is the model that conference talks present, that AWS documentation walks through, and that most teams have in mind when they decide to go event-driven. And it works — genuinely works — for the first few services, the first few event types, the first few months of production traffic.
But the diagram makes promises it doesn’t label. It assumes:
Every event the producer emits will be delivered “exactly once”
Consumers will receive events in the “order they were produced”
Failures will surface visibly and immediately
Event schemas won’t change in ways that break downstream consumers silently
You can observe, trace, and debug any event’s journey through the system
Reliable delivery. Correct ordering. No duplicates. Observable failures. Schema stability.
Five assumptions. None of them on the diagram. All of them load-bearing.
Delivery Is Not Business Correctness
Most brokers default to “at-least-once delivery”. The name says it plainly: your consumer will receive the message at least once. Possibly more.
This isn’t a bug. It’s the design. The broker retries when it doesn’t get an acknowledgement. Consumer crashes, network timeouts, slow processing — all of these look the same to the broker. No ACK received means re-deliver.
Here’s the problem: the broker’s job ends at delivery. It has no concept of what your application did with the message.
Consider an `AccountDebited` event carrying a $200 debit. Your consumer reads the event, subtracts $200 from the account balance, and then crashes — right before sending the ACK. The broker sees a failed delivery. It retries. Your consumer comes back up, processes the same event, subtracts another $200.
The balance is now wrong by $200. No error was thrown. No exception logged. No alert fired. The broker’s metrics show a successful delivery. Your monitoring is green. The customer’s account is silently incorrect.
That’s the gap. Delivery semantics and processing semantics are not the same thing. The broker guarantees the message arrived. It says nothing about what happened next.
The following diagram shows the disconnect between what the broker reports and what actually happens:
Teams using Kafka often assume that exactly-once semantics solve this. They don’t — not in the way most people think. Kafka’s exactly-once guarantee applies to Kafka-to-Kafka flows: reading from one topic, transforming, writing to another topic, all within the Kafka transaction boundary. The moment your consumer reads from Kafka and writes to PostgreSQL, you’ve left that boundary. You’re back in at-least-once territory.
The only thing that closes this gap is idempotency at the “application layer”. Not broker configuration. Not infrastructure tuning. Application code that can process the same message twice and produce the same result once.
Two separate design concerns. Two separate responsibilities:
“Delivery” semantics — what the broker promises about getting messages to consumers
“Processing” semantics — what your application guarantees about the business outcome of handling those messages
The broker owns the first. You own the second.
Conflating them is where the silent failures start.
Duplicate Events and Idempotency
If your broker guarantees at-least-once delivery — and most do — then duplicate events are not edge cases. They are normal operations. Network blips, consumer restarts, rebalancing partitions: all of these trigger redelivery. Your system will process the same event more than once. The question is whether it handles that safely.
The answer is idempotent consumers. An idempotent consumer produces the same result whether it processes a message once or five times. Debit an account $50 on the first delivery, and the balance drops by $50. Receive that same event again, and nothing changes. No double-debit. No corrupted state.
Every consumer that mutates business state needs this property. Not some of them. All of them.
This diagram walks through how an idempotent consumer handles the first delivery versus a duplicate:
First delivery: process and record. Second delivery: check the ID, discard, ACK. The business outcome stays correct regardless of how many times the broker retries.
There are two standard patterns for implementing this.
Pattern 1: Dedicated tracking table: Create a `
PROCESSED_MESSAGES` table that stores message IDs. Before processing any event, check if the ID already exists.
— If it does, skip the event and ACK it.
— If it doesn’t, insert the ID and apply the business logic in the same database transaction.
The transaction is critical — if you check and write in separate operations, you’ve introduced a race condition between duplicate deliveries.Pattern 2: Embed the ID in the business entity: Instead of a separate table, store something like `
last_processed_event_id` directly on the entity being modified — the account record, the order row, whatever owns the state. Same principle: check the ID before processing, update it atomically alongside the business mutation. This approach avoids the extra table and keeps the idempotency check co-located with the data it protects.
Both patterns work. Pattern 1 scales more cleanly across multiple event types. Pattern 2 is simpler when a consumer only modifies a single entity per event.
A note on AWS `SQS FIFO` queues: they offer exactly-once processing within a five-minute deduplication window. That helps at the “queue level”. But it does not replace application-level idempotency. The window expires. Consumers restart. Infrastructure-level deduplication is a convenience, not a contract your business logic can rely on.
Here’s the nuance that trips teams up. Idempotency is not about preventing the broker from sending duplicates. You can’t control that. It’s about making your consumer safe to call multiple times with the same input. The broker’s job is delivery. Your job is correctness. Build the consumer so that redelivery is a non-event — literally.
Ordering Is Fragile
Most teams carry an implicit assumption into event-driven design: events will be processed in the order they were produced. Producer emits `OrderPlaced`, then `OrderConfirmed`, then `OrderShipped`, and the consumer will see them in exactly that sequence. This assumption is wrong. Not sometimes wrong. Structurally wrong.
Start with what brokers actually guarantee. Kafka guarantees ordering within a single partition. Cross-partition ordering does not exist. If your topic has 12 partitions and two related events land on different ones, their processing order is undefined. `SNS` standard topics feeding `SQS` standard queues provide no ordering guarantee whatsoever — messages can arrive in any sequence, and AWS documents this explicitly. `SQS FIFO` queues and single-partition Kafka topics do preserve order within a message group or partition, but both come with throughput ceilings and operational caveats that teams frequently underestimate.
Even when the broker provides ordering guarantees on paper, three failure modes break them in practice:
Retries. A message fails processing and returns to the queue. By the time the consumer retries it, subsequent messages from the same producer have already been processed. The retry lands after its logical successors. The consumer sees events out of their original temporal sequence — not because the broker reordered them, but because failure recovery did.
Parallel consumers. Multiple consumer instances pulling from the same queue process messages concurrently. A consumer group with five instances does not guarantee that instance 1 finishes message A before instance 3 finishes message B. Each instance operates on its own timeline. Concurrency and ordering are fundamentally in tension.
Partition rebalancing. In Kafka, when consumers join or leave a consumer group, the broker reassigns partition ownership. During the rebalance window, in-flight messages can be re-delivered, processing gaps appear, and the ordering guarantee within a partition — the one guarantee Kafka does make — is temporarily disrupted.
This diagram shows how retries and parallel consumers break the original event sequence:
The consequences are not abstract. An `OrderCancelled` event processed before the `OrderPlaced` event it logically depends on. The consumer tries to cancel an order that does not exist yet in its local state. The cancellation fails or, worse, gets silently dropped. The order then arrives and gets created — an order that should have been cancelled is now active. This is not a theoretical edge case. It is a Tuesday.
Ordering does not come for free. It requires explicit design. Use partition keys to route related events to the same partition so the broker’s single-partition guarantee actually applies. Attach sequence IDs to events so consumers can detect gaps and reorder if necessary. Build idempotent state machines that tolerate out-of-order arrival — a consumer that receives `OrderCancelled` before `OrderPlaced` should be able to reconcile both once they arrive, regardless of sequence. Or accept that ordering cannot be guaranteed at the infrastructure level and design every consumer to handle it.
The broker moves messages. It does not sequence your business logic. That is your job.
Retries, Poison Messages, and Dead-Letter Queues
When a consumer fails to process a message, the broker retries delivery. This is correct behavior. Transient failures — a downstream service briefly unavailable, a network hiccup, a cold start timeout — resolve themselves on the second or third attempt. Retries exist because most failures are temporary.
But retry policies are not gentle. They are aggressive by design, and most engineers never look at the actual numbers.
AWS `SNS`, for example, uses a four-phase retry policy for AWS-managed endpoints:
Phase one: 3 immediate retries with no delay
Phase two: 2 retries, 1 second apart
Phase three: 10 retries with exponential backoff ranging from 1 to 20 seconds
Phase four: up to 100,000 retries, 20 seconds apart
That is a total of 100,015 delivery attempts over 23 days. For a single message. If your consumer cannot process that message, the broker will spend over three weeks trying to convince it otherwise.
Now consider what happens when the message itself is the problem. A malformed payload. A schema mismatch between producer and consumer. A bug in consumer logic that only triggers for a specific event shape. That’s a poison message — one your consumer will never successfully process, no matter how many times the broker retries. Without isolation, that message blocks the queue, burns through retry budgets, floods logs with identical stack traces, and triggers alert fatigue that masks real incidents happening elsewhere.
The following diagram traces the full retry lifecycle from first delivery attempt to the dead-letter queue:
Dead-letter queues exist to isolate these messages. After a configurable number of failed processing attempts, the broker moves the message out of the main queue and into the DLQ. The main queue unblocks. Healthy messages flow again. In `SQS`, you configure this through a redrive policy with a `maxReceiveCount` threshold — after that many failed receives, the message moves. `EventBridge` works similarly, routing failed events to a configured DLQ after exhausting its own retry attempts.
Most teams stop here. They configure the DLQ, verify it catches poison messages, and move on. That is the mistake.
A dead-letter queue is not a solution. It is an isolation mechanism.
Every message sitting in a DLQ is an unprocessed business event. A user placed an order that was never fulfilled. A payment was captured but inventory was never reserved. A state change propagated halfway through a workflow and stopped. These are not abstract failures. They are real operations your system promised to complete and didn’t.
Ignoring a DLQ is not the same as having no problem. It means your problem is accumulating silently, in a queue nobody is watching, while dashboards show green.
Good DLQ operations require active discipline:
CloudWatch alarms on DLQ depth so you know the moment messages land there
Runbooks for the most common failure types so on-call engineers don’t start from scratch
A regular review cadence, weekly at minimum, to inspect what’s accumulating
Replay pipelines that let you reprocess DLQ messages back into the main queue once the underlying bug is fixed
A DLQ you don’t monitor is a graveyard for business events you owe your users.
Eventual Consistency Is a Product Problem, Not Just a Tech Problem
In a monolith with a single database, a transaction either commits or it doesn’t. The user clicks “Place Order,” the system does its work, and the response reflects the final state. There is no window where the order exists but payment hasn’t been confirmed. No gap where inventory is reserved but the order status still says “pending.” The user sees the result. Clean.
Event-driven architectures don’t give you that luxury. When a business transaction spans multiple services — order creation, payment processing, inventory reservation — there is no single ACID transaction wrapping the whole thing. Each service owns its own data store and its own local transaction. Consistency across those boundaries is eventual. The system will get there, but not atomically, and not instantly.
The saga pattern addresses this. A saga is a sequence of local transactions, each publishing an event that triggers the next step. There are two coordination styles:
Choreography — each service listens for events and reacts autonomously
Orchestration — a central saga coordinator directs the sequence explicitly
Both achieve the same goal — completing a multi-step business transaction without a distributed lock — but neither gives you isolation.
This diagram compares the two saga coordination approaches side by side:
That’s the critical gap. While a saga is in progress, the “I” in ACID does not exist. Other read queries, other sagas, and most importantly the user can observe intermediate states. A customer who checks their order status between the `OrderPlaced` event and the `PaymentConfirmed` event will see “Status: PENDING.” Technically accurate. Functionally confusing. They don’t know if the system is still working or if something broke.
This sequence diagram shows how intermediate states become visible to users during a saga:
Now consider what happens when something fails mid-saga. There is no automatic rollback. If the payment service rejects the charge after the order service already created the record, you need an explicit compensating transaction — an event that tells the order service to release or cancel that order. This is not free. Every saga step that can succeed needs a corresponding compensation path for when a downstream step fails. You are designing the undo logic yourself, and it has to account for the fact that users may have already seen the intermediate state.
These aren’t edge cases. They’re the normal operating mode:
Orders sitting in “
pending” while payment processes asynchronouslyInventory counts briefly wrong after a write because the read model hasn’t caught up
Dashboards showing stale totals
Checkout flows that need polling or WebSocket connections just to tell the user “yes, it actually worked”
Eventual consistency creates a design obligation. Teams adopting EDA must treat these windows as real product concerns, not backend quirks to hand-wave away. That means optimistic UI updates with rollback paths. Clear loading and transitional states. Async notification patterns that tell the user when the saga completes, rather than leaving them to guess. The system’s consistency gaps are not invisible to the end user. If you don’t design the experience around them deliberately, your users will design their own interpretation — and it won’t be the one you want.
The Transactional Outbox: Why Publishing Events Is Harder Than It Looks
Here’s the failure mode that catches almost every team the first time. Your order service receives a request. It writes the order to the database. Then it publishes an `OrderPlaced` event to `SNS`. Two operations, one after the other. It works fine — until the service crashes between the database commit and the publish call. The order exists. The event doesn’t. Downstream services never learn about it. Inventory isn’t reserved. The customer is never notified. And nothing in your system flags that anything went wrong, because from the database’s perspective, the write succeeded.
You might think: flip the order. Publish the event first, then write to the database. But now you have the inverse problem. The event goes out, consumers start processing it, and then the database write fails — maybe a constraint violation, maybe a timeout. The event described a state change that never actually happened. Consumers are now acting on a phantom.
Both directions break because you’re performing two operations against two different systems — a database and a message broker — without a shared transactional boundary. The textbook answer is a distributed transaction using two-phase commit (2PC). In practice, most message brokers don’t participate in 2PC. Even where it’s technically possible, it’s slow, tightly coupled, and fundamentally at odds with the autonomy that microservices are supposed to give you. 2PC is not a real option here.
The transactional outbox pattern solves this by removing the second system from the critical path entirely. Instead of publishing directly to the broker, you write the event as a row into an outbox table in the same database, inside the same local transaction as the business state change. One transaction. One commit. The order row and the outbox row either both persist or neither does.
A separate message relay process — a background worker, a `Lambda`, a CDC reader — polls the outbox table, picks up unprocessed rows, publishes them to the broker, and marks them as sent. The relay is decoupled from the business logic. It doesn’t need to be fast. It needs to be reliable.
This diagram shows how the outbox pattern eliminates the dual-write problem:
There’s a catch, and it should sound familiar. If the relay crashes after publishing an event but before marking the outbox row as sent, it will re-publish that event on restart. This is the at-least-once delivery problem from Section 5 resurfacing in a different layer. The outbox guarantees that every state change produces an event. It does not guarantee exactly once. Your consumers still need to be idempotent — and now you understand why that requirement isn’t optional, it’s structural.
On AWS, you don’t always need to build the relay yourself. `DynamoDB Streams` gives you a built-in change data capture (CDC) stream that a `Lambda` function can consume directly. `Aurora` supports CDC through AWS Database Migration Service or `EventBridge` Pipes. The outbox table becomes a `DynamoDB` item or an `Aurora` row, and the relay becomes managed infrastructure. The pattern stays the same — you’re just letting AWS operate the polling loop.
The transactional outbox isn’t glamorous. It’s a table and a background process. But it closes the gap that silently breaks event-driven systems: the gap between “the state changed” and “someone knows about it.”
Schema Evolution and Consumer Drift
In synchronous APIs, breaking changes surface fast. A client makes a call, the call fails, someone gets paged, and the field gets fixed. The feedback loop is tight because the failure is immediate.
Event-driven systems have no such luxury. Consumers process events asynchronously, often on their own deployment schedules. A producer can rename a field, ship the change on Tuesday, and not hear about the downstream breakage until the following week — when a consumer finally processes an event with the new shape and silently produces wrong results.
That’s the core problem: events are implicit contracts. The producer promises a certain structure. Every consumer builds its logic around that structure. But unlike a versioned REST API with OpenAPI specs and client libraries, there is no compile-time enforcement. When the shape changes, the contract breaks — silently, asynchronously, sometimes weeks after the offending commit merged.
How Schemas Break Consumers
The ways schemas break are mundane, which is exactly what makes them dangerous.
A producer renames `userId` to `customerId`. The consumer still reads, `userId` gets `null`, and either stores bad data or throws an error deep in its processing logic.
No compile-time warning existed. A producer removes a field a consumer depended on — same outcome. A producer changes a field type from integer to string. Serialization works fine, but the consumer’s arithmetic breaks at runtime. A producer adds a required field with no default. New consumers expect it; old consumers have never seen it.
This diagram shows how a simple field rename silently breaks downstream consumers:
Every one of these failures shares a trait: they are hard to detect because the consumer may not encounter the new event shape immediately. The broken events sit in a stream or queue, waiting to be processed, and the damage only becomes visible when they are.
Three Compatibility Strategies
There are three standard approaches to evolving schemas without breaking consumers.
Backward compatibility means new schemas work with old consumers. Old consumers can process new events without modification. You achieve this by only adding optional fields, never removing or renaming existing fields, and never changing field types. This is the most practical default for most teams.
Forward compatibility means old schemas work with new consumers. New consumers can handle events produced before the schema update. This requires consumers to tolerate missing fields and ignore unknown ones rather than failing on unexpected input.
Full compatibility combines both: new events work with old consumers, and old events work with new consumers. This is the gold standard but the most restrictive — it limits what you can change in any direction.
The following diagram compares how each strategy handles schema changes:
The practical recommendation: default to backward compatibility, design every consumer to be tolerant of unknown fields, and version events explicitly when a genuinely breaking change is unavoidable.
Schema Registries
A schema registry is a centralized store that tracks event schemas and enforces compatibility rules at publish time. When a producer attempts to register a schema change that violates the configured compatibility policy — say, removing a required field under backward compatibility — the registry rejects the change before the event ever reaches a consumer.
AWS Glue Schema Registry integrates with `Kinesis`, MSK, and `SQS`. Confluent Schema Registry is the standard for Kafka-native deployments. Both support Avro, JSON Schema, and Protobuf.
What registries do not solve: they only protect event types that are actually registered. Ad-hoc events, undocumented fields, and events published through paths that bypass the registry remain uncontrolled. A registry is only as good as the discipline around it. Teams that skip registration for “internal” events or bypass validation for speed get none of the protection.
Schema discipline is not optional in long-running event-driven systems. It is an operational concern that compounds in cost the longer it is deferred.
Debugging and Observability Across Async Flows
In a synchronous system, debugging follows a familiar path: you pick up the call stack, walk through the execution thread, and trace cause to effect in a single linear chain. The mental model is sequential, and the tooling assumes it.
Event-driven systems break that model completely. A single user action produces an event that fans out to multiple consumers, each of which may emit further events. There is no single execution thread to follow. A failure in Service D might trace back to a malformed event produced by Service A two hours earlier, but nothing in the infrastructure connects those two moments for you. The relationship is temporal and indirect.
Consumer failures make this worse. A downstream service might silently fail, retry, and eventually land the message in a dead-letter queue. By the time you inspect the DLQ, the original producer context — why the event looked the way it did, what state the system was in — is gone. And if you replay events from the DLQ, those replayed messages appear in today’s trace timeline with no indication they originated last week. Temporal causality in your traces is now broken.
Correlation IDs: The Minimum Baseline
The standard mitigation is straightforward in theory: every event carries a correlation ID, and every downstream event produced as a result of processing that event propagates the same ID. This allows you to reconstruct the full event chain by filtering on a single identifier across all logs and traces.
The catch is consistency. For correlation IDs to work, every producer in the system must populate and forward them, in every code path, including error handlers, retry logic, and fallback branches. One service that strips the ID or fails to propagate it breaks the entire chain downstream of that point.
Correlation ID propagation should be a platform-level contract, not a per-team convention. If it depends on individual developers remembering to pass a header, it will fail exactly when you need it most.
AWS X-Ray and EventBridge Tracing
AWS `X-Ray` integrates with `EventBridge` to propagate trace context automatically through event targets. When a `Lambda` function instrumented with the `X-Ray` SDK publishes an event to `EventBridge`, the trace header is carried downstream to every target: `Lambda` consumers, `SQS`-triggered functions, `Step Functions`, and others.
In practice, this means you can follow an order from the moment it enters API Gateway, through the `Lambda` handler that publishes the event, into `EventBridge`, and out to each downstream `Lambda` — all rendered as a single trace. The `X-Ray` service map visualizes the full chain: which service produced the event, which targets received it, and the latency at each hop.
This diagram shows how `X-Ray` propagates trace context across an event-driven chain:
That’s useful — when it works. The requirement is that every service in the chain is instrumented. Which brings us to the limits.
What Observability Still Cannot Do For You
`X-Ray` trace propagation breaks the moment one service in the chain is not instrumented. A single uninstrumented `Lambda` or container service creates a gap in the trace, and everything downstream of that gap becomes invisible to the trace context.
Correlation IDs have the same fragility. They work perfectly until one producer, in one code path, in one error handler, fails to forward them. Replayed DLQ events compound the problem: they insert stale events into current trace timelines with no metadata distinguishing them from live traffic. Your dashboards show activity that looks current but originated days ago.
Infrastructure-level tracing and business-level tracing are not the same thing. `X-Ray` can tell you that a `Lambda` function was invoked and how long it took. It cannot tell you what happened to order #48721. That requires application-level logging — structured, consistent, and correlated — layered on top of infrastructure traces.
Observability in event-driven systems is not something you bolt on after the first production incident. It must be designed in from the start, enforced at the platform level, and treated as an operational requirement alongside the event flows themselves.
The Operational Checklist
Each failure mode above has a well-understood fix. Here’s the full list in one place:
The saga orchestration/choreography choice deserves a diagram:
None of these are novel. The operational cost of EDA comes not from missing some clever technique, but from skipping the proven ones.
When EDA Is Worth It — and When It’s Overkill
EDA earns its operational cost when the architecture genuinely needs it. The strongest signal is true fanout — multiple independent consumers reacting to the same event, owned by different teams, scaling on their own terms. An `OrderPlaced` event consumed by inventory, notifications, analytics, and fraud detection is EDA doing what nothing else does as cleanly.
Other strong indicators:
Services that must scale independently based on event volume
Long-running background workflows that should never block a user-facing request thread
Domains that are reactive by nature — fraud detection, audit logging, real-time analytics — where the core model is “something happened and others need to know”
But capability alone is not enough. EDA is worth it when your team has the operational maturity to back it up. That means running DLQs with active monitoring, enforcing schemas through a registry, propagating correlation IDs across every service boundary, and maintaining replay pipelines for recovery.
You are not running event-driven architecture. You are running distributed hope.
Without that operational foundation, EDA gives you the complexity of distributed systems without the reliability guarantees you assumed came with it.
When a Direct Call Beats an Event
If only one consumer ever processes a given event, you do not have event-driven architecture. You have indirect function calling with extra latency and a broker in the middle. A direct call or a job queue does the same work with half the operational surface.
The same applies when the workflow is sequential with no real fanout, or when the team is small enough that async complexity slows delivery more than it helps. If you do not yet have the observability infrastructure to trace an event from producer through every consumer to its final state, EDA will cost you more in debugging time than it saves in decoupling.
Watch for two particularly common traps:
Adopting EDA when the consistency requirements are strict and the team has not designed for eventual consistency or compensating transactions
Using events to decouple two services that share the same data model, the same team, and the same deploy pipeline — those services should probably just be one
The Decision Framework
Before committing to EDA, run through these questions honestly. Each one has a clear signal.
Do multiple independent services need to react to this event? If no, a direct call or job queue is simpler and cheaper to operate.
Does processing need to happen asynchronously without blocking the user? If yes, both EDA and job queues work — EDA wins specifically when fanout exists.
Can your team operate DLQs, schema registries, and correlation IDs today? If no, build that infrastructure first. Adopting EDA without it is taking on debt you cannot service.
Is your domain reactive — things happen and others need to know? Strong signal for EDA. Reactive domains map naturally to event streams.
Are service and team ownership boundaries clean? When consumers are owned by different teams, EDA’s loose coupling is a real advantage. When the same team owns everything, you may be adding indirection without the corresponding benefit.
Are you prepared to design for eventual consistency and write compensating transactions? If no, resolve this before committing to EDA for any transactional workflow.
This decision tree maps each question to a clear recommendation:
If you hit a stop at any point, that is not a permanent no. It is a “not yet.” Fix what is missing, then revisit.
If You Can’t Operate It, You’re Not Ready to Adopt It
Event-driven architecture is powerful — that was true at the start of this article and it’s still true. But the operational surface area is larger than the architecture diagrams suggest, and the patterns that keep it under control — dead letter queues, schema registries, correlation IDs, idempotent consumers, replay pipelines — are not advanced techniques. They are table stakes.
EDA handles more than synchronous request-response can. It also breaks in ways that are harder to detect and harder to fix. The operational patterns in this article exist because the architecture demands them — not as optional improvements, but as the foundation the whole thing rests on.
Choose EDA when the problem genuinely demands it. Build the operational infrastructure before you deploy the first event.