mgarden

Event-Driven Architecture

4 min read

serverless aws messaging claude-curated

Event-driven architecture (EDA) decouples producers from consumers via an asynchronous bus. Producers emit events describing what happened; consumers react however they like, on their own schedule. Compare with command-driven systems where the caller tells the callee what to do and waits for a response.

Core concepts

  • Event — a fact about the past. “OrderPlaced”, “PaymentReceived”. Immutable.
  • Command — an instruction to do something. “PlaceOrder”. Has one logical handler.
  • Producer — emits events without knowing who consumes them.
  • Consumer — subscribes to events of interest.
  • Choreography — services react to each other’s events; no central coordinator. Loosely coupled, harder to reason about end-to-end.
  • Orchestration — a coordinator (e.g. Step Functions) drives the workflow explicitly. Easier to observe, more centralised.

Most real systems mix both — choreography between bounded contexts, orchestration within one.

AWS primitives

EventBridge

A managed event bus. Producers PutEvents; rules pattern-match on the event payload and route to targets (Lambda, SQS, Step Functions, Kinesis, HTTP endpoints, cross-account buses). Features:

  • Schemas — auto-discovery and a registry; generate code bindings.
  • Archives & replay — record events, replay them later for backfills or debugging.
  • Third-party SaaS sources — many SaaS vendors publish events directly into EventBridge.
  • Pipes — point-to-point integrations with optional filtering and enrichment (e.g. DynamoDB stream → enrichment Lambda → Step Functions).

SNS

Pub/sub topics. Publishers send a message; SNS fans it out to all subscribers (SQS, Lambda, HTTPS, email, mobile push). FIFO topics preserve ordering and deduplication within a message group.

SQS

Pull-based queues. A consumer polls for messages, processes them, and deletes them. Two flavours:

  • Standard — at-least-once delivery, best-effort ordering, near-unlimited throughput.
  • FIFO — exactly-once processing within a message group, strict ordering, throughput cap.

Key tunables: visibility timeout (how long a message is hidden from other consumers after receive), long polling (ReceiveMessageWaitTimeSeconds — wait up to 20 s for a message rather than spinning), batch size.

Kinesis Data Streams

Ordered, partitioned, replayable streams. Producers write records to shards; the partition key picks the shard. Consumers checkpoint their position. Records persist for 24 hours by default (extendable to 365 days). Use when ordering within a partition matters and replay is required — clickstreams, IoT telemetry, change-data-capture (CDC).

Fan-out pattern

A canonical EDA pattern: SNS → multiple SQS subscribers. The publisher writes once to SNS; each consumer team owns its own queue with its own visibility timeout, retries, and DLQ. New subscribers can be added without touching the producer.

Producer → SNS topic → SQS A → Service A
                     → SQS B → Service B
                     → SQS C → Service C

EventBridge handles the same pattern with richer routing rules.

Dead-letter queues

A DLQ captures messages that repeatedly fail processing — “poison pills”. Configure on SQS, SNS, Lambda async invocations, EventBridge targets. After N failed delivery attempts the message is moved to the DLQ for manual inspection or automated re-drive.

A DLQ implies a retry budget: how many attempts before giving up. Set maxReceiveCount deliberately — too low and transient blips lose data, too high and a single bad message blocks the queue for hours.

Delivery semantics

  • At-least-once — the default for SNS, SQS Standard, EventBridge, Lambda async. Messages may be delivered more than once on retry.
  • Exactly-once — SQS FIFO and SNS FIFO offer this within a deduplication window (5 minutes by default). True end-to-end exactly-once across services is generally not achievable.

Idempotency

Because most AWS messaging is at-least-once, consumers must be idempotent. Strategies:

  • Idempotency key — include a unique ID per logical event; the consumer records processed IDs (DynamoDB with TTL works well).
  • Conditional writes — PutItem with ConditionExpression: attribute_not_exists(id).
  • Natural idempotency — operations that are inherently safe to repeat (set X=5, upsert by primary key).

The AWS Lambda Powertools library ships an idempotency utility that handles the bookkeeping.

See also

References

Graph View

Backlinks