Event-Driven Systems with NATS JetStream KV Bucket and Consumer Replay

1.3 Idempotency and Deduplication Strategies in Consumers

Event systems often deliver the same message more than once, especially when acknowledgments are delayed or a consumer restarts. Idempotency is the discipline of making processing safe to repeat, while deduplication is the mechanism that prevents repeats from doing harm. In practice, you usually combine both: deduplication reduces work, and idempotency guarantees correctness even when deduplication fails.

The Core Problem and Why It Happens

Consider a consumer that updates an order total when it receives an order.paid event. If the consumer crashes after applying the update but before sending an acknowledgment, the message can be redelivered. Without protection, the update runs twice and the total becomes wrong.

A second failure mode is concurrency: two workers may process the same event if the consumer is configured with multiple deliveries in flight. Even if your system is “mostly reliable,” these edge cases show up under load and during deployments.

Idempotency Models for Consumer Handlers

Idempotency can be achieved at different layers. Choose the layer that matches your data model and performance needs.

Idempotency by State Overwrite

If processing an event results in a deterministic state for a key, you can overwrite rather than apply increments. For example, if user.profile.updated contains the full profile, storing it by userId is naturally idempotent: the same payload produces the same stored value.

Example: a handler writes profiles[userId] = payload.profile. Reprocessing the same event simply writes the same value again.

Idempotency by Conditional Apply

If events represent transitions, use a guard condition. A common pattern is “apply only if the event sequence is newer.” Maintain lastProcessedSequence per entity and ignore events with sequence less than or equal to that value.

Example: when handling invoice.status.changed, store invoice.lastSeq. If event.seq <= lastSeq, return success without changing the invoice.

Idempotency by External Side Effects with Keys

When the handler calls an external system (like sending an email or creating a payment record), make the side effect keyed. Many systems support idempotency keys; if yours does not, you can create a local record of “side effect already done” and check it before calling the external API.

Example: before sending an email, check outbox[emailId] exists. If it does, skip sending.

Deduplication Strategies That Complement Idempotency

Deduplication is about recognizing “same event again.” The trick is choosing a stable identifier and storing enough metadata to decide what to do.

Event Identity and Stable Keys

Use a message identifier that is consistent across redeliveries. Typical choices include eventId generated by the producer, or a tuple like (entityId, sequenceNumber).

Example: producer sets eventId = uuid and consumer stores processedEvents[eventId] = timestamp.

Where to Store Deduplication State

You need a store with the right durability and access pattern.

• In-memory cache: fast but loses state on restart, so it only reduces duplicates during short outages.
• Database table: durable and queryable, but adds write load.
• KV-style state: compact and designed for key lookups, often a good fit for “processed marker” records.

A practical rule: if correctness depends on deduplication, store it durably. If it only saves work, an in-memory cache can be acceptable.

Time-to-Live and Storage Growth

Deduplication markers can grow without bounds. Use TTL when you can bound how long duplicates might arrive. For example, if your system guarantees redelivery within a known window, you can expire markers after that window.

Example: store processedEvents[eventId] with a 60-day TTL, so old markers don’t accumulate forever.

Mind Map: Idempotency and Deduplication

Example: Sequence Guard with Deduplication Marker

Assume each event includes entityId and seq. The consumer uses two checks: a durable “last sequence” guard and an optional “processed marker” for extra safety.

Handler(event):
  key = event.entityId
  lastSeq = state.lastSeq[key]

  if event.seq <= lastSeq:
    return success  // already applied or older duplicate

  // Optional extra dedup marker
  if state.processed[event.eventId] exists:
    state.lastSeq[key] = max(lastSeq, event.seq)
    return success

  applyBusinessChange(event)
  state.lastSeq[key] = event.seq
  state.processed[event.eventId] = now
  return success

This structure prevents double-application even if the consumer crashes at awkward times. If the crash happens after applyBusinessChange but before updating lastSeq, the next delivery will still see lastSeq as old and attempt to apply again; that’s where the optional processed marker helps. If you omit the marker, you still remain correct when applyBusinessChange is itself idempotent or overwrite-based.

Practical Checklist for Consumer Implementations

• Ensure every event has a stable identity (eventId) or a deterministic ordering key (entityId + seq).
• Make the handler safe to run multiple times by using overwrite, conditional apply, or keyed side effects.
• Store deduplication state durably if correctness depends on it.
• Add TTL to processed markers to control storage growth.
• Treat “return success” as part of correctness: once the handler decides the event is already applied, it should ack to stop endless redelivery loops.

1.4 Backpressure and Flow Control in Message Driven Architectures

Message-driven systems move work by sending messages, but the real challenge is deciding what happens when the receiver can’t keep up. Backpressure is the set of techniques that slows down producers or redistributes load so queues don’t grow without bound and latency doesn’t turn into a surprise hobby.

The Problem in Plain Terms

Consider a producer that publishes 10,000 events per second. If consumers can only process 2,000 per second, the system must either:

• Buffer the difference somewhere (memory, disk, broker storage)
• Slow the producer down
• Drop or coalesce messages
• Increase consumer capacity

Flow control is the mechanism that chooses among those options in a controlled way. Without it, you get “works in testing” and “pain in production,” usually because buffering hides the mismatch until it becomes expensive.

Foundational Building Blocks

1) Acknowledgments define “in progress.” If a consumer acknowledges only after processing, the broker can treat unacked messages as still being worked on. That naturally limits how many messages the consumer should receive at once.

2) Concurrency is your local queue. Even if the broker delivers messages quickly, your application might process them with a fixed worker pool. A bounded pool prevents unbounded memory growth.

3) Timeouts turn waiting into decisions. When downstream calls are slow, you need timeouts so the consumer can fail fast, retry safely, or stop pulling.

Broker-Side Flow Control Patterns

Pull-based consumption. With pull consumers, you request messages in batches. If you request 100 messages and only process 100, you’ve created a simple feedback loop: request rate follows processing capacity.

Max in-flight messages. Even with push delivery, you can cap how many messages are delivered without acknowledgments. This prevents a consumer from becoming a “message hoarder.”

Ack timing as a throttle. If you delay acknowledgments until after expensive work, the broker will see more messages as unacked and will slow delivery. That’s useful when processing is the bottleneck, but it can also increase redelivery pressure if processing fails frequently.

Application-Side Flow Control Patterns

Bounded worker pools. Use a fixed number of workers and a bounded channel/queue for tasks. When the queue fills, you can stop accepting new work from the consumer loop.

Coalescing updates. For state-like events (for example, “user profile updated”), you can reduce pressure by keeping only the latest update per key. Instead of processing every intermediate change, you process the final one.

Circuit breakers for downstream dependencies. If a database call starts failing or timing out, continuing to process new messages only increases load. A circuit breaker can temporarily stop pulling or quickly fail handlers, letting the system recover.

A Concrete Example: Bounded Concurrency with Backpressure

Imagine an order service that writes to a database and emits an “order processed” event. The consumer pulls messages in batches of 50, processes them with 10 workers, and acknowledges only after the database write succeeds.

type Job struct{ MsgID string; Payload []byte }

jobs := make(chan Job, 200) // bounded local queue
workerCount := 10

for i := 0; i < workerCount; i++ {
  go func() {
    for j := range jobs {
      err := processOrder(j.Payload) // includes DB write
      if err == nil {
        ack(j.MsgID)
      } else {
        // do not ack; let retry/redelivery handle it
      }
    }
  }()
}

for {
  batch := pull(50) // broker-side pacing
  for _, m := range batch {
    jobs <- Job{MsgID: m.ID, Payload: m.Data} // blocks if full
  }
}

The key behavior is the bounded jobs channel. When workers can’t keep up, the consumer loop blocks, which slows pulling. That’s backpressure implemented with plain mechanics.

Handling Retries Without Making Things Worse

Backpressure and retries interact. If processing fails and messages are redelivered quickly, you can create a retry storm that worsens load. Two practical rules help:

• Retry with jitter so failures don’t synchronize.
• Differentiate transient vs permanent errors so permanent errors don’t keep consuming capacity.

Metrics That Tell You Whether Flow Control Works

Track these together, not separately:

• Processing rate vs publish rate to detect sustained mismatch.
• In-flight count and worker utilization to see whether the bottleneck is broker delivery or application work.
• Ack latency to understand how long messages remain unacked.
• Retry rate and error rate to spot retry storms early.

When these metrics move in the same direction, you can reason about the system. When they move in opposite directions, you likely have a hidden bottleneck, like a downstream dependency that’s stalling while the broker keeps delivering.

Practical Design Checklist

• Cap delivery with max in-flight or pull batch sizing.
• Bound local queues and worker concurrency.
• Acknowledge only after the critical side effect succeeds.
• Use timeouts and circuit breakers for downstream calls.
• Coalesce or dedupe when messages represent intermediate state.
• Monitor lag, in-flight, ack latency, and retry behavior as a set.

1.5 Designing for Observability with Correlation Identifiers and Tracing

Observability is easiest when you can answer three questions quickly: What happened? Where did it happen? Why did it happen? Correlation identifiers and tracing are the tools that make those questions answerable across producers, brokers, and consumers.

Correlation Identifiers as the Thread Through the System

A correlation identifier is a value you generate once at the start of a request or workflow and then carry through every hop. The key idea is that logs, metrics, and traces can all be filtered by the same identifier.

Use cases that benefit immediately:

• A user action triggers multiple events; you need to find all related processing.
• A consumer fails and retries; you need to group attempts under one logical operation.
• A dashboard shows an anomaly; you need to trace it back to the originating request.

A practical convention is to have two identifiers:

• trace_id: identifies the end-to-end trace.
• span_id: identifies a specific step within that trace.

If you only want one value to start, use correlation_id and keep it consistent. Later, you can map it to trace_id without changing your business logic.

Tracing Fundamentals That Match Message Systems

Tracing represents work as a tree of spans. In synchronous calls, spans naturally nest. In event-driven systems, spans often form a graph: one producer span leads to multiple consumer spans.

To keep the model consistent, treat each message handling as a span. When a consumer receives a message, it creates a new span that is a child of the producer’s span context.

A message broker doesn’t automatically know your trace context, so you must propagate it in message metadata (headers). That propagation is the difference between “we have logs” and “we can reconstruct the story.”

Propagation Rules That Prevent Confusing Data

Define rules so every service behaves the same way:

1. Generate a new trace context only at the boundary where the workflow starts (for example, an HTTP request).
2. Propagate trace context in message headers for every publish.
3. Extract trace context in every consumer before starting the handler span.
4. Create a new span per handler attempt, even if the message is redelivered.

Redelivery is where people get tripped up. If you reuse the same span_id across attempts, you’ll lose the ability to see timing differences. Keep correlation stable, but create a fresh span per attempt.

Example: Correlation Headers in a Message Workflow

Below is a minimal pattern: attach correlation and trace context to message headers when publishing, then extract them in the consumer.

// Producer
correlation_id = getOrCreateCorrelationId()
trace_ctx = startOrGetTraceContext()
headers = {
  "correlation_id": correlation_id,
  "trace_id": trace_ctx.traceId,
  "span_id": trace_ctx.parentSpanId
}
publish(subject, payload, headers)

// Consumer
headers = message.headers
correlation_id = headers["correlation_id"]
trace_ctx = extractTraceContext(headers)
span = startSpan("handle_message", parent=trace_ctx)
try {
  handle(payload)
  span.setStatus("ok")
  ack()
} catch (err) {
  span.setStatus("error", err)
  // do not ack to trigger redelivery policy
  throw err
} finally {
  span.end()
}

Notice what’s intentional: correlation_id stays the same across attempts, while each handler attempt creates a new span.

Example: Logging That Stays Useful Under Load

When you log, include the correlation_id and the message identity. A good log line answers “which message” and “which workflow.”

A simple structure:

• timestamp
• service name and version
• correlation_id
• trace_id
• stream name and sequence number (or message id)
• action (received, processed, failed)
• error summary when applicable

This makes it possible to filter by correlation_id and then sort by sequence number to see ordering and retry behavior.

Practical Failure Scenarios and What You Should See

1. Consumer crashes before ack: you should see multiple handler spans with the same correlation_id and different attempt timing.
2. Handler throws a deterministic error: spans should show error status consistently, and logs should group under one correlation_id.
3. Fan-out to multiple consumers: you should see one producer span context leading to multiple consumer spans, all sharing correlation_id.

If those patterns don’t appear, the issue is usually missing header propagation or inconsistent correlation generation at the boundary.

A Simple Checklist for Implementation

• Correlation_id exists at the workflow boundary.
• Every publish includes correlation_id and trace context headers.
• Every consumer extracts headers before starting the handler span.
• Each handler attempt creates a new span.
• Logs include correlation_id, trace_id, and message identity.
• Errors record enough detail to classify the failure without guessing.

2.1 NATS Subjects Wildcards and Routing Patterns

NATS routes messages by subject strings. A subject is a dot-separated path like orders.created. Producers publish to a subject, and consumers subscribe to patterns that match those subjects. The key idea is simple: subject matching happens on the server before your code runs, so good subject design reduces both wasted traffic and wasted CPU.

Subject Tokens and Matching Rules

A subject is split into tokens separated by .. Wildcards let you match multiple subjects without subscribing to each one.

• * matches exactly one token.
• > matches one or more tokens and must appear only at the end.

Example: orders.* matches orders.created and orders.cancelled, but not orders.us.created.

Example: orders.> matches orders.created, orders.us.created, and orders.us.eu.created.

Designing Subject Hierarchies That Don’t Fight You

Start with a stable prefix that groups related traffic, such as app or domain. Then decide which token positions are fixed and which vary.

A common pattern for event streams is:

• domain.entity.action

For example:

• billing.invoice.paid
• billing.invoice.voided

If you later need tenant scoping, insert it as an additional token:

• billing.tenantA.invoice.paid

Now you can choose between two subscription styles:

• Fixed-depth subscriptions using *
• Hierarchical subscriptions using >

Example: Narrow Subscriptions with *

Use * when you know the subject depth and want to match only one varying token.

// Subscribe to all invoice actions at a fixed depth
const sub = nc.subscribe('billing.invoice.*', {
  callback: (msg) => {
    console.log('subject:', msg.subject);
  }
});

// Publishes to matching subjects
nc.publish('billing.invoice.paid', Buffer.from('ok'));
nc.publish('billing.invoice.voided', Buffer.from('ok'));

This matches billing.invoice.paid and billing.invoice.voided. It does not match billing.tenantA.invoice.paid because the token depth differs.

Example: Hierarchical Subscriptions with >

Use > when you want to accept additional nesting after a prefix.

// Subscribe to all invoice events for any tenant
const sub = nc.subscribe('billing.>.invoice.>', {
  callback: (msg) => {
    console.log('subject:', msg.subject);
  }
});

nc.publish('billing.tenantA.invoice.paid', Buffer.from('ok'));
nc.publish('billing.tenantB.invoice.voided', Buffer.from('ok'));

This pattern matches subjects that start with billing., then include any tenant token(s), then invoice, then any action token(s). The > at the end is what makes it flexible; keep it at the end to avoid surprises.

Avoiding Accidental Overlaps

Overlaps happen when a broad subscription catches messages meant for a narrower consumer. For example, billing.> will match everything under billing, including admin events and internal signals. If you have separate processing paths, prefer narrower prefixes like billing.invoice.> or billing.invoice.*.

A practical rule: if two consumers both do work on the same message type, make their subscriptions intentionally overlap. If they should not overlap, make the subject prefixes different early.

Mapping Subjects to Consumer Intent

Think of each subscription as a contract between subject naming and consumer responsibility.

• billing.invoice.* means “I handle invoice events at this exact structure.”
• billing.>.invoice.> means “I handle invoice events regardless of tenant nesting.”

When you keep that contract stable, you can reason about routing without reading logs for every test run.

Quick Checklist for Subject Patterns

• Keep token positions consistent across producers.
• Use * for fixed-depth matching.
• Use > only at the end and only where extra nesting is expected.
• Make prefixes reflect ownership boundaries, not just categories.
• Prefer narrower subscriptions for high-volume consumers.

With these rules, subject matching becomes predictable: your consumers receive exactly the messages they expect, and your system spends less time filtering and more time processing.

2.2 Publish Subscribe Versus Request Reply Patterns

Publish subscribe and request reply solve different problems even though both move messages between services. The key difference is who decides when the interaction ends.

Publish Subscribe Pattern

In publish subscribe, a producer publishes an event to a subject, and any number of consumers receive it. The producer does not wait for responses, so the interaction is naturally decoupled in time.

When it fits

• You want fan-out: multiple services react to the same fact.
• You can tolerate asynchronous processing.
• You model changes as events, not as direct answers.

How it behaves

• Consumers independently acknowledge or handle messages.
• Ordering is per subject stream semantics, not per “conversation.”
• Backlogs can build when consumers are slower than producers.

Concrete example
A checkout service publishes orders.created after persisting an order. A billing service listens to orders.created to create an invoice, and an email service listens to the same subject to send a receipt. Neither service blocks the checkout flow.

Practical best practices

• Use subject naming that reflects meaning: orders.created, orders.cancelled, not msg.1.
• Keep payloads self-describing enough for independent consumers to act without calling back to the producer.
• Design consumers to be idempotent because redelivery can happen when processing fails.

Request Reply Pattern

In request reply, a client sends a request and expects a response. The server processes the request and replies to a return address.

When it fits

• You need a direct answer: “give me the current value,” “validate this command,” or “perform this operation.”
• You want a bounded interaction where the client controls timeouts.
• You can keep the server logic centralized.

How it behaves

• The client typically waits for a response, so latency matters.
• The server can treat each request as a unit of work.
• Retries must be handled carefully to avoid duplicate side effects.

Concrete example
A pricing service receives GetPrice requests from a cart service. It replies with the computed price for a product and region. The cart service can proceed immediately once it has the response.

Practical best practices

• Make request handlers side-effect free when possible, or require explicit idempotency keys.
• Set timeouts on the client and define what “no response” means.
• Keep request payloads small and stable; large payloads increase tail latency.

Choosing Between Them

A useful rule: if the producer’s job is to announce something that others react to, use publish subscribe. If the producer’s job is to answer a question or perform an operation for a specific caller, use request reply.

Decision checklist

• Do you need fan-out? Prefer publish subscribe.
• Does the caller need a result before continuing? Prefer request reply.
• Can the system tolerate asynchronous completion? Prefer publish subscribe.
• Is the interaction naturally a command with a single authoritative responder? Prefer request reply.

Example: Combining Patterns Safely

Many systems use both: request reply for the initial command, publish subscribe for the resulting events. That keeps the “answer” path simple while still letting other services react.

Scenario
1. A client sends CreateOrder via request reply to the order service. 2. The order service persists the order and replies with orderId. 3. The order service publishes orders.created so billing, inventory, and email can react.

This avoids forcing every consumer to participate in the request-response conversation. It also keeps the event payload aligned with what downstream services need.

Example: Idempotency in Both Patterns

• In publish subscribe, idempotency is usually about handling the same event more than once. For example, orders.created might be delivered twice after a consumer crash; the billing consumer should detect that an invoice for orderId already exists.
• In request reply, idempotency is about repeated requests. If a client times out and retries CreateOrder, the order service should return the same orderId when given the same idempotency key.

Summary

Publish subscribe is for broadcasting facts to multiple independent consumers without waiting. Request reply is for direct, bounded interactions where the caller expects a response. The most reliable designs often use request reply to get a result and publish subscribe to distribute the consequences.

2.3 JetStream Concepts Streams Consumers and Durable State

JetStream is NATS’s way of turning plain message passing into something closer to a log with retention and controlled delivery. To reason about it, it helps to separate three ideas: streams store data, consumers define how clients read it, and durable state remembers where a consumer left off.

Streams as Stored Event Logs

A stream is a named container for messages. When you publish to subjects that match the stream’s configuration, JetStream stores those messages according to the stream’s retention policy. Think of a stream as the “where the bytes live” part.

Key stream concepts:

• Subject filters: the subjects that map into the stream. If you publish to orders.created, and the stream is configured to include orders.*, the message lands in that stream.
• Retention policy: controls how long data stays. For example, a “limits by time” policy keeps recent messages, while a “limits by size” policy keeps the most recent data until storage is full.
• Storage type: determines where the data is stored on the server side.

A practical example: you create a stream for orders.* so every order-related event is captured. Later, you can rebuild a view by replaying from the stream rather than relying on the producer to still be around.

Consumers as Delivery Strategies

A consumer is the reader configuration attached to a stream. It decides how messages are delivered, how acknowledgments work, and what starting point to use.

Common consumer knobs:

• Delivery mode: push or pull. Push sends messages to your client; pull lets your client request batches.
• Acknowledgments: with acks enabled, the server tracks which messages are processed and can redeliver if they aren’t acknowledged.
• Start position: where reading begins, such as “from the beginning” or “from the last acknowledged message.”

Example: a billing service might use a push consumer with acknowledgments so it can process events continuously. A backfill tool might use a separate consumer with a different start position to reprocess older events.

Durable State as Remembered Progress

Durable state is the server-side memory of a consumer’s progress. When you create a consumer with a durable name, JetStream can resume delivery after restarts without you manually tracking offsets.

What durable state typically remembers:

• Last acknowledged sequence for the consumer.
• Redelivery behavior when messages are not acknowledged.
• Delivery configuration that affects how the server continues sending.

This is the difference between “I read some messages” and “I am a specific reader that can pause and resume.” If you run the same durable consumer again, it continues from where it left off.

Example: Two Consumers, One Stream

Suppose you have a stream named ORDERS that stores orders.*. You attach two consumers:

1. Live processing consumer

   • Durable name: billing-live
   • Start position: last acknowledged
   • Ack enabled: yes
   • Delivery mode: push

2. Backfill consumer

   • Durable name: billing-backfill
   • Start position: from a specific point or from the beginning
   • Ack enabled: yes
   • Delivery mode: pull

Because the durable names differ, each consumer maintains its own progress. The backfill can run without disturbing the live consumer’s position.

Example: Minimal Consumer Reasoning with Sequence Numbers

JetStream stores messages in a stream with an internal ordering. Consumers receive messages that correspond to those stored positions. When acknowledgments are enabled, the consumer’s durable state advances as you ack messages.

A simple mental model:

• Message arrives with a stream sequence number.
• Your handler processes it.
• You ack it.
• Durable state records that sequence as processed.

If your service crashes after receiving but before acking, the server can redeliver those messages. That behavior is controlled by the consumer’s configuration and the durable state it maintains.

Putting It Together

When you design an event-driven system with JetStream, you typically start by defining streams that capture the right subjects with the right retention. Then you define consumers that match how each service should read and acknowledge data. Finally, you choose durable state when you need reliable resume behavior without external offset tracking. This separation keeps the system understandable: storage rules live in streams, delivery rules live in consumers, and progress memory lives in durable state.

2.4 Acknowledgments Redelivery and Consumer Configuration Basics

JetStream consumers control two things that matter in practice: when a message is considered handled, and what happens when it is not. The basic loop is simple—deliver, process, acknowledge—but the configuration details decide whether your system behaves like a careful librarian or a forgetful courier.

Acknowledgments: What They Mean and Why They Exist

An acknowledgment (ack) is a signal from the consumer to JetStream that the message was processed successfully. Without an ack, JetStream assumes the consumer might have crashed, timed out, or simply failed to finish.

A concrete example: imagine an order service consumes orders.created events and writes an order record to a database. If the consumer crashes after the database write but before sending an ack, JetStream will redeliver the same event later. That is why acking is tied to “safe completion,” not “I started processing.”

In many systems, you can choose between:

• Ack after success: you only ack once the side effects are durable.
• Ack early: you ack before side effects, which reduces redeliveries but risks losing work if the process fails mid-flight.

For most real workloads, ack after success is the sane default.

Redelivery: How JetStream Recovers from Missing Acks

Redelivery is JetStream’s mechanism for re-sending messages that were not acked within a configured window. The key idea is the ack wait: a duration during which JetStream expects an ack. If it doesn’t arrive, the message becomes eligible for redelivery.

A practical scenario: a consumer processes a message, then stalls due to a slow database. If the ack wait is 30 seconds and the database call takes 45 seconds, JetStream will redeliver even though the original processing might eventually finish. This is not a bug; it’s a consequence of timeouts.

To handle this safely, your handler should be idempotent. A common pattern is to store a processed marker keyed by message sequence or event ID, so repeated deliveries do not create duplicate records.

Consumer Configuration: The Knobs That Control Behavior

Consumer configuration determines delivery mode, ack policy, and redelivery timing. The most important knobs are:

• Ack Policy: whether messages require acks. If acks are required, missing acks trigger redelivery.
• Ack Wait: how long JetStream waits for an ack before redelivering.
• Deliver Policy: where the consumer starts reading from (for example, from the beginning or only new messages).
• Replay and Start Position: how you define the initial cursor for the consumer.
• Max Ack Pending: a cap on how many unacked messages the consumer can have in flight.

A useful rule of thumb: set Max Ack Pending to match your processing capacity. If it’s too high, you’ll accumulate work you can’t finish before ack wait expires.

Example: A Safe Handler with Idempotency

Suppose each event includes event_id. The consumer stores event_id in a table with a unique constraint. On each delivery, it attempts the insert; if the insert already exists, it skips side effects and still acks.

On message delivery:
1) Parse event
2) Try insert into processed_events(event_id)
3) If insert succeeds:
   - Write order record
4) If insert fails due to duplicate:
   - Skip side effects
5) Send ack

This makes redelivery harmless: even if the same message arrives twice, the database state remains correct.

Example: Tuning Ack Wait and in Flight Limits

If your handler typically finishes in 200–500 ms but sometimes hits a slow dependency, you can set ack wait to cover the worst observed latency plus a small buffer. Then set max ack pending so the consumer doesn’t start more work than it can complete before the buffer runs out.

A concrete setup might look like this:

• ack wait: 10 seconds
• max ack pending: 50

If each message takes about 1 second on average, 50 in flight means you can keep the pipeline busy without constantly timing out.

Common Configuration Pitfalls

1. Ack wait too short: you’ll see frequent redeliveries and duplicate processing attempts.
2. Ack policy misaligned with side effects: acking before durable writes breaks correctness.
3. No idempotency: redelivery becomes data duplication.
4. Max ack pending too high: you create a backlog of unacked messages that will all time out together.

A consumer that acknowledges only after durable completion, combined with idempotent handlers, turns redelivery from a threat into a predictable recovery mechanism.

2.5 Operational Setup for Local and Containerized Deployments

A good operational setup makes the rest of the book easier: you can reproduce behavior, inspect state, and run the same consumer replay logic in development and in containers. The goal is simple—start NATS with JetStream enabled, create streams and KV buckets deterministically, and run consumers with predictable configuration.

Local Setup That Mirrors Production

Start with a single NATS server for local work, but keep the configuration explicit so you can copy it into containers later.

• Use a dedicated JetStream storage directory so you can wipe state cleanly.
• Enable JetStream and set a consistent server name so logs and tooling are stable.
• Expose the client port and optionally the monitoring port.

A practical local workflow is:

1. Start NATS.
2. Create streams and KV buckets via a small bootstrap script.
3. Run producers and consumers.
4. Stop everything, wipe storage, and re-run bootstrap to confirm idempotent setup.

Example: Minimal NATS Server Configuration

# nats-server.conf
server_name: local-jetstream
jetstream: {}
store_dir: ./jetstream
max_payload: 4MB
port: 4222

Run the server with that config, then verify JetStream is reachable by checking server logs for JetStream initialization.

Containerized Setup with Deterministic Networking

In containers, the main operational differences are hostnames, persistence, and startup ordering.

• Use a stable service name like nats for the hostname inside the Docker network.
• Persist JetStream storage with a volume so consumer replay state survives restarts.
• Add a healthcheck so your bootstrap job waits for the server.

Example: Docker Compose for NATS and Bootstrap

services:
  nats:
    image: nats:2.10
    command: ["-c", "/etc/nats/nats-server.conf"]
    ports: ["4222:4222"]
    volumes:
      - natsdata:/data
      - ./nats-server.conf:/etc/nats/nats-server.conf:ro
    healthcheck:
      test: ["CMD", "nats", "ping", "-s", "nats://localhost:4222"]
      interval: 2s
      timeout: 1s
      retries: 10

volumes:
  natsdata:

If you bootstrap streams on every container start, you should make the bootstrap idempotent by using the same stream and bucket names and by tolerating “already exists” responses.

Bootstrap Scripts for Streams and KV Buckets

Operational reliability comes from repeatable creation. Treat stream and KV definitions as code.

• Keep subject patterns and retention policies in one place.
• Use consistent naming conventions for streams and buckets.
• Set replication and storage options intentionally, even if you keep them simple locally.

Example: Idempotent Stream and KV Creation

# bootstrap.sh
set -e
nats --server nats://nats:4222 stream add EVENTS \
  --subjects 'events.*' \
  --retention limits \
  --max-age 72h || true

nats --server nats://nats:4222 kv add CONFIG \
  --history 10 || true

The || true is not a blanket permission slip; it’s a deliberate choice to allow re-runs during development.

Verification Steps That Catch Real Mistakes

After setup, verify behavior rather than just connectivity.

1. Publish a small set of test events to events.* and confirm the stream receives them.
2. Create a KV entry, update it, then confirm the KV history depth matches expectations.
3. Start a consumer with a durable name, process a few messages, and restart it to confirm the consumer resumes correctly.
4. Run a controlled replay by resetting consumer acknowledgment state and observing that the handler sees the same sequence range.

For a concrete test dataset, use a fixed run date like 2026-03-15 in your event payloads so you can visually confirm which messages were produced during a specific run without relying on timestamps from the environment.

Common Operational Pitfalls

• Using different stream or bucket names across environments, which breaks replay assumptions.
• Forgetting persistence in containers, which makes “replay” look like “start over.”
• Letting bootstrap scripts drift from code, which causes subtle subject mismatches.
• Running consumers before streams exist, leading to confusing “no messages” behavior that is actually a setup issue.

A clean operational setup is mostly boring configuration work, but it pays off immediately when you need to debug consumer replay and KV-driven state transitions.

3.1 KV Bucket Semantics Key Value Updates and Sequence Ordering

KV buckets store state as a sequence of updates per key. Each update is assigned a monotonically increasing sequence number within the bucket, which gives you a shared timeline for reasoning about ordering and replay.

What “Key Value” Means in KV Buckets

A KV bucket is not a log of arbitrary events; it is a map from keys to the latest value. When you write to a key, you create a new version of that key’s value. Reads can ask for the latest version, or they can request a specific version by sequence number.

A practical way to think about it: the bucket is a ledger, but your application usually cares about the current entry for each key. Sequence numbers are the ledger’s page numbers.

How Updates Are Ordered

Every write to any key increments the bucket’s global sequence counter. That means:

• Updates across different keys are totally ordered by sequence number.
• Updates to the same key are also ordered, and the latest version is the one with the highest sequence number for that key.

This global ordering matters when you need to correlate state changes across keys. For example, if you store workflow progress under keys like wf:123:step and wf:123:status, the sequence numbers let you determine which update happened first even if they were written by different components.

Versioning and What You Can Read

KV buckets expose multiple read styles:

• Latest value: “Give me the current value for key k.”
• Versioned value: “Give me the value for key k at sequence s.”
• Watches: “Notify me when key k changes, starting from sequence s.”

The key point is that sequence numbers are the bridge between “what changed” and “when it changed.” Without sequence numbers, you can only guess ordering from timestamps, which are often inconsistent across machines.

Sequence Numbers as a Consistency Tool

Sequence ordering lets you build deterministic workflows even under concurrency. Consider two writers updating different keys:

• Writer A updates cart:7:items.
• Writer B updates cart:7:status.

If you later read both keys and compare their sequence numbers, you can decide whether the status update was based on the items update. If status has a higher sequence number than items, you know the status write occurred after the items write in the bucket’s timeline.

Example: Tracking Two Related Keys

Imagine a checkout system that stores:

• order:100:items as a JSON list
• order:100:payment as a status string

Writers update them in separate steps. Your consumer can store the last processed bucket sequence lastSeq. When it receives a change notification, it can:

1. Read order:100:items at the sequence it observed.
2. Read order:100:payment at the latest sequence not exceeding the observed sequence.

That second step prevents mixing a newer payment value with older items. The consumer’s logic becomes: “Use a consistent snapshot boundary defined by sequence.”

Example: Building a Simple Snapshot Boundary

Below is a minimal pattern for using a sequence boundary to read a coherent view.

lastSeq = loadCheckpoint()

onKvUpdate(event):
  boundary = event.sequence

  items = kv.getAtSequence("order:100:items", boundary)
  payment = kv.getAtSequence("order:100:payment", boundary)

  process(items, payment)

  lastSeq = boundary
  saveCheckpoint(lastSeq)

This works because sequence numbers define a single ordering for the bucket. You are not relying on wall-clock time, and you are not assuming that updates arrive in the same order they were written.

Advanced Detail: Handling Concurrent Writers Safely

When multiple writers update the same key, the bucket still provides a single sequence order. Your application should treat each new version as authoritative and avoid “merge by guessing.” If you need merges, do it at the application level using the values you read at the sequence boundary.

If you need to ensure that a multi-key update is processed atomically, KV buckets don’t magically bundle keys into one transaction. Instead, you use sequence boundaries to process a consistent set of reads, and you design your keys so that the consumer can reconstruct the intended state from those reads.

Example: Detecting Out-of-Order Assumptions

Suppose a consumer assumes that payment is updated after items. With sequence numbers, you can verify that assumption:

• If payment’s sequence is lower than items’s sequence for the same order, then the consumer should not treat the payment as derived from the items update.

Rather than failing silently, you can route the case to a “wait for next update” path by keeping the checkpoint at the observed sequence and reprocessing when the bucket advances.

Summary of Key Semantics

KV bucket updates create new versions per key, and every write contributes to a global, monotonically increasing sequence. By reading values at or before a chosen sequence boundary, you can reason about ordering across keys and process state consistently even when writers run concurrently.

3.2 Choosing Key Design Patterns for Multi Tenant and Partitioned Data

Multi-tenant KV buckets work best when your key design makes ownership and partitioning obvious, stable, and cheap to query. In JetStream KV, each update is stored under a single key, so your key format becomes your primary indexing strategy. A good key pattern prevents accidental cross-tenant reads, keeps operational tooling simple, and makes replay and cleanup predictable.

Key Design Goals for Multi Tenant KV

Start with three practical goals:

1. Tenant isolation by construction. If a bug publishes under the wrong tenant prefix, the key format should make it hard to overwrite or read other tenants’ state.
2. Efficient retrieval by prefix. KV reads and watches typically operate by key or key prefix patterns. Even when your client fetches specific keys, prefix structure helps you reason about what exists.
3. Stable evolution. Keys should remain valid across versions. If you change the payload schema, you should not need to rename keys.

A simple rule: treat the key like a URL path. If you wouldn’t want to break it in production, don’t make it fragile.

Tenant Prefix Patterns

Use a tenant prefix that is consistent and unambiguous. Common options:

• Opaque tenant ID prefix: tenant:{tenantId}/... where tenantId is a stable identifier.
• Human-friendly prefix: tenant:{slug}/... where slug is readable but must be immutable.

Prefer opaque IDs. Slugs invite the “we renamed the customer” problem, which turns into key migration work.

Example key roots:

• tenant:acme/
• tenant:globex/

Partitioning Patterns Beyond Tenant

Once you have tenant isolation, partition the rest of the state by what you need to manage independently. Typical partition dimensions:

• Entity type: user, session, workflow, featureFlag
• Entity ID: user:{userId}
• Environment or region: env:prod, region:us-east

A practical hierarchy is:

tenant:{tenantId}/env:{env}/entity:{entityType}/{entityId}

This keeps the “who owns it” part first, then the “where it lives,” then the “what it is,” then the “which instance.”

Versioning and Schema Compatibility

KV stores the latest value per key, so schema changes are your responsibility. Two safe approaches:

• Embed schema version in the value, not the key. The key stays stable; consumers interpret the value based on a schemaVersion field.
• Embed schema version in the key only when you must keep multiple representations. For example, if you need to run two decoders side by side during a migration.

If you do version in the key, keep it near the value identity, not at the tenant root. That avoids multiplying tenant prefixes.

Handling Deletes and Tombstones

KV updates can represent deletion via tombstones depending on how you manage them. Your key pattern should make deletion unambiguous:

• Use the same key for the entity.
• Publish a tombstone value under that key.

Avoid inventing separate “deleted” keys like .../deleted/{id} unless you truly need historical deletion markers. Otherwise, you’ll create extra keys to clean up and extra logic to reason about.

Example: Session State Keys

Suppose you store session progress for a workflow. You want tenant isolation and quick lookups by session ID.

Key format:

• tenant:{tenantId}/env:prod/entity:session/{sessionId}

Value fields might include:

• schemaVersion
• workflowId
• step
• updatedAt

When you replay events to rebuild session state, you update the same key. That means the replay is idempotent at the storage layer as long as your consumer logic is consistent.

Example: Feature Flags as KV Entries

Feature flags are naturally key-value. Partition by tenant and flag name:

• tenant:{tenantId}/env:prod/entity:featureFlag/{flagName}

If you need per-user overrides, add another segment:

• tenant:{tenantId}/env:prod/entity:featureFlag/user:{userId}/{flagName}

This avoids mixing global and user-specific flags under the same key namespace.

Example: Partitioned Workflow State

For workflow state, you often need to separate workflow instances from their definitions.

• Definition metadata: tenant:{tenantId}/env:prod/entity:workflowDef/{workflowType}
• Instance state: tenant:{tenantId}/env:prod/entity:workflowInst/{workflowId}

That separation keeps updates to instance state from overwriting definition metadata, even if both are produced by the same service.

Common Pitfalls to Avoid

• Putting mutable fields early. If you expect the value to change, don’t place it in the key.
• Using free-form strings without normalization. If flagName can contain slashes or inconsistent casing, normalize it before building keys.
• Over-partitioning. More segments can mean more keys and more complexity. Partition only along dimensions you actually query or manage independently.

A key pattern is a contract. When it’s clear and consistent, both your consumers and your operators can reason about state without guesswork.

3.3 Handling Deletes Tombstones and Versioned State

JetStream KV buckets treat every key as a small, versioned timeline. A normal update stores a new value with an increasing sequence. A delete is represented as a tombstone entry, which means “this key used to exist, but it doesn’t anymore as of this version.” The practical trick is to handle tombstones as first-class events rather than as an afterthought.

Versioned State Basics That Matter for Deletes

In a KV bucket, each key has multiple versions. When you read the latest value, you’re really asking for the most recent version that is not deleted. When you watch or iterate, you’ll see every version, including tombstones. That difference is why “latest read” and “event stream of changes” can disagree if you don’t account for deletes.

A tombstone typically carries metadata indicating deletion, while the value payload may be empty or omitted. Your application should not assume the payload is meaningful when the version indicates deletion.

Tombstones as Data, Not Errors

Treat tombstones as a normal state transition:

• Created or Updated: key exists with a value.
• Deleted: key does not exist as of that version.
• Recreated: a later update adds a new value again.

This matters for correctness. If you ignore tombstones, you can keep stale state in memory even though the bucket says the key is gone.

Designing Your In-Memory Representation

A common pattern is to maintain a map from key to a struct containing both the value and the last seen version.

• On update: replace value and record the version.
• On tombstone: remove the key from the map, or mark it as deleted with the version.

Recording the version helps when processing is concurrent or when you replay from an earlier point. It also prevents “last writer wins” bugs where an older event arrives after a newer one.

Example: Update Then Delete Then Recreate

Imagine a feature flag key: tenantA:betaCheckout.

1. Version 10 stores true.
2. Version 11 stores a tombstone.
3. Version 12 stores false.

If your handler removes the key on tombstone and then sets it on version 12, your in-memory state ends up correct. If you only handle updates and skip tombstones, you’ll incorrectly keep true forever.

Example: Watch Handler with Version Checks

Below is a minimal handler sketch. It assumes you receive a record that includes a deletion indicator and a version/sequence.

type State struct {
  Value   []byte
  Version uint64
}

func applyKVEvent(store map[string]State, key string, rec KVRecord) {
  if rec.IsDelete {
    delete(store, key)
    return
  }
  store[key] = State{Value: rec.Value, Version: rec.Version}
}

If you need to guard against out-of-order delivery, add a version check before applying:

func applyKVEvent(store map[string]State, key string, rec KVRecord) {
  cur, ok := store[key]
  if ok && rec.Version < cur.Version {
    return
  }
  if rec.IsDelete {
    delete(store, key)
    return
  }
  store[key] = State{Value: rec.Value, Version: rec.Version}
}

Advanced Details Without the Headaches

1. Latest Reads vs Watches: A “get latest” call should return no value for a deleted key, but a watch will still show the tombstone. Your code should reflect the mode it’s in.
2. Replay Safety: During replay, tombstones are what make the reconstructed state match the bucket. If you rebuild state from scratch, you must process deletes the same way you process updates.
3. Idempotent Handlers: If your handler can be called multiple times for the same version, version checks make it safe. If it can’t, ensure your consumer configuration and ack strategy avoid duplicates, or make the handler tolerant.

Practical Checklist for Deletes

• Confirm your handler distinguishes delete records from value records.
• Remove or mark the key on tombstone.
• Store and compare versions when ordering isn’t guaranteed.
• Verify behavior with a sequence like update → delete → update.

When deletes are treated as real state transitions, versioned KV becomes predictable: the bucket’s timeline is the truth, and your application simply follows it.

3.4 Reading KV Data with Watches and Consistent Retrieval Patterns

KV buckets store the latest value per key, plus a monotonically increasing sequence number for each update. Reading “the latest” is straightforward, but reading “the latest consistently” across multiple keys requires a retrieval pattern that respects how updates move through the bucket.

The Mental Model for KV Reads

Think of a KV bucket as two layers:

1. Key space: key -> latest value.
2. Update timeline: each write advances a sequence number.

A watch lets you observe changes on the timeline, while a point-in-time read lets you fetch a snapshot. The consistent retrieval patterns below combine both ideas so your application doesn’t mix values from different moments.

Pattern 1: Read-Then-Watch for Safe Startup

Use this when you need an initial state and then continuous updates.

Step A: Read initial values. Fetch the keys you care about (or a bounded set) using point reads. Record the sequence number you observed as your “starting boundary.”

Step B: Start a watch from that boundary. Begin watching updates starting at or after the recorded sequence. Apply updates in order.

Why it works: any update that happened before your boundary is already reflected in your initial reads; updates after the boundary arrive through the watch.

Example: A service maintains a local cache of feature flags stored in a KV bucket.

• At startup, it reads flags/ui and flags/billing.
• It notes the sequence number returned by the reads.
• It starts a watch from that sequence and updates the cache as changes arrive.

If an update occurs between the two steps, the watch starting boundary ensures the cache converges without needing to re-read everything.

Pattern 2: Watch-Then-Read for Fast Bootstraps

Use this when you want to start processing immediately and tolerate a short reconciliation window.

Step A: Start the watch first. Begin watching from the bucket’s current sequence (or a known boundary).

Step B: Read missing keys. As you receive watch updates, populate your cache. For keys that never appear in the watch stream during startup, perform targeted reads.

Why it works: watch updates are ordered by sequence, so your local state will reflect the timeline. Targeted reads fill gaps without forcing a full snapshot.

Example: A worker tracks active sessions in KV. It starts watching session keys and updates its in-memory map. After the watch is running, it reads only the session keys required to resume a specific workflow.

Pattern 3: Consistent Multi-Key Snapshot via Sequence Boundary

When you need values across multiple keys to represent the same moment, you must anchor reads to a shared sequence boundary.

Approach:

1. Obtain a sequence boundary S.
2. Read each key at version S (or the closest version at or before S, depending on the API semantics).
3. Apply the results as one coherent snapshot.

Why it works: all keys are evaluated against the same point on the update timeline, so you avoid mixing “old” and “new” values.

Example: A pricing service reads currency/rate and discount/rules and uses them together to compute a quote. If you read them at different times, you can produce quotes that don’t match what the system intended. Anchoring both reads to the same sequence boundary keeps the computation internally consistent.

Pattern 4: Rebuild a Projection from Watch History

Sometimes you don’t just want “latest values,” you want a derived view (projection) that you can rebuild deterministically.

Method:

• Maintain a local projection store.
• Start a watch from the last processed sequence number.
• For each update, apply it to the projection using deterministic rules.
• Persist the last processed sequence number after successful application.

Example: A reporting component builds a “current inventory per SKU” projection from KV updates. If the process restarts, it resumes from the last processed sequence and re-applies updates in order, producing the same final projection.

Handling Deletes and Tombstones

KV deletes typically appear as tombstones in the watch stream. Your projection logic should treat a tombstone as “remove the key from local state,” not as “set value to null and keep it.”

Example: If user/123 is deleted, the cache should remove the user entry so downstream reads fail fast rather than returning stale data.

Practical Consistency Rules

• Single key: a point read is usually enough.
• Multiple keys: use a shared sequence boundary for snapshot consistency.
• Continuous updates: use watches anchored to a boundary and apply updates in sequence order.
• Recovery: persist the last processed sequence number and resume from there.

If you follow these rules, your KV reads behave like a controlled timeline rather than a grab bag of “whatever was latest when we asked.”

3.5 Practical KV Bucket Examples for Configuration and Feature Flags

KV buckets are a good fit when you want shared, versioned state with simple semantics: each key holds the latest value, and updates are ordered by sequence. The trick is choosing key structure and update rules so consumers can read efficiently and safely.

Configuration Example with Prefix Watches

Suppose a billing service needs a currency setting per tenant. Store it in a KV bucket named config. Use a key format that makes prefix reads natural:

• Key: tenantA:billing:currency
• Value: JSON like { "value": "USD", "updatedBy": "ops", "updatedAt": "2024-03-20T10:15:00Z" }

A consumer can watch tenantA:billing: to react to any configuration change for that tenant. This avoids scanning unrelated keys and keeps the update logic straightforward.

{
  "key": "tenantA:billing:currency",
  "value": {
    "value": "USD",
    "updatedBy": "ops",
    "updatedAt": "2024-03-20T10:15:00Z"
  }
}

When updating, replace the entire value rather than patching fields. That keeps the consumer’s state model simple: “latest value wins.” If you need partial updates, encode them as separate keys so consumers can compose them deterministically.

Feature Flag Example with Deterministic Evaluation

For feature flags, store a small decision payload per flag. Use keys that separate environment and service:

• Key: prod:checkout:flag_free_shipping
• Value: { "enabled": true, "rollout": 100, "updatedAt": "2024-03-20T10:20:00Z" }

Here rollout is an integer from 0 to 100. Consumers can evaluate a request by hashing a stable identifier (like userId) into a 0–99 bucket and comparing it to rollout. The KV read gives you the latest policy; the hash gives you deterministic behavior without extra coordination.

{
  "key": "prod:checkout:flag_free_shipping",
  "value": {
    "enabled": true,
    "rollout": 25,
    "updatedAt": "2024-03-20T10:20:00Z"
  }
}

A practical pattern is to cache the flag value briefly in memory, but still treat KV as the source of truth. If you watch the relevant prefix (for example prod:checkout:), you can update the cache immediately on changes and avoid frequent reads.

Example: Minimal Consumer Logic for KV Reads and Watches

The following pseudocode shows the core flow: load initial values, then update on watch events. It also demonstrates how to handle missing keys without crashing.

// Pseudocode
cfgKey := tenant + ":billing:currency"
flagKey := env + ":checkout:flag_free_shipping"

currency := kv.Get(cfgKey) // latest
if currency == nil { currency = "USD" }

flag := kv.Get(flagKey)
if flag == nil { flag = {enabled:false, rollout:0} }

watchPrefix := env + ":checkout:"
for evt := range kv.Watch(watchPrefix) {
  if evt.Key == flagKey { flag = decode(evt.Value) }
}

This approach keeps the consumer deterministic: it always uses the most recently observed KV value, and it never depends on message ordering from unrelated streams.

Key Design Rules That Prevent Pain Later

1. Choose prefixes that match your watch needs. If you will watch by environment and service, bake those into the key.
2. Keep values self-describing. Include updatedAt and the semantic fields you need for evaluation.
3. Treat deletes as meaningful. If a key is deleted, decide whether that means “use default” or “disable behavior.” Encode that decision in the consumer.
4. Avoid mixing unrelated domains in one prefix. It makes watches noisy and increases the chance of accidental coupling.

Example: Handling Deletes for Feature Flags

If a flag key is deleted, a consumer should fall back to a safe default. For feature flags, the safe default is usually disabled.

{
  "onDelete": "disable",
  "default": { "enabled": false, "rollout": 0 }
}

With that rule in place, operational actions like removing a flag entry behave predictably: the system returns to a conservative baseline rather than guessing.

4.1 Mapping Domain Events to Subject Naming Conventions

Subject names are the “address” of your events. Good conventions make it easy to route messages, reason about ownership, and keep consumers from accidentally subscribing to the wrong stream. The goal is not to be clever; it’s to be consistent enough that a new teammate can predict the subject for a new event.

Start with Domain Vocabulary and Event Intent

Before choosing separators or wildcards, list the domain concepts that will appear in events: bounded context (like billing), entity (like invoice), action (like paid), and sometimes scope (like region or tenant). Then decide what the event means to consumers.

A practical rule: the subject should describe what happened, not how it was produced. For example, “invoice.paid” communicates the outcome. “invoice.payment-processed” might be accurate, but it ties the event to an internal implementation detail.

Choose a Subject Shape That Scales

A common shape is:

• <domain>.<entity>.<event>
• optionally extended to <domain>.<entity>.<event>.<scope>

Keep segments stable. If you later change the meaning of a segment, you’ll either break consumers or carry compatibility baggage.

Example mapping:

• Domain: billing
• Entity: invoice
• Event: paid
• Scope: tenant (optional)

So you get billing.invoice.paid or billing.invoice.paid.tenant.<tenantId>.

Define Naming Rules for Each Segment

Use the same casing and separators everywhere. A simple, readable convention is lowercase with dots as separators.

1. Domain segment: bounded context name, like billing, orders, identity.
2. Entity segment: nouns, like invoice, order, user.
3. Event segment: past-tense outcomes or clear verbs, like created, updated, paid, canceled.
4. Scope segment: only when you truly need routing differences.

Avoid mixing styles like invoicePaid in one place and invoice.paid in another. Consistency beats expressiveness.

Decide How Much to Put in the Subject

Subjects are for routing; payloads are for facts. Put stable routing keys in the subject, and keep variable details in the payload.

Good in subject:

• tenant routing when you need isolation: billing.invoice.paid.tenant.<id>
• entity type routing: billing.invoice.*

Avoid in subject:

• customer email, full order IDs, or long text fields

If you include high-cardinality values in subjects, you’ll create a subscription explosion and make operational debugging harder.

Use Wildcards Intentionally

NATS supports wildcards, so you can design subjects that make safe subscriptions easy.

• Use domain.entity.* for “all events for an entity.”
• Use domain.*.created for “all created events in a domain.”
• Avoid patterns that are too broad, like *.*.*, unless you’re building a generic audit consumer.

A useful practice: write down the top three consumer subscription patterns you expect, then shape subjects so those patterns are precise.

Keep Versioning Out of the Subject Unless You Must

If you need schema evolution, prefer versioning inside the payload (for example, schemaVersion) so routing stays stable. Only version the subject when you must separate incompatible semantics.

Example: From Event List to Concrete Subjects

Suppose your billing context emits these events:

• Invoice created
• Invoice paid
• Invoice canceled

Using billing.invoice.<event> you get:

• billing.invoice.created
• billing.invoice.paid
• billing.invoice.canceled

Now add a consumer that maintains a read model for invoices. It subscribes to billing.invoice.* and updates state based on the payload’s eventType and schemaVersion.

If you also need tenant isolation, you can route by tenant:

• billing.invoice.paid.tenant.<tenantId>

Then the read model consumer subscribes to billing.invoice.*.tenant.<tenantId> for its tenant, while an admin audit consumer can subscribe to billing.invoice.* to see all tenants.

Example: Preventing Accidental Misrouting

A common mistake is using ambiguous event names like processed. Two different workflows might both emit processed, and consumers can’t tell which one they’re handling.

Prefer outcome-specific names:

• billing.invoice.paid instead of billing.invoice.processed
• orders.order.shipped instead of orders.order.processed

When the subject communicates the outcome, consumers can write simpler logic and fewer guard checks.

Practical Checklist for Consistency

• Every event has a single, stable subject pattern.
• Subjects use lowercase and dot separators.
• Variable identifiers live in the payload, not the subject.
• Wildcard subscriptions match real consumer needs.
• Event names describe outcomes, not internal steps.

If you can explain the subject format to a teammate in one minute and they can predict the subject for a new event, you’ve chosen a naming convention that will stay usable.

4.2 Stream Configuration Retention Policies and Storage Choices

Retention policies decide what JetStream keeps and for how long; storage choices decide how it keeps it. Together they shape disk usage, replay behavior, and how quickly consumers can catch up after downtime.

Retention Policies That Control What Stays

Start by mapping your requirement to one of the retention modes.

• Limits by time: Keep messages for a fixed duration. This is a good fit for event logs where “older than X” is meaningless, such as telemetry windows or short-lived audit trails.
• Limits by count: Keep only the most recent N messages. This works well for “latest state history,” like recent status changes where you only need a bounded backlog.
• Limits by size: Keep messages until storage reaches a cap, then evict older data. This is useful when you want a hard ceiling on disk usage and can tolerate losing the oldest events.
• Work-queue style: Keep messages until they are acknowledged by consumers. This is the closest match to “process each task once,” but it requires careful consumer ack behavior to avoid stuck backlogs.

A practical way to choose is to ask: “If a consumer is down for 30 minutes, should it replay everything it missed?” If yes, pick a time-based policy with a duration that covers the outage plus some buffer. If no, pick a count or size policy that bounds replay.

Storage Choices That Control How It’s Stored

JetStream storage choices affect performance and operational behavior.

• File-based storage: Data is persisted on disk. It’s the default choice when you want durability and predictable replay. It also makes disk sizing a first-class task.
• Memory-based storage: Data lives in memory. It can reduce disk I/O and improve latency for small workloads, but it’s more sensitive to memory pressure and restarts.

A simple rule: if you need replay after restarts or you expect consumers to lag, prefer file-based storage. If you’re building a short-lived pipeline where losing old events is acceptable, memory-based storage can be reasonable.

How Retention and Storage Interact

Retention determines the eviction rule; storage determines the cost of holding data.

• With time retention, the system continuously ages out old messages, so disk usage tracks your duration and message rate.
• With count retention, disk usage tracks your N and average message size, which is often easier to reason about when payload sizes are stable.
• With size retention, disk usage is capped, but replay length becomes variable when payload sizes fluctuate.
• With ack-based retention, disk usage depends on consumer progress; a misconfigured consumer that never acks can cause unbounded growth until you intervene.

If you’re unsure, start with a conservative policy that matches your operational tolerance for consumer downtime, then validate with a small load test.

Example: Choosing Policies for Three Workloads

Example 1: Short-lived telemetry

• Requirement: Consumers need the last 10 minutes of data; older points are irrelevant.
• Choice: Time retention for 10 minutes (plus a small buffer), file storage for durability.
• Result: Replay after a brief outage covers the window you care about.

Example 2: Status updates with bounded history

• Requirement: Keep the latest 500 status changes per stream.
• Choice: Count retention of 500, file storage if you want replay after restarts.
• Result: Backlog is predictable even if message rate varies.

Example 3: Task processing queue

• Requirement: Each task must be processed; redelivery should happen until ack.
• Choice: Work-queue style retention, file storage to survive restarts.
• Result: If a worker crashes, unacked tasks remain available.

Example: A Concrete Configuration Sketch

Below is a conceptual configuration outline. The exact field names depend on your client library, but the intent is consistent.

Stream: orders-events
Retention: Time
Retention Duration: 2h
Storage: File
Max Messages: optional
Max Bytes: optional

Consumer: replayable-worker
Ack Policy: explicit
Replay: from earliest on demand

This setup keeps a two-hour event window for replayable recovery while preventing the stream from growing without bound.

Operational Checks That Prevent Surprises

Before you rely on replay, verify three things.

1. Backlog expectations: Estimate message rate × retention window to confirm storage sizing.
2. Consumer ack discipline: For ack-based retention, ensure consumers explicitly ack after successful processing.
3. Payload size variability: If payloads can spike, prefer count or time retention over size retention when you need stable replay length.

A stream that matches your operational reality is usually better than one that matches a theoretical ideal. Retention and storage are the knobs that make that reality concrete.

4.3 Consumer Types and Their Fit for Different Workloads

JetStream gives you multiple consumer styles, and the “right” one depends on how you want to read, how you want to recover, and how much control you need over timing. The key idea is simple: a consumer is the component that decides which messages you receive, when you receive them, and how you acknowledge them.

Push Consumers for Continuous Work

A push consumer sends messages to your application as they become available. This is a good fit when you want steady processing and you can keep up with the incoming rate.

When it fits well

• You have a long-running worker service.
• You want low-latency delivery without polling.
• You can implement backpressure by limiting concurrency and acknowledging promptly.

Practical example
Imagine an order service that emits orders.created events. A push consumer can feed a fulfillment worker that updates shipment status in near real time. If the worker processes 200 messages concurrently and acks after each update, JetStream can redeliver only the ones that were not acked.

Operational nuance
If your handler sometimes takes longer than usual, push delivery can accumulate. The fix is not “make it faster”; it’s to cap concurrency, use timeouts, and ack only after the side effects are durable.

Pull Consumers for Controlled Fetching

A pull consumer requires your application to request messages. This shifts control from JetStream to your code, which is useful when you need explicit pacing.

When it fits well

• You want to batch work for efficiency.
• You have variable load and want to avoid building queues in your process.
• You need to coordinate reads with other resources like databases.

Practical example
A reporting service reads orders.* events and writes aggregates every minute. A pull consumer lets the service fetch a bounded number of messages, process them, and then sleep. This avoids a constant stream of work when the reporting job is not running.

Operational nuance
Pull consumers make it easier to implement “work permits.” For example, you can fetch only when your database connection pool has capacity.

Shared Consumers for Horizontal Scaling

Shared consumers allow multiple instances to share the same subscription workload. Each message is delivered to one member, which makes scaling straightforward.

When it fits well

• You run multiple replicas of the same worker.
• You want parallelism without duplicating work.
• You can tolerate that ordering is not guaranteed across instances.

Practical example
A fraud-check worker processes payments.* events. With a shared consumer, you can run 10 replicas and let JetStream distribute messages. If one replica is slow, others keep working.

Operational nuance
If you require strict per-entity ordering, shared consumers can still work, but you must partition by key using subject design and separate consumers per partition.

Durable Consumers for Replay and Recovery

Durable consumers remember their position. That matters when you need to restart processing without losing track, or when you want to replay from a known point.

When it fits well

• You need deterministic recovery after outages.
• You plan to reprocess events after a bug fix.
• You want consistent state rebuild behavior.

Practical example
A user-profile projection consumes users.updated events and stores the result in a KV bucket. If the projection service restarts, a durable consumer can resume from the last acknowledged sequence, preventing gaps.

Operational nuance
Durability pairs naturally with idempotent handlers. If a message is redelivered, your handler should not corrupt state.

Exclusive Consumers for Single-Writer Semantics

Exclusive consumers are intended for one active consumer group at a time. This is useful when you want a single writer to a resource.

When it fits well

• You have a single-writer database table or a non-concurrent state machine.
• You want to avoid coordination between replicas.
• You prefer correctness over throughput.

Practical example
A workflow engine updates a workflow state document and must ensure only one processor mutates it. An exclusive consumer ensures only one instance handles the stream.

Operational nuance
If you later scale, you typically move to partitioning plus shared consumers rather than trying to force exclusivity into concurrency.

Mind Map of Consumer Choice

Putting It Together with a Concrete Scenario

Suppose you maintain a “session progress” projection keyed by sessionId. You want replay when a handler bug is fixed, and you want per-session ordering.

• Use a durable consumer so you can resume and replay.
• Use pull if you need to fetch bounded batches and coordinate with database load.
• Partition subjects by sessionId so each session’s events go to a consistent consumer group.
• Use shared consumers within each partition for throughput, while keeping ordering per session by design.

That combination avoids the common trap of scaling first and then discovering that ordering and recovery requirements were quietly doing the heavy lifting.

4.4 Schema and Payload Conventions for Interoperable Messages

Interoperable messages work when producers and consumers agree on structure, meaning, and evolution rules. In practice, that means you standardize three layers: the envelope (how the message is wrapped), the payload (what the message contains), and the schema lifecycle (how changes stay compatible). JetStream will happily store bytes, but your system needs conventions so those bytes still make sense months later.

Message Envelope Conventions

Use a small, consistent envelope so consumers can route, validate, and debug without inspecting every payload field.

• eventType: a stable identifier like order.created.
• schemaVersion: an integer you control, not a random string from a library.
• producerId: helps trace which service emitted the event.
• correlationId: ties related actions together across services.
• eventTime: the time the producer observed the event.
• id: a unique event identifier for deduplication.

A practical rule: consumers should be able to reject unknown eventType or unsupported schemaVersion without failing the whole stream.

Payload Conventions for Clarity

Inside the payload, keep naming and typing boring and consistent.

• Use explicit field names in lower_snake_case or lowerCamelCase and stick to one style.
• Prefer strings for identifiers (UUIDs, account IDs) to avoid accidental numeric coercion.
• Represent money as integers plus currency (e.g., amount_cents and currency).
• Use ISO-8601 timestamps as strings (e.g., 2026-03-31T10:15:30Z).
• Avoid implicit defaults. If a field matters, include it.

When you need optional fields, make the optionality explicit in the schema rather than relying on “missing means false.”

Schema Evolution Rules That Don’t Break Consumers

Schema changes are inevitable, so define compatibility rules up front.

• Backward compatible changes: adding optional fields, adding new enum values, widening numeric ranges.
• Backward incompatible changes: renaming fields without aliasing, changing a field’s type, removing required fields.
• Versioning strategy: bump schemaVersion when you make a change that could affect validation or interpretation.

A simple operational pattern: consumers accept multiple versions for a limited time by mapping older payloads into a canonical internal model.

Validation Strategy and Failure Handling

Validation should be deterministic and cheap.

• Validate the envelope first (type, version, required metadata).
• Validate the payload next against the schema.
• On failure, decide whether to drop, park, or retry based on the error category.

For example, a malformed payload is usually not transient, so retrying wastes time. A missing required envelope field might indicate a producer bug, so route it to a dead-letter stream for inspection.

Example: Order Created Event with Versioned Payload

Below is a concrete convention you can apply consistently across event types.

{
  "eventType": "order.created",
  "schemaVersion": 2,
  "producerId": "checkout-service",
  "correlationId": "c0a80123-7b2a-4d2c-9f1a-1b2c3d4e5f60",
  "eventTime": "2026-03-31T10:15:30Z",
  "id": "9b2f6c2a-1f0a-4a2b-8d3c-7e6f5a4b3c2d",
  "payload": {
    "order_id": "ORD-10042",
    "customer_id": "CUS-7781",
    "amount_cents": 12999,
    "currency": "USD",
    "items": [
      {"sku": "SKU-1", "qty": 2, "unit_cents": 4999}
    ],
    "notes": null
  }
}

Notice how the payload uses consistent types and naming, and how notes is present but nullable, which avoids ambiguity about whether it was omitted by mistake.

Example: Consumer Mapping Across Schema Versions

When a consumer receives schemaVersion 1, it can map into the canonical shape used internally.

{
  "eventType": "order.created",
  "schemaVersion": 1,
  "payload": {
    "order_id": "ORD-10042",
    "customer_id": "CUS-7781",
    "total_cents": 12999,
    "currency": "USD"
  }
}

A consumer can map total_cents into the canonical amount_cents and set items to an empty list if that field did not exist yet. The key is that the mapping is explicit and deterministic, not inferred.

Example: Payload Rules Checklist

Use this checklist during implementation reviews.

• Envelope fields are always present and named consistently.
• eventType is stable and not derived from internal class names.
• schemaVersion increments only when compatibility could be affected.
• Payload timestamps are ISO-8601 strings.
• Money uses integer cents plus currency.
• Optional fields are represented explicitly in the schema.
• Consumers validate and route invalid messages without blocking healthy traffic.

4.5 Implementing a Minimal End-to-End Event Pipeline Example

This section builds a tiny but complete pipeline: a producer publishes domain events, a JetStream consumer processes them with acknowledgments, and a KV bucket stores the latest derived state. The goal is to show the moving parts in the smallest shape that still behaves like a real system.

Minimal Scenario and Message Contract

Imagine an order service that emits order.created and order.paid. A separate projection service consumes those events and maintains a KV entry per order: { status, lastUpdatedAt }.

Use a consistent envelope so consumers can rely on fields without guessing:

• eventId: unique ID for deduplication
• type: event type like order.created
• subject: original subject
• occurredAt: ISO timestamp
• data: event payload

A practical rule: keep data stable and version it when you must change meaning. If you only change formatting, you can often keep the same schema.

Step 1: Define Stream and Consumer Behavior

Create a stream that retains enough history for replay. For a minimal example, keep retention based on time so you can test replay without filling storage forever.

Consumer settings matter more than people expect. A durable consumer lets you resume after restarts, and explicit acknowledgments keep the system honest: if the consumer crashes mid-update, the message becomes eligible for redelivery.

Step 2: Create a KV Bucket for Derived State

KV buckets are ideal for “latest known value” state. In this example, the projection service writes one key per order.

Key design is simple: order:{orderId}. This keeps the bucket readable and avoids collisions if you later add other entity types.

When you update the KV value, treat it as the result of processing a specific event. That means your handler should compute the new state from the event and then write it once.

Step 3: Implement the Producer

The producer publishes events to a subject that the stream captures. It should also generate eventId deterministically when possible, such as hashing the business identifiers and event type.

Example envelope (conceptual):

• eventId: evt:{orderId}:{type}
• type: order.created
• data: { orderId, customerId, total }

Even in a minimal pipeline, deterministic IDs make replay safer because duplicates map to the same identity.

Step 4: Implement the Consumer with Idempotency

The consumer reads events, checks whether it has already processed eventId, updates KV, and then acknowledges.

For a minimal approach, store processed IDs in KV as well, or embed a “last processed eventId” inside the derived state value. The key point is that the decision to ack should happen only after the state write.

Here’s a compact pseudocode outline:

onMessage(msg):
  event = parse(msg.data)
  if alreadyProcessed(event.eventId):
    msg.ack()
    return

  state = computeNewState(event)
  kv.put(keyFor(event), stateWithEventId(event.eventId))
  msg.ack()

This order prevents a common bug: acknowledging before the KV write completes.

Step 5: Minimal End-to-End Example Code

Below is a single-process sketch that shows the relationships. It omits setup boilerplate so the logic stays readable.

type Event struct {
  EventID   string    `json:"eventId"`
  Type      string    `json:"type"`
  Subject   string    `json:"subject"`
  OccurredAt time.Time `json:"occurredAt"`
  Data      any       `json:"data"`
}

type OrderState struct {
  Status         string    `json:"status"`
  LastUpdatedAt time.Time `json:"lastUpdatedAt"`
  LastEventID   string    `json:"lastEventId"`
}

And the consumer handler logic:

func handle(msg Msg, kv KVBucket) {
  var e Event
  json.Unmarshal(msg.Data(), &e)

  orderID := extractOrderID(e.Data)
  key := fmt.Sprintf("order:%s", orderID)

  prev, _ := kv.Get(key)
  if prev != nil {
    var s OrderState
    json.Unmarshal(prev.Value(), &s)
    if s.LastEventID == e.EventID {
      msg.Ack()
      return
    }
  }

  next := computeOrderState(e)
  kv.Put(key, marshal(next))
  msg.Ack()
}

Step 6: Test Replay Without Surprises

To test replay, stop the consumer after it reads a message but before it writes KV, then restart it. With acknowledgments, the message should be redelivered. With idempotency, the KV value should not regress.

A clean test sequence:

1. Publish order.created.
2. Let the consumer process it and confirm KV shows created.
3. Publish order.paid.
4. Trigger replay from before order.paid.
5. Confirm KV ends in paid and lastEventId matches the latest processed event.

This minimal pipeline is small enough to run locally, but it already demonstrates the core discipline: stable message identity, explicit acknowledgments, and derived state updates that remain correct under redelivery.

5.1 What Consumer Replay Means and When It Is Needed

Consumer replay is the act of reprocessing messages that a consumer has already seen, using the consumer’s stored position and JetStream’s retention history. In practice, you configure a consumer so it starts reading from an earlier point (for example, a sequence number or a time), then you run the same handler logic again with the same or updated code.

Replay is not the same as “restarting the service.” A restart continues from the consumer’s last acknowledged position. Replay is different because it intentionally moves the read position backward (or otherwise changes the starting point) so previously delivered messages are processed again.

When Replay Is Needed

Replay is useful when the correctness of your output depends on processing history, and you need to re-run that history under controlled conditions.

1. Backfills for missing data
If a producer was down for an hour, or a consumer was misconfigured and skipped messages, replay lets you rebuild the missing portion. A common pattern is: publish events continuously, keep retention long enough for the backfill window, then replay from the start of the gap.

2. Bug fixes that affect derived state
Suppose your consumer builds a read model (like a dashboard projection) and a bug caused incorrect aggregation. You fix the handler, then replay the relevant range so the read model converges to the correct result.

3. Schema or business rule changes
When message fields change meaning, you may need to reprocess older events with new interpretation logic. Replay is the mechanism that makes “apply the new rules to old events” concrete.

4. Recovery after data loss in downstream systems
If your database was restored from an earlier snapshot, replay can rebuild the missing updates by reapplying events from the snapshot time.

5. Controlled reprocessing for verification
Replay can also be used to validate that a new consumer version produces the same outputs as the old version for a known range, without relying on luck or manual spot checks.

What Replay Actually Does Under the Hood

JetStream stores messages in a stream according to the stream’s retention policy. A consumer maintains its own progress via acknowledgments. When you request replay, you change where the consumer begins reading from, but the stream still provides the historical messages.

This leads to two important behaviors:

• Handlers may see duplicates. Replay intentionally reintroduces messages that were previously processed.
• Acknowledgment state matters. If you replay without resetting or adjusting the consumer’s acknowledgment position, you may not get the messages you expect.

A good mental model is: the stream is the archive, the consumer is the reader with a bookmark, and replay is moving the bookmark to an earlier page.

Mind Map: Replay Decisions

Example: Backfilling a Missing Hour

Imagine an order service publishes orders.created events to a stream. Your projection service consumes them and updates a table keyed by orderId.

1. The projection service was offline from 2026-03-15 10:00 to 11:00.
2. The stream retention is set to at least 2 hours, so the missing events are still available.
3. You configure the consumer to start from the sequence range that corresponds to 10:00.
4. You run the consumer replay.

To make this safe, your handler should be idempotent. For instance, it should upsert by orderId and ignore events that have already been applied. If you do that, replay becomes a “rebuild from history” operation rather than a “create duplicates” operation.

Example: Bug Fix That Changes Aggregation

Suppose your handler computes dailyRevenue by summing lineItems. A bug caused it to treat refunds as revenue. After fixing the logic, you replay the affected date range.

The key is to ensure the replayed handler updates the same aggregation keys deterministically. If you store dailyRevenue per day and recompute from events, replay will correct the totals. If instead you only incrementally add without accounting for prior incorrect increments, replay will compound the error.

Practical Replay Checklist

• Confirm the stream retention covers the replay window.
• Choose a replay boundary that matches the problem scope.
• Ensure the consumer’s acknowledgment state won’t prevent the intended messages from being read.
• Make the handler idempotent or otherwise replay-safe.
• Verify correctness by checking a small range first, then expanding if needed.

5.2 Replay Boundaries Using Sequence Numbers and Time Windows

Replay boundaries answer one practical question: “Which messages should I reprocess, and which ones should I leave alone?” In JetStream consumer replay, you typically anchor that decision on sequence numbers (precise ordering) and time windows (human-friendly ranges). Using both together is often the cleanest approach: sequence numbers for correctness, time windows for operational convenience.

Sequence Numbers as the Ground Truth

JetStream streams assign an increasing sequence number to each message. A consumer’s replay can start from a specific sequence, which makes it deterministic even when messages arrive out of order at the network level. The key idea is that sequence numbers represent the stream’s internal ordering, not wall-clock time.

A common boundary pattern is “replay from the last processed point plus one.” That avoids reprocessing the message that already succeeded. For example, if your consumer stored that it processed up through sequence 1200, the next replay should begin at 1201.

Example: sequence-based replay boundary

• You run a payment processor consumer.
• It records the last successful stream sequence in its own durable state.
• After a bug fix, you want to reprocess only the affected range.

If the bug affected sequences 1180–1200, you set the replay start at 1180 and stop after 1200 by using an upper bound you compute from the stored “last known good” sequence. Even if the consumer had redeliveries earlier, the sequence boundary keeps the reprocessing scope stable.

Time Windows for Operational Control

Time windows let you say “replay messages published between T1 and T2.” This is useful when you know roughly when the incident happened, even if you don’t know the exact sequence numbers.

However, time is not the stream’s ordering. Two messages can share similar timestamps, and clock skew can make “published at” differ from “ingested at.” So time windows are best treated as a first filter, then tightened with sequence checks.

Example: time-window replay boundary

• A schema change was deployed at 10:15.
• You discover that messages published between 10:14:30 and 10:16 were encoded incorrectly.
• You replay that time window, then validate that the replayed set matches the expected sequence range.

If your consumer handler is idempotent, replaying a slightly larger set is usually safe. If it is not, you should convert the time window into sequence boundaries by inspecting the stream’s sequence range for that period.

Combining Both for Precision Without Guesswork

A robust workflow is:

1. Use a time window to identify the approximate incident period.
2. Determine the corresponding stream sequence range.
3. Replay using sequence numbers for deterministic coverage.

This reduces the chance that you miss messages due to timestamp ambiguity. It also prevents replaying far more than intended when the system was busy.

Example: practical boundary workflow

• Incident window: 2026-03-20T10:14:30Z to 2026-03-20T10:16:00Z.
• You map that to stream sequences 1180–1200.
• You replay from 1180 and stop at 1200.

Now your replay is both operationally understandable and technically exact.

Safety Rules That Make Boundaries Actually Work

Replay boundaries only solve the “which messages” part. The “what happens when they run again” part is handled by consumer design.

1. Idempotent processing: If the same message is processed twice, the final state should be the same. A typical technique is to store a deduplication key derived from the message identity.
2. State updates after success: Update your “last processed” sequence only after the handler completes successfully. Otherwise, a failure can cause you to skip messages on the next replay.
3. Separate boundary logic from business logic: Keep the replay selection in one place (consumer configuration and boundary computation), and keep the handler focused on applying changes.

Example: deduplication key aligned with boundaries

• Each event includes an eventId.
• The handler writes eventId into a KV bucket before applying side effects.
• If eventId already exists, the handler returns without reapplying.

With this, even if your time window pulls in a few extra messages, the system stays correct.

A Concrete Checklist for Setting Boundaries

• Identify the incident period (time window) when you can.
• Convert that period to a sequence range when you need precision.
• Replay from the correct start sequence and ensure you have a clear stop condition.
• Confirm the handler is replay-safe using idempotency and “update after success.”
• Verify the replayed set size matches expectations before you run it against production data.

This approach keeps replays predictable: sequence numbers provide the exact cut, time windows provide the human entry point, and replay-safe handlers make the whole process resilient to the inevitable messiness of real systems.

5.3 Durable Consumers and Replay Control with Acknowledgment State

Durable consumers are JetStream consumers that keep their progress on the server. That progress is tracked through acknowledgment state, so replay control becomes a matter of choosing where the consumer should resume and how it should interpret “already processed.” The key idea is simple: if you ack messages, JetStream can safely move the consumer’s cursor forward; if you don’t, JetStream will keep offering those messages again.

Durable Consumer Fundamentals

A durable consumer is identified by a durable name. When you create it, you also choose how it should behave when it starts consuming. The most important knobs for replay control are:

• Ack policy: whether the consumer expects acknowledgments.
• Replay policy: where consumption begins relative to the stream’s stored messages.
• Ack state: the server-side record of what has been acknowledged.

A practical mental model is a “checkpoint ledger.” Each delivered message has a sequence number. When your handler successfully processes a message, you send an acknowledgment. JetStream records that sequence number as processed for that durable consumer.

Acknowledgment State as the Replay Boundary

Replay boundaries are not only about time or configuration; they are also about what the durable consumer has already acked. Consider a stream with sequences 100–200. If your durable consumer has acked up to 150, then replaying from the beginning of the stream will still skip messages that are already acked, because the ack ledger tells JetStream they are safe to treat as done.

This is why acknowledgment discipline matters. If you ack too early, you can lose work. If you never ack, you’ll reprocess everything every time the consumer restarts, which might be correct for some workflows but is usually expensive.

Replay Control Patterns That Work in Practice

Pattern 1: Resume After Restart

Use a durable consumer with explicit acknowledgments. On restart, reuse the same durable name. JetStream will continue from the last acked position.

Example scenario: a worker crashes after processing sequence 143 but before acking it. When it restarts, JetStream may redeliver 143. Your handler must therefore be idempotent (or otherwise safe to run twice). The durable consumer gives you continuity; it doesn’t magically prevent duplicates.

Pattern 2: Backfill a Known Range

When you need to reprocess a subset, you can create a new durable consumer with a different durable name and a replay start position. The new durable has a fresh ack ledger, so it will process the chosen range even if the original durable has already advanced.

Example scenario: you discover a bug affecting only events from sequence 120–140. You create a one-off durable consumer that starts at 120, processes through 140, and then you stop it. Your main consumer remains unaffected.

Pattern 3: Reprocess Everything for a Fresh Projection

If you maintain a read model (for example, a dashboard view) and want to rebuild it from scratch, you can use a new durable consumer dedicated to the projection. Because it has no prior ack state, it will replay from the configured start point.

This pattern is clean because it separates “business processing” from “projection rebuilding.” The projection consumer can be restarted or replaced without touching the primary workflow.

Example: Ack Timing and Replay Safety

Assume your handler processes an event and updates a KV bucket key. The safe sequence is: validate input, apply the update, then ack the message. If you ack before the KV update, a crash can cause the message to be considered done while the state update never happened.

A simple idempotency approach is to store the last processed sequence number per entity in the KV bucket. When a message arrives, compare its sequence number to the stored value. If it’s older or equal, skip the update and still ack the message. This turns redelivery into a harmless no-op.

Example: Two Durables for One Stream

You can run two durable consumers on the same stream:

• Primary consumer: processes commands and acks normally.
• Projection consumer: rebuilds a read model and can be recreated with a new durable name.

Because each durable has its own ack ledger, the projection can replay without disturbing the primary consumer’s progress. This separation keeps replay control predictable and reduces the chance of accidental double-processing in the wrong place.

5.4 Building Replay Safe Handlers with Idempotent Processing

Replay means the same logical event may be delivered again, sometimes after partial failures or during backfills. Your handler must therefore tolerate duplicates without corrupting state or producing inconsistent side effects. The simplest mental model is: “processing the same event twice should have the same effect as processing it once.”

Idempotency First, Side Effects Second

Start by separating pure state transitions from external effects.

• State transition: update KV or database rows so the system converges to the same result.
• External effect: send emails, call HTTP services, publish follow-up events, or write to third-party systems.

If you make the state transition idempotent, you can often gate external effects behind that state. For example, store a “processed marker” keyed by the event identity, then only perform the external effect when the marker is newly created.

Choosing an Event Identity That Survives Replay

Idempotency requires a stable key. In JetStream, you can use the stream sequence as a replay-safe identity, but it is only meaningful within that stream. A practical approach is a composite identity:

• Stream identity: stream name (or a fixed stream ID)
• Sequence identity: the message sequence number
• Optional domain identity: event ID from your payload when you need cross-stream deduplication

When you read from a consumer, you can access the message metadata (including sequence) and combine it with your payload’s event ID if present.

KV Marker Pattern for Exactly Once Effects

A common pattern uses a KV bucket to record processed identities.

1. Compute dedupeKey from message metadata.
2. Attempt to create or set a marker in a KV bucket.
3. If the marker already exists, stop early.
4. Otherwise, apply the state transition and then perform external effects.

This works because KV updates are versioned, and you can treat “marker exists” as the idempotency check.

Example: Dedupe Marker with KV

function handle(msg):
  dedupeKey = "orders:" + msg.stream + ":" + msg.seq

  if kv.get(dedupeKey) != null:
    msg.ack()
    return

  // Apply state transition first
  order = parse(msg.data)
  db.upsert(order.id, order)

  // Record marker after successful transition
  kv.put(dedupeKey, { at: now() })

  // External effects gated by marker
  notify(order)

  msg.ack()

If the handler crashes after the state transition but before the marker, replay will re-run the transition. That is why the state transition must also be idempotent (for example, using upsert with deterministic fields).

Making State Transitions Idempotent

Idempotent state updates usually fall into three categories:

• Upsert by primary key: writing the same final values twice yields the same row.
• Compare-and-set with versioning: only apply if the stored version is older.
• Append with dedupe: if you must append, store a dedupe key per append item.

For workflow progress, prefer storing the latest completed step index. Replaying an event that corresponds to an already completed step should be a no-op.

Ordering Assumptions and Their Limits

Replay safety is not the same as correctness under reordering. If your handler assumes strict ordering, duplicates are only half the problem. To avoid gaps:

• Use sequence-based gating when order matters within a stream.
• Store the last processed sequence per entity in KV, then ignore older sequences.

This turns “replay” into “replay plus ordering enforcement,” which is usually what you want.

Mind Map: Idempotent Replay Safe Handler

Example: Workflow Step Handler with Sequence Gating

A workflow service might receive StepCompleted events. Store lastSeqByWorkflow in KV.

• If msg.seq <= lastSeq, ignore.
• If greater, apply the step completion and update lastSeq.

This prevents both duplicates and older replays from regressing progress.

Practical Checklist for Replay Safety

• Use a dedupe key derived from message identity you can reproduce during replay.
• Ensure state transitions are idempotent even if the marker write is delayed.
• Gate external side effects behind the same idempotency decision.
• Enforce ordering when your domain requires it by storing last processed sequence per entity.
• Acknowledge only after the handler reaches a safe completion point.

When these pieces are in place, replay becomes a controlled inconvenience rather than a correctness risk. Your system still moves forward, even when the same message shows up more than once—because it was designed to expect that.

5.5 Practical Replay Scenarios for Backfills and Bug Fixes

Consumer replay is most useful when you can explain what “correct” means for each message. In practice, that usually comes down to three things: a stable mapping from messages to state, a clear boundary for what to replay, and handlers that tolerate duplicates.

Backfill Scenario: Rebuilding a Read Model After Downtime

Imagine a service that writes orders to a JetStream stream and a separate projector that builds a “current order status” table. If the projector was down for a few hours, the table is stale. The backfill goal is to rebuild it without corrupting newer updates.

1. Choose a boundary. Use the stream sequence range that covers the projector downtime. If you know the last sequence the projector processed, replay from that point onward. If you only know time, replay from the earliest timestamp that likely includes the gap.
2. Make the projector idempotent. Each order update should include an order ID and a monotonically increasing event version (or rely on stream sequence). The projector stores the latest version it has applied. If it receives an older version during replay, it ignores it.
3. Isolate side effects. During replay, write only to the read model table. Avoid sending notifications or triggering downstream actions until the replay is complete.

A concrete handler pattern looks like this: read the current stored version for the order, compare it to the incoming event version, and update only if the incoming version is newer. That single comparison prevents replay from “rewinding” state.

Bug Fix Scenario: Correcting a Faulty State Transition

Suppose the consumer previously mapped a payment event to the wrong internal status. The stream already contains the historical events, so the fix is to replay those events with corrected logic.

1. Create a new consumer configuration. Keep the original consumer running if it is still needed for live traffic. For the replay, use a separate consumer name so you don’t mix old and new handler behavior.
2. Version the projection logic. Store a projection version in the read model. When the handler runs, it writes the new projection version alongside the updated state. This helps you verify that the table was rebuilt using the corrected logic.
3. Validate invariants after replay. For example, if “paid” orders must always have a non-empty receipt ID, scan the affected keys after replay and fail fast if the invariant doesn’t hold.

If you can’t afford a full scan, validate per message during replay and record the first violation with the message metadata (sequence, subject, and event ID). That turns debugging from guesswork into a reproducible trail.

Backfill Scenario: Schema Compatibility Without Guessing

Backfills often coincide with schema changes. The safe approach is to replay with a decoder that can handle both the old and new payload shapes.

• Use explicit fields. If the new schema adds a field, treat it as optional during replay. Do not infer missing values from unrelated fields.
• Keep mapping deterministic. The same input event should always produce the same normalized internal representation. Determinism matters because replay repeats the same sequence.

A practical example: an event originally had customerId as a string, and later it became an object { id, region }. During replay, if the payload contains the old string form, normalize it into { id: customerId, region: "unknown" } so downstream logic sees a consistent internal shape.

Bug Fix Scenario: Handling Partial Processing After Crashes

Sometimes a consumer crashes after it performs side effects but before it acknowledges the message. Replay then re-delivers the message, and the handler must not double-apply side effects.

1. Use a deduplication key. Include a unique event ID in every message. Store processed event IDs in a KV bucket or in the target state record.
2. Check before side effects. The handler should first check whether the event ID was already applied. If yes, it returns success without repeating the side effect.
3. Acknowledge only after completion. Even with deduplication, acknowledge after the side effects and state updates are consistent.

This pattern is boring in the best way: it makes replay safe even when the system fails at the worst possible moment.

Mind Map: Replay Execution Checklist

Example: A Replay Plan for One Stream and One Projection

• Replay range: from the last known processed sequence up to the stream’s current end.
• Consumer: a dedicated replay consumer with the corrected handler.
• Handler: idempotent by order ID plus event version, and it writes projection version metadata.
• Verification: after replay, check that every order with a “paid” status has a receipt ID and that the number of updated orders matches the number of distinct order IDs in the replay range.

When these steps are followed, replay becomes a controlled repair tool rather than a risky rerun. The system still does what it always does—process messages—but now it does so with guardrails that make correctness measurable.

8. End-to-End Application Patterns with JetStream and KV

8.1 Command Handling With Event Emission and State Updates

Command handling is the part of your system that turns “do something” into two outcomes: (1) state changes that represent the new truth, and (2) events that tell other parts of the system what happened. With JetStream, you can keep this flow consistent by treating the command handler as the single writer of a state record, then emitting events that describe the result.

Mind Map: Command to State to Events

- Command Handling - Inputs - Command payload - Correlation ID - Actor or tenant ID - Expected version or preconditions - State Update - Source of truth - KV bucket key - Versioning via sequence or explicit version - Idempotency - Dedup key per command - Store last processed command ID - Atomicity approach - KV update as single-writer - Validate preconditions before write - Event Emission - Event schema - event type - command ID - state version - timestamps - Subject naming - domain.entity.action - Publication strategy - Publish after successful state write - Include correlation ID - Consumer Responsibilities - Apply events to read models - Acknowledge after processing - Handle duplicates safely - Failure Modes - State write fails - Publish fails - Handler crashes after state write - Redelivery and replay

Command Handler Responsibilities

A practical handler usually does five steps in order.

1. Validate the command: check required fields, tenant boundaries, and preconditions. If you use an expected version, reject commands that don’t match the current state version.
2. Load current state: read the KV value for the entity key. If the key doesn’t exist, treat it as an initial state and set version to 0.
3. Enforce idempotency: store a “last processed command ID” alongside the state (or in a separate KV key). If the same command ID arrives again, return success without changing state or emitting duplicate events.
4. Update state: write the new state to the KV bucket with the next version. This is where you keep the system’s truth.
5. Emit an event: publish an event to a JetStream stream subject that matches the domain action. The event should include the command ID, the new state version, and the correlation ID.

This ordering matters. If you update state first, you can safely handle retries by checking idempotency before publishing again.

Example: Place Order Command with KV State and Event Emission

Assume you model an order as a KV key like order:{orderId}. Your command is PlaceOrder with commandId, orderId, and items.

State record shape (stored in KV):

• orderId
• status (e.g., Placed)
• version
• lastCommandId

Event shape (published to JetStream):

• type: OrderPlaced
• commandId
• orderId
• version
• correlationId
• occurredAt

Here’s a minimal pseudocode flow.

function handlePlaceOrder(cmd):
  stateKey = "order:" + cmd.orderId
  state = kv.get(stateKey) or defaultState(cmd.orderId)

  if state.lastCommandId == cmd.commandId:
    return { ok: true, idempotent: true }

  if cmd.expectedVersion != null and cmd.expectedVersion != state.version:
    throw VersionMismatch

  newState = {
    orderId: cmd.orderId,
    status: "Placed",
    version: state.version + 1,
    lastCommandId: cmd.commandId
  }

  kv.put(stateKey, newState)

  event = {
    type: "OrderPlaced",
    commandId: cmd.commandId,
    orderId: cmd.orderId,
    version: newState.version,
    correlationId: cmd.correlationId,
    occurredAt: now()
  }

  js.publish("orders.place", event)
  return { ok: true, idempotent: false }

Handling the “State Updated, Publish Failed” Case

If the handler crashes after the KV write but before the publish, you’ll see the command again (or a retry will occur). Idempotency prevents double state updates, but you still need to ensure the event gets emitted.

A simple approach is to store an additional flag in KV, such as lastPublishedCommandId. Then:

• If lastCommandId matches but lastPublishedCommandId differs, publish the event using the stored state version.
• After successful publish, update lastPublishedCommandId.

This keeps the handler deterministic: every command ID maps to exactly one state version and exactly one emitted event.

Subject and Payload Conventions That Keep Consumers Calm

Use a consistent subject pattern such as domain.entity.action (for example, orders.place). Keep payloads self-describing: include type, commandId, and version. Consumers can then:

• Ignore duplicates by checking commandId.
• Apply updates in order by using version.
• Correlate logs using correlationId.

Practical Checklist for Command Handling

• One writer per entity key: ensure only the command handler updates the KV record for that entity.
• Idempotency keys: store lastCommandId (and optionally lastPublishedCommandId).
• Versioning: increment version on every successful state change.
• Event includes version: consumers can verify they’re applying the right state step.
• Acknowledge after processing: downstream consumers should ack only after they’ve updated their read models.

When these pieces line up, command handling becomes boring in the best way: predictable, retry-friendly, and easy to reason about when messages arrive more than once.

8.2 Event Fan Out for Multiple Independent Consumers

Fan out means one published event is processed by several consumers that do not depend on each other’s timing. In JetStream terms, you typically achieve this by using either multiple consumers on the same stream or multiple consumers on different streams that receive the same source event. The key is to keep each consumer’s contract small: it should know what it needs, acknowledge when it is done, and store its own progress.

Core Idea and Why It Works

Start with a single producer that publishes a domain event, for example order.created. Each independent consumer subscribes to the subject(s) that match that event and performs its own work: one builds a read model, another sends notifications, and a third updates workflow state.

A simple rule prevents accidental coupling: consumers must not write to each other’s state. If consumer A needs data that consumer B computes, A should either compute it itself or read it from a shared source designed for that purpose (like a KV bucket used as a state store).

Subject Design for Clean Fan Out

Use subject naming so consumers can subscribe narrowly. For example:

• orders.created for the event itself
• orders.created.* if you want variants like orders.created.us later

When consumers subscribe, prefer the narrowest subject that still covers their responsibility. That reduces wasted deliveries and makes it easier to reason about what each consumer will process.

Consumer Setup Patterns

There are two common patterns.

1. Multiple consumers on the same stream: all consumers read the same event history. This is convenient when you want consistent ordering and replay behavior across consumers.
2. Multiple streams fed from the same producer event: you publish once, then route or copy into specialized streams. This is useful when retention or processing policies differ per consumer.

Either way, each consumer should have its own durable name and its own acknowledgment state. That way, a slow consumer does not block faster ones.

Mind Map: Fan Out Responsibilities and Boundaries

# Event Fan Out for Multiple Independent Consumers - Producer - Publishes domain event - Uses stable subject naming - Includes correlation id and event id - Event Stream - Stores ordered event history - Retention matches operational needs - Independent Consumers - Consumer a Read Model - Updates query-friendly storage - Acks after persistence - Consumer B Notifications - Sends messages externally - Acks after side effect recorded - Consumer C Workflow State - Updates KV or DB state - Acks after state commit - Boundaries - No consumer writes to another consumer state - Shared state uses explicit stores - Reliability - Idempotent handlers - Ack on success - Replay uses consumer position

Example: Three Consumers for One Event

Assume the producer publishes an event with fields like eventId, correlationId, orderId, and createdAt. Each consumer uses eventId to deduplicate.

Consumer A builds a read model

• It writes a row like orders_view[orderId] = {status: "created"}.
• It only acknowledges after the write succeeds.
• If it receives the same eventId again, it detects the duplicate and skips.

Consumer B sends notifications

• It writes a record notification_outbox[eventId] before sending.
• It sends only if the outbox record indicates it hasn’t been sent.
• It acknowledges after the outbox write and send attempt are complete.

Consumer C updates workflow state

• It updates a KV key like workflow:order:{orderId} to step=created.
• It acknowledges after the KV update commits.
• On replay, it overwrites the same step deterministically.

Practical Replay Safety for Fan Out

When you replay, each consumer reprocesses from its own position. That means fan out stays independent as long as each consumer is replay-safe:

• Idempotency: use eventId or a derived unique key to avoid double writes.
• Deterministic state updates: updates should converge to the same result when applied multiple times.
• Acknowledgment discipline: acknowledge only after the consumer’s side effects are durable.

Diagram: Fan Out Flow and Acknowledgment Points

    flowchart LR
  P[Producer publishes orders.created] --> S[JetStream Stream stores event]
  S --> A[Consumer a Read Model]
  S --> B[Consumer B Notifications]
  S --> C[Consumer C Workflow State]

  A -->|Ack after DB write| S
  B -->|Ack after outbox record and send| S
  C -->|Ack after KV update| S

Operational Checks That Prevent Surprises

• Verify subject filters: confirm each consumer subscribes to exactly what it should.
• Check durable names: durable consumers keep their own replay position.
• Measure consumer lag separately: one consumer can fall behind without affecting others.
• Test duplicate delivery: simulate redelivery and confirm idempotency holds.

With these pieces in place, fan out becomes a predictable pattern: one event, many independent processors, and clear boundaries for state and acknowledgments.

8.3 Multi Step Workflows with Correlation Identifiers

Multi step workflows are where event-driven systems either become pleasantly predictable or quietly confusing. The difference is usually correlation identifiers: a value that ties together all events and state transitions that belong to the same logical workflow instance.

At a minimum, a correlation identifier should be:

• Stable across the whole workflow instance.
• Present in every event that participates in the workflow.
• Easy to log and easy to index in your storage.

A practical pattern is to generate a workflowId at the start of the workflow, then propagate it through every command and event. If you also need to track retries or sub-steps, add a stepId and a attempt counter, but keep workflowId as the anchor.

Correlation Identifier Data Model

Use a small, consistent envelope so every handler can extract the same fields. For example, each event can include:

• workflowId: the workflow instance key.
• step: the logical step name, like payment.authorize.
• sequence: an integer step sequence or business ordering.
• attempt: retry attempt number.
• causationId: the event id that caused this event.
• correlationId: often the same as workflowId, but keep the naming consistent.

A simple envelope makes debugging less guessy. When something goes wrong, you can filter by workflowId and reconstruct the story without reading every topic.

Mind Map: Correlation Flow Through Steps

- Multi Step Workflow - Correlation Identifier - workflowId - correlationId - stepId (optional) - attempt (optional) - Event Envelope - causationId - eventId - timestamp - payload - Step Execution - command handler - state update - event emission - Consumer Behavior - idempotency key - ack on success - retry on transient errors - State Tracking - KV bucket for workflow state - status per step - last processed sequence - Replay Safety - dedupe by eventId or stepId - conditional transitions

Step Orchestration Pattern Using KV and Events

A reliable workflow needs two things: a way to know what step comes next, and a way to prevent duplicate transitions when events are replayed or redelivered.

A common approach is:

1. Store workflow state in a KV bucket keyed by workflowId.
2. Each step handler reads the workflow state, checks whether the transition is allowed, then writes the updated state.
3. Emit the next event with the same workflowId.

KV buckets are especially useful because they give you a compact “current status” view. For example, the KV value can be JSON like:

• status: running, completed, failed
• steps: a map from step name to done or pending
• lastSequence: the last accepted step sequence

When a consumer receives an event, it should:

• Verify the workflow exists or initialize it if this is the first step.
• Check whether the step is already marked done.
• If done, ack and stop.
• If not done, perform the side effect, update KV, then emit the next event.

This makes the workflow replay-safe because the state check turns duplicates into no-ops.

Example: Payment Then Fulfillment Workflow

Imagine a workflow with two steps: payment.authorize then order.fulfill.

Event 1: Start

• Producer emits payment.authorize.request with workflowId = W123 and sequence = 1.

Consumer for Step 1

• Reads KV at key W123.
• If steps.payment.authorize is done, it acks immediately.
• Otherwise it calls the payment service.
• Writes KV: mark payment.authorize as done, set lastSequence = 1.
• Emits order.fulfill.request with the same workflowId = W123 and sequence = 2.

Consumer for Step 2

• Performs the same idempotency check for order.fulfill.
• On success, marks the workflow completed and emits order.completed.

Here’s the key point: every emitted event carries workflowId, so logs and state updates line up even when steps run on different consumers.

Mind Map: Idempotency and Transition Rules

Practical Replay Behavior

When you replay from an earlier point, you’ll often re-deliver events for steps that already completed. The workflow remains correct because:

• KV state prevents re-running side effects.
• Sequence or step guards prevent out-of-order transitions.
• Correlation identifiers keep the workflow instance consistent across all replays.

If you need to re-run a step after a bug fix, you can reset only the relevant step in KV (for that workflowId) and then replay the corresponding request events. The correlation id ensures you don’t accidentally mix steps from different workflow instances.

Operational Checklist for Correlation Identifiers

• Generate workflowId once at workflow start and reuse it everywhere.
• Include workflowId in every event and command payload.
• Use KV state checks before side effects.
• Make step transitions conditional on KV state and sequence.
• Log workflowId, step, and sequence in every handler.

With these pieces in place, multi step workflows become a set of deterministic transitions rather than a pile of messages that happen to share a topic name.

8.4 Coordinating Replays with Workflow State and Versioning

Replays are easiest to reason about when workflow state is explicit, versioned, and tied to the consumer’s progress. The goal is simple: when you reprocess events, you should either (a) produce the same observable outcome as the first run, or (b) stop safely at the point where the workflow already advanced.

Workflow State as a Versioned Contract

Store workflow state in a JetStream KV bucket keyed by a stable workflow identifier, such as orderId or sessionId. Each state update should include:

• workflowVersion: a monotonically increasing integer for the workflow’s logical steps.
• lastEventSeq: the last stream sequence number the workflow accepted.
• status: pending, running, completed, or failed.
• idempotencyKey: a deterministic key derived from the event identity.

A practical rule: only advance workflowVersion when the handler accepts an event that is newer than what the workflow already recorded. That makes replay deterministic.

Versioning Strategy for Step Handlers

Treat each step handler as a small state machine. For example, a workflow might have steps validate, reserve, charge, ship. Each step has a required minWorkflowVersion and an expectedEventType.

When replay delivers an event, the handler checks three things:

1. Workflow status allows the step to run.
2. Event identity matches the recorded idempotencyKey for the target step.
3. Version gate ensures the workflow hasn’t already moved past this step.

If any check fails, the handler returns success without side effects. That’s the “safe stop” behavior you want during replay.

Coordinating Consumer Replay with State Updates

Consumer replay is driven by JetStream sequence numbers and acknowledgment state. The workflow state should mirror what the consumer has accepted, not what it has merely seen.

A clean pattern is:

1. Read KV state for workflowId.
2. Decide whether the event should be applied.
3. Apply side effects only if the event is new for that workflow step.
4. Update KV state with the new workflowVersion, lastEventSeq, and idempotencyKey.
5. Acknowledge the stream message.

If the process crashes after side effects but before KV update, the next replay will re-run the handler. The version gate and idempotency key prevent duplicate side effects.

Mind Map: Replay Coordination with KV State

- Coordinating Replays with Workflow State and Versioning - Workflow State Contract - KV bucket keyed by workflowId - Fields - workflowVersion - lastEventSeq - status - idempotencyKey - Step Handler State Machine - Steps - validate - reserve - charge - ship - Gates - status allows step - event identity matches - workflowVersion is at expected point - Replay Flow - Read KV state - Decide apply or safe stop - Side effects only on apply - Update KV state - Ack message - Failure Modes - Crash before KV update - replay re-runs - idempotency prevents duplicates - Crash after KV update - ack may be retried - version gate avoids re-apply

Example: Order Workflow with Replay-Safe Steps

Assume events arrive on a stream with sequence numbers. Each event has:

• eventId: unique per business event
• type: OrderValidated, OrderReserved, etc.
• workflowId: the order id

The handler computes idempotencyKey = "<type>:<eventId>".

Case: replay delivers OrderReserved twice

• First delivery: KV shows workflowVersion=2, idempotencyKey for reserve step is empty. Handler applies reservation, updates KV to workflowVersion=3, sets idempotencyKey to OrderReserved:<eventId>, and records lastEventSeq=1050.
• Replay delivery: KV now has workflowVersion=3 and the reserve step idempotency key already set. The version gate fails for the reserve step, so the handler returns without calling the reservation side effect. The message is acknowledged.

This works even if replay starts from an earlier point, because the workflow state already contains the outcome of the first run.

Example: Version Gate Using lastEventSeq

Sometimes you want a stricter rule: never accept an event with a sequence number lower than what the workflow recorded. In that case:

• If eventSeq <= lastEventSeq, treat it as already processed and stop.
• Otherwise, run the step checks and then update lastEventSeq.

This reduces the chance of applying an older event after a newer one, which can happen when multiple producers publish related events.

Practical Checklist for Coordinated Replay

• KV state is the workflow’s source of truth for step progress.
• Every step has a version gate and an idempotency key.
• KV updates happen before acknowledgment.
• Side effects are conditional on “apply” decisions, not on message receipt.
• lastEventSeq is recorded only when the workflow accepts the event.

With these pieces aligned, replay becomes a controlled re-execution rather than a guessing game about what already happened.

8.5 Reference Implementation Walkthrough for a Real Time Dashboard

This walkthrough builds a small real-time dashboard that shows live order counts, recent order events, and a “current session state” panel. It uses JetStream streams for event history and a KV bucket for the latest state. The key idea is simple: events are append-only facts, while KV holds the current view used by the UI.

Architecture Overview

Producer publishes orders.created and orders.updated events to JetStream. A consumer processes events, updates KV session state, and also emits projection events for the dashboard. The dashboard UI subscribes to projection events and reads KV for the current snapshot.

# Real Time Dashboard Reference Implementation - Goal - Live metrics panel - Recent events list - Current session state - Data Flow - Producer publishes domain events - Consumer processes events - Consumer updates KV state - Consumer emits projection events - UI renders from projection + KV - JetStream Components - Stream for order events - Consumer with durable name - Replay for backfills and bug fixes - KV bucket for session state - Reliability Practices - Manual acks - Idempotent handler using event id - Replay-safe KV writes - Error handling with retry and dead letter - Operational Practices - Subject naming conventions - Consumer lag monitoring - Schema version in payload

Subject and Stream Design

Use subjects that map cleanly to domain meaning:

• orders.created
• orders.updated
• dashboard.projection.orders

Create one stream for order events with retention long enough for replay testing. For example, keep 24 hours of events so you can re-run projections after a handler change. Configure a durable consumer for the projection worker so it can resume after restarts.

Message Shape and Idempotency

Each event includes:

• event_id (unique, stable)
• occurred_at (timestamp)
• schema_version
• order_id
• type (created or updated)

The projection consumer stores processed event_ids in KV or derives idempotency from KV versioning. A practical approach is to write KV state keyed by event_id for dedupe, then update the session snapshot keyed by session_id. That keeps replays safe without relying on timing.

KV Bucket Layout

Create a KV bucket named dashboard.session_state.

Use keys like:

• session:{session_id} for the current snapshot
• processed:{event_id} for dedupe markers

Snapshot value fields:

• last_event_at
• order_count_total
• recent_order_ids (bounded list)
• last_processed_event_id

Dedupe marker value fields:

• processed_at
• schema_version

Consumer Replay Strategy

When you fix a projection bug, you replay the stream from a known point. In practice, you set the consumer to start from a specific sequence number or time window, then let the handler rebuild KV state.

Replay-safe rule: the handler must tolerate duplicates. If processed:{event_id} already exists, skip state mutation and just ack.

Minimal Reference Flow Example

Below is a compact pseudo-implementation showing the projection worker logic. It assumes manual ack and KV-based dedupe.

// Pseudocode for projection worker
func handle(msg JetStreamMsg) {
  evt := decode(msg.Data)
  dedupeKey := fmt.Sprintf("processed:%s", evt.EventID)

  if kv.Get(dedupeKey) != nil {
    msg.Ack()
    return
  }

  // Update snapshot
  snapKey := fmt.Sprintf("session:%s", evt.SessionID)
  snap := kv.GetOrDefault(snapKey, defaultSnap())

  snap.OrderCountTotal++
  snap.LastEventAt = evt.OccurredAt
  snap.RecentOrderIDs = appendBounded(snap.RecentOrderIDs, evt.OrderID, 20)
  snap.LastProcessedEventID = evt.EventID

  kv.Put(dedupeKey, markerValue(evt.SchemaVersion))
  kv.Put(snapKey, snap)

  msg.Ack()
}

A second consumer can publish dashboard.projection.orders events after KV updates, or the UI can read KV directly on a timer. If you want lower UI latency, emit projection events right after KV writes.

Dashboard Rendering Contract

The UI maintains two pieces:

1. Snapshot from KV session:{session_id}.
2. Recent events from dashboard.projection.orders.

When a projection replay runs, snapshot updates may jump. The UI should treat snapshot as authoritative and recent events as a stream that can be re-ordered; it renders by occurred_at and truncates to the last N items.

Mind Map: Implementation Steps

# Implementation Steps - Step 1 - Define event payload fields - Choose subject names - Step 2 - Create order events stream - Create durable projection consumer - Step 3 - Create KV bucket dashboard.session_state - Define keys processed and session snapshots - Step 4 - Implement idempotent handler - Use KV dedupe marker - Update snapshot and ack - Step 5 - Add projection event emission - UI subscribes to projection subject - Step 6 - Test replay - Run handler from earlier sequence - Verify snapshot matches expected counts

Practical Example Walkthrough with Replay

Start with a stream containing 100 orders.created events for a single session. Run the projection consumer once and confirm order_count_total becomes 100. Then change the handler to include recent_order_ids ordering by occurred_at. Run a replay from the beginning; the dedupe markers will prevent double-counting, while the snapshot fields that depend on ordering will be corrected if you either version the snapshot key or allow overwriting snapshot fields during replay. The simplest safe method is to include a projection_version in the snapshot and only skip dedupe markers for the same version.

This keeps the dashboard consistent: counts remain correct, and the “recent list” reflects the latest projection logic without manual cleanup.

INFO  correlation_id=9f3a... nats_subject="orders.created" stream="ORDERS" consumer="worker-A" seq=120 redelivered=false handler="CreateOrder" event_type="OrderCreated" outcome="received"
INFO  correlation_id=9f3a... handler="CreateOrder" outcome="started" kv_bucket="order_state" kv_key="order-77" action="read"
INFO  correlation_id=9f3a... handler="CreateOrder" outcome="skipped" reason="already_processed" kv_bucket="order_state" kv_key="order-77" kv_revision=41
INFO  correlation_id=9f3a... handler="CreateOrder" outcome="success" action="ack" seq=120

import { connect, StringCodec } from 'nats';

const nc = await connect({ servers: 'nats://localhost:4222' });
const js = nc.jetstream();
const codec = StringCodec();

function makeHeaders(traceId, spanId, parentSpanId, messageId) {
  return {
    'trace-id': traceId,
    'span-id': spanId,
    'parent-span-id': parentSpanId,
    'message-id': messageId,
  };
}

const traceId = 'trc_01';
const spanId = 'spn_pub_01';
const parentSpanId = 'spn_req_01';
const messageId = 'msg_01';

await js.publish('orders.created', codec.encode(JSON.stringify({ id: 7 })), {
  headers: makeHeaders(traceId, spanId, parentSpanId, messageId),
});

import { connect, StringCodec } from 'nats';

const nc = await connect({ servers: 'nats://localhost:4222' });
const js = nc.jetstream();
const codec = StringCodec();

const sub = await js.subscribe('orders.created', {
  durable: 'orders-worker',
  manualAck: true,
});

for await (const m of sub) {
  const h = m.headers || {};
  const traceId = h['trace-id'];
  const parentSpanId = h['span-id'];
  const messageId = h['message-id'];

  // startSpan({ traceId, parentSpanId, name: 'handle orders.created', messageId })
  // process payload, record errors, then ack
  m.ack();
}

1) Sample consumer lag and ack rate every 10s.
2) Sample stream stored bytes every 30s.
3) If lag grows for 3 consecutive samples, classify as processing bottleneck.
4) If redeliveries increase while ack rate drops, classify as handler failure.
5) If stored bytes exceed expected range for the retention window, classify as retention mismatch.
6) Emit one actionable log line with the classification and the top suspected cause.

Identity: orders-writer
- Connect: allowed
- Publish: prod.orders.created, prod.orders.updated
- Subscribe: none
- JetStream: allowed to publish to stream bound subjects only
- Consumer management: denied
- KV access: denied

{
  "alg": "A256GCM",
  "kid": "kek-2026-03",
  "dek": "<encrypted-dek-by-kek>",
  "iv": "<base64-iv>",
  "ct": "<base64-ciphertext>",
  "tag": "<base64-aead-tag>",
  "ts": "2026-03-15T10:20:30Z",
  "nonce": "<random-nonce>"
}

{
  "envelope": {"alg":"A256GCM","kid":"sig-2026-03", "ct":"...", "iv":"...", "tag":"...", "ts":"2026-03-15T10:20:30Z", "nonce":"..."},
  "sigAlg": "Ed25519",
  "sigKid": "sig-2026-03",
  "signature": "<base64-signature>"
}

Observed:
- Producer publish latency increases with CPU high
- Consumer lag grows
- Consumer handler time is stable

Interpretation:
- Producer is likely blocked by client buffering or serialization
- Consumer may be fine, but it can’t drain fast enough due to producer pacing

Next steps:
- Profile producer payload building
- Reduce allocations and trim payload
- Publish asynchronously or in batches

Observed:
- Producer publish latency stable
- Consumer handler time high
- Ack rate low
- Lag increases

Interpretation:
- Handler is the bottleneck, not storage

Next steps:
- Optimize handler CPU and downstream I/O
- Increase worker concurrency if contention is low
- Batch downstream writes

// Pseudocode: bounded in-flight processing
batchSize := 20
for {
  msgs := pullConsumer.Fetch(batchSize, timeout)
  if len(msgs) == 0 { continue }

  for _, m := range msgs {
    // Decode only needed fields
    fields := decodeHeaderOnly(m.Data)
    if fields.Size > maxAllowed { m.Ack(); continue }

    // Do durable work
    err := handle(fields, m.Data)
    if err != nil {
      // Let redelivery happen; do not ack
      continue
    }
    m.Ack()
  }
}

// Pseudocode: flush when count or bytes exceed limits
maxCount := 100
maxBytes := 2 * 1024 * 1024
var batch []Item
var batchBytes int

for msg := range workQueue {
  item := toItem(msg)
  itemBytes := estimateBytes(item)

  if len(batch) >= maxCount || batchBytes+itemBytes > maxBytes {
    flush(batch)
    batch = batch[:0]
    batchBytes = 0
  }

  batch = append(batch, item)
  batchBytes += itemBytes
}

if len(batch) > 0 { flush(batch) }

consumer:
  ack_wait: 2s
  redelivery_delay: 200ms
  max_ack_pending: 100
  max_deliveries: 10

Run Label: replay_AckBatch_MaxInflight
Date Tag: 2026-03-31
Primary Metric: p95 ack latency
Warm-up: 45s
Measure: 5m
Workload: 1M events, 1KB avg, 5% 32KB
Consumer: durable replay from seq=500000
Config A: max_in_flight=50, ack=per_batch
Config B: max_in_flight=200, ack=per_batch
Outputs: p95 latency, throughput, peak lag, lag slope, redeliveries, CPU

type Event struct {
  EventID   string
  Type      string
  OccurredAt string
  Payload   map[string]any
}

type Effect struct {
  Kind    string
  Subject string
  Payload map[string]any
}

type Result struct {
  NewState map[string]any
  Effects  []Effect
  Ack      bool
}

func HandleEventCore(e Event, state map[string]any) Result {
  // deterministic logic only
  if e.Type == "OrderPlaced" {
    return Result{
      NewState: map[string]any{"lastOrder": e.Payload["orderId"]},
      Effects:  []Effect{{Kind: "publish", Subject: "orders.confirm", Payload: e.Payload}},
      Ack:      true,
    }
  }
  return Result{NewState: state, Effects: nil, Ack: true}
}

func TestDuplicateEventProducesNoEffects(t *testing.T) {
  fixedEvent := Event{EventID: "evt-1", Type: "OrderPlaced", OccurredAt: "2026-03-31T10:15:30Z", Payload: map[string]any{"orderId": "o-9"}}
  repo := NewFakeIdempotencyRepo(map[string]bool{"evt-1": true})
  state := map[string]any{}

  res, err := HandleEventWrapper(fixedEvent, state, repo)
  if err != nil { t.Fatal(err) }
  if len(res.Effects) != 0 { t.Fatalf("expected no effects") }
  if !res.Ack { t.Fatalf("expected ack") }
}

// Pseudocode style Go example
func TestOrderWorkflow(t *testing.T) {
  nc := StartEphemeralNATS(t)
  js := jetstream.New(nc)

  stream := CreateStream(js, "test.orders", "t.orders.*")
  kv := CreateKV(js, "test.kv")

  done := make(chan struct{}, 1)
  StartConsumer(js, stream, "orders-cons", func(msg Msg){
    kv.Put("order-1", []byte("paid"))
    msg.Ack()
    done <- struct{}{}
  })

  Publish(nc, "t.orders.create", []byte(`{"id":"order-1"}`))

  select {
  case <-done:
  case <-time.After(2 * time.Second):
    t.Fatal("consumer did not process")
  }

  v, _ := kv.Get("order-1")
  if string(v.Value()) != "paid" { t.Fatal("kv mismatch") }
}

1) Publish fixture events to a stream.
2) Clear KV bucket state for the test namespace.
3) Create a durable consumer with a known ack policy.
4) Set replay window to sequences [start, end].
5) Run the consumer worker with controlled concurrency.
6) Wait until the worker reports it processed the window.
7) Assert KV values and projection outputs.
8) Repeat with a different window and with duplicate events.

{
  "envelope": {
    "schemaVersion": "number",
    "eventId": "string",
    "correlationId": "string",
    "occurredAt": "string"
  },
  "payload": {
    "type": "object",
    "required": ["orderId", "amountCents"],
    "properties": {
      "orderId": {"type": "string"},
      "amountCents": {"type": "integer", "minimum": 0},
      "currency": {"type": "string", "optional": true}
    },
    "compatibility": {
      "v2_additions": ["currency"],
      "breaking_changes": []
    }
  }
}

function handleEvent(event, kv, processedStore):
  current = kv.get(event.workflowId)
  if processedStore.contains(event.eventId):
    assert kv.get(event.workflowId) == current
    return ACK

  expected = applyTransition(current, event.step)
  if expected is INVALID:
    assert kv.get(event.workflowId) == current
    return NACK

  kv.put(event.workflowId, expected)
  processedStore.add(event.eventId)

  actual = kv.get(event.workflowId)
  assert actual == expected
  return ACK