Dead letter queue

Stream Trace builds a dead letter queue (DLQ) from JetStream advisory events. When repeated delivery attempts reach MaxDeliver, or an application terminates a message, JetStream emits a terminal advisory; Stream Trace captures it as an actionable case.

Setup: the advisory stream

Stream Trace reads terminal advisories from a capture stream named STREAM_TRACE_DLQ by default. You have two options:

• Let Stream Trace create it - set ST_DLQ_ADVISORY_STREAM_AUTO_CREATE=true, and Stream Trace creates or updates the advisory stream when the indexer role starts.
• Manage it yourself - leave auto-create off and create the stream with the exact terminal advisory subjects:

nats stream add STREAM_TRACE_DLQ \
  --subjects='$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>' \
  --subjects='$JS.EVENT.ADVISORY.CONSUMER.MSG_TERMINATED.>' \
  --storage=file \
  --retention=limits \
  --replicas=1 \
  --max-age=168h \
  --no-allow-direct \
  --defaults

The command matches Stream Trace’s default managed configuration: file storage, LimitsPolicy retention, one replica, and seven-day retention. Adjust replicas for your cluster and keep --max-age aligned with ST_TRACE_OBSERVATION_RETENTION_MS. The NATS CLI credentials running this command need stream-create permission; the Stream Trace NATS user does not need that permission when auto-create remains off.

Override the name with ST_DLQ_ADVISORY_STREAM=<name> if needed.

Without an advisory stream, Stream Trace has no terminal events to capture and the DLQ will stay empty.

Changes required in application consumers

The advisory stream alone does not make a consumer produce useful DLQ cases. Application-owned consumers must use an acknowledgement policy that represents processing outcomes. Configure a finite MaxDeliver when you want a delivery-exhaustion case:

{
  "ack_policy": "explicit",
  "max_deliver": 5
}

• ACK messages only after successful processing.
• NAK or allow AckWait to expire for retryable failures.
• terminate a message for a permanent failure when your client supports Term.

Term can produce a message-terminated case independently of MaxDeliver; a finite MaxDeliver is specifically what produces a max-deliveries case after repeated retryable failures.

JetStream emits the max-deliveries or message-terminated advisory; Stream Trace does not change these application consumer settings. See the full application integration checklist.

Working with cases

The DLQ view summarizes cases that need action, plus those already retried, discarded, or whose source message is no longer available. You can filter by:

• status (open, failed action, retried, discarded, unavailable),
• event type (max-deliveries or terminated),
• time range, stream, consumer, and subject,
• and payload (full-text / JSON path, when the payload index is enabled).

Open a case to see the source message when it is still retained, terminal advisory, payload and headers, and remediation timeline.

Recovering messages

From a case you can:

• Retry - republish the original message or publish a new copy to the target consumer’s configured replay subject.
• Resolve - close the case without reprocessing. This preserves the source message in its JetStream stream.

Both work on a single case or in bulk. Retrying needs dlq.manage; viewing needs dlq.read. Every retry or resolution outcome is recorded, and the DLQ audit view (requires audit.read) shows the history with actor, time, and result.