Application integration checklist

Stream and consumer monitoring works once Stream Trace can connect to JetStream. Tracing, enriched ACK observations, actionable DLQ cases, consumer-scoped replay, and metadata-based ownership require explicit changes in your applications or NATS configuration. Stream Trace reports missing setup; it does not silently rewrite these settings for you. Keep the reserved managed-consumer names stream_trace_message_search_index and stream_trace_dlq_observations out of your application configuration because Stream Trace owns their create/update and cleanup lifecycle.

Feature	Change outside Stream Trace	Required?
Cross-service trace correlation	Publishers attach and propagate one stable trace-ID header.	Required for traces.
Consumer/ACK trace enrichment	Consumers set SampleFrequency / sample_freq.	Optional; required for sampled ACK latency and consumer observations.
DLQ cases	Consumers use explicit ACK semantics; configure finite MaxDeliver for exhaustion or Term for an explicit permanent failure.	Required for the corresponding terminal advisory.
Consumer-scoped replay	Consumer metadata declares streamtrace.replay_subject; the stream captures it and the application routes it to the intended handler.	Required for targeted replay/retry.
Metadata-based team ownership	Stream metadata declares streamtrace.teams.	Optional alternative to provisioned or UI ownership.

1. Propagate a stable trace ID

Every publisher in a related message flow must attach the same correlation value to one of the headers configured in ST_TRACE_ID_HEADERS. The defaults are Nats-Trace-Id and X-Trace-Id.

Nats-Trace-Id: order-01JABC...

When a service consumes one message and publishes the next, copy the same header and value forward. Generating a new value at every service creates separate traces. Stream Trace performs an exact lookup for a known trace ID; it does not discover or inject IDs for the application.

2. Enable sampled ACK enrichment

Trace spans come from indexed messages. Consumer names, delivery counts, ACK status, and delivery-to-ACK latency require JetStream ACK metric advisories. Set the application consumer’s SampleFrequency (sample_freq in JSON/API configuration) to the percentage you want NATS to sample:

nats consumer edit ORDERS ORDERS_WORKER \
  --sample=100 \
  --force

100 provides complete ACK sampling and the highest advisory volume. Choose a lower percentage when that tradeoff is more appropriate. This is application-consumer configuration; Stream Trace only reads the resulting metrics.

3. Configure terminal failure behavior

For DLQ cases, application consumers need explicit acknowledgement semantics. Finite MaxDeliver is required for delivery-exhaustion cases:

{
  "ack_policy": "explicit",
  "max_deliver": 5
}

ACK only after success. NAK or allow AckWait to expire for retryable failures, and use Term for permanent failures where supported. Term independently produces a message-terminated case without exhausting MaxDeliver. NATS emits the corresponding terminal advisory captured by the advisory stream.

4. Route consumer-scoped replay

Republishing to an original subject can reach every matching consumer. To target a specific recovery path, add streamtrace.replay_subject to the intended consumer’s metadata:

nats consumer edit ORDERS ORDERS_WORKER \
  --filter='orders.process' \
  --filter='orders.worker.replay' \
  --metadata='streamtrace.replay_subject=orders.worker.replay' \
  --force

Replace orders.process with every existing filter for that consumer; the example adds orders.worker.replay to FilterSubjects rather than replacing the application’s normal delivery filters. An unfiltered consumer already receives the replay subject and does not need a filter change.

Then verify all four conditions:

1. The source stream captures orders.worker.replay directly or through an existing wildcard subject.
2. The application routes that subject to the same handler or recovery path used by ORDERS_WORKER.
3. The intended consumer is unfiltered or its FilterSubject / FilterSubjects includes the replay subject.
4. No unintended consumer filter also matches it.

Stream Trace publishes a new message to this replay subject with provenance and deduplication headers. It does not push the old message through the consumer or add the application routing for you.

5. Add optional stream ownership metadata

If you want NATS stream metadata to provide the ownership fallback, set streamtrace.teams to a comma-separated list of existing Stream Trace team slugs:

nats stream edit ORDERS \
  --metadata='streamtrace.teams=payments,platform' \
  --force

An Admin UI override or ST_AUTH_STREAM_OWNERSHIP_JSON assignment takes precedence. Stream Trace reads this metadata; it does not write it back to NATS.

6. Review indexing retention before production

The default indexer creates stream_trace_message_search_index only on eligible, non-excluded streams and ACKs messages after indexing. LimitsPolicy streams are the supported default. WorkQueue message history and search are explicitly unsupported; Stream Trace detects and excludes every WorkQueuePolicy stream automatically, so no application or exclusion-list change is required. Exclude InterestPolicy streams unless you explicitly accept the retention changes. See Connecting to NATS for the exact effects and exclusion settings.