Indexing & performance

Stream Trace runs a background indexer that records message metadata so listing, filtering, and counting stay fast on large streams. A separate, optional payload index powers content search.

Message indexer

The indexer is enabled by default and discovers new streams automatically. Key settings (full list in the Configuration reference):

It creates the durable pull consumer stream_trace_message_search_index on every eligible, non-excluded stream, reads messages through that consumer, and explicitly ACKs each message after indexing. WorkQueue streams are never eligible for indexing.

Variable	Default	Purpose
ST_INDEXER_ENABLED	true	Turn indexing on/off.
ST_INDEXER_STREAM_EXCLUDE	_*	Stream name patterns to skip.
ST_INDEXER_RETENTION_HOURS	168	Reserved; automatic indexed-row cleanup is not implemented.
ST_INDEXER_BATCH_SIZE	100	Messages per batch.
ST_INDEXER_MAX_CONCURRENT_STREAMS	4	Streams indexed in parallel.
ST_INDEXER_DISCOVERY_INTERVAL_MS	60000	How often new streams are discovered.

Retention warning. LimitsPolicy supports message history. Stream Trace automatically excludes WorkQueuePolicy streams and never creates a managed index consumer for them. Exclude InterestPolicy streams unless you intentionally accept Stream Trace as another interest holder that changes when messages become eligible for deletion. ST_INDEXER_STREAM_EXCLUDE matches stream names and lets you exclude additional streams.

WorkQueuePolicy support boundary

Message history isn’t available for WorkQueue streams. NATS removes messages when workers acknowledge them, so any scanner can miss messages processed between polls or while Stream Trace is offline. A partial index cannot be presented as trustworthy history.

For WorkQueuePolicy streams, the support boundary is:

• Unsupported: message indexing, historical message search, payload search, and trace-header search.
• Supported: stream configuration and state, consumer configuration, pending and redelivery metrics, and advisory observations.
• Optional snapshot: a view of currently retained messages can be useful only when clearly labeled as a point-in-time snapshot, not message history.

Enforced behavior

Stream Trace enforces this boundary automatically. It:

• never creates stream_trace_message_search_index for a WorkQueue stream,

• exposes an explicit stream capability:

  {
    "message_search": {
      "status": "unsupported",
      "reason": "work_queue_retention"
    }
  }

• returns a structured 422 from stream-scoped search instead of an empty result,

• discloses excluded WorkQueue streams in every global search result,

• and ensures partial WorkQueue index rows are hidden or removed so they cannot appear authoritative.

A future opt-in capture feature would need producers to dual-write, or NATS RePublish to copy messages into a separate LimitsPolicy stream. It must show a coverage start time and cannot backfill messages processed before capture began.

For high-volume deployments, raise ST_INDEXER_BATCH_SIZE and ST_INDEXER_MAX_CONCURRENT_STREAMS, and consider running dedicated indexer processes on PostgreSQL - see Scaling.

Payload index (search)

Payload indexing is off by default because it costs storage and write throughput. Turn it on to enable full-text and JSON path scalar message search:

ST_PAYLOAD_INDEXING_ENABLED=true

It extracts scalar values from JSON payloads within configurable limits:

Variable	Default	Purpose
ST_PAYLOAD_INDEX_MAX_BYTES	32768	Max payload size indexed.
ST_PAYLOAD_INDEX_MAX_JSON_DEPTH	12	Max JSON nesting depth.
ST_PAYLOAD_INDEX_MAX_JSON_SCALARS	512	Max scalar values per message.
ST_PAYLOAD_INDEX_MAX_SCALAR_TEXT_BYTES	1024	Max bytes per scalar value.
ST_PAYLOAD_INDEX_STREAM_EXCLUDE	(empty)	Streams to exclude from payload indexing.

After enabling it, the indexer backfills existing messages. Until a stream’s payload index is ready, payload filters are disabled for that stream and the UI shows a “syncing” state.

Tip. Exclude high-volume streams you never search with ST_PAYLOAD_INDEX_STREAM_EXCLUDE to keep index size and write cost down.

Because automatic retention cleanup is not implemented, database growth follows the streams and payload fields you index. Monitor storage and use tested database backups before enabling payload indexing on high-volume streams.