StreamTrace BETA
Documentation menu

Troubleshooting

Troubleshooting & FAQ

Fixes for the most common setup and usage issues.

Start with the readiness probe - GET /api/health/ready reports the registered database, NATS, and DLQ readiness checks. It does not certify every background worker or prove that historical indexing has completed.

Can’t connect to NATS

  • Confirm ST_NATS_URL is reachable from the container and that JetStream is enabled on the server.
  • If your cluster requires auth, encode it in the URL (nats://user:pass@host:4222) or use a token - see Connecting to NATS.
  • For TLS, use the tls:// scheme and make sure the server certificate is trusted by the container.

No streams show up

  • If auth is on, you only see streams owned by your teams. Assign stream ownership and give yourself a team-scoped role.
  • instance_admin sees all streams - use it to confirm Stream Trace can read NATS.
  • ST_INDEXER_STREAM_EXCLUDE only skips historical indexing. It does not hide a stream from inventory; missing streams indicate NATS permissions or Stream Trace ownership/RBAC instead.

WorkQueue message search reports “unsupported”

This is expected. Stream Trace automatically detects WorkQueuePolicy, does not create stream_trace_message_search_index, and keeps message history, payload search, and trace-header search disabled. Operational stream and consumer views, pending/redelivery metrics, and advisories remain available. A direct view of currently retained messages is labeled as a point-in-time snapshot. See Retention-policy safety.

If application consumers break or messages disappear on an InterestPolicy stream, disable the indexer and review its retention effects. Exclude the stream unless you intentionally accept Stream Trace as another interest holder.

Payload search is disabled or “syncing”

Payload search needs the payload index, which is off by default. Set ST_PAYLOAD_INDEXING_ENABLED=true and wait for the indexer to backfill. See Indexing & performance.

The DLQ is empty

DLQ cases come from JetStream advisories captured via an advisory stream. Either set ST_DLQ_ADVISORY_STREAM_AUTO_CREATE=true, or pre-create the advisory stream yourself. See Dead letter queue.

OIDC login fails

  • The callback_url in your provider config must exactly match what’s registered with the IdP, and its path must be /api/auth/oidc/<id>/callback.
  • Make sure the email_claim and groups_claim match the claims your IdP issues.

Login works but sessions don’t stick / CSRF errors

  • Over plain HTTP, set ST_AUTH_COOKIE_SECURE=false so cookies are stored.
  • Behind a cross-site setup, review ST_AUTH_COOKIE_SAME_SITE.
  • State-changing API calls must include the CSRF token from login.

I can see messages but not their payloads

Full payload access requires the streams.messages.read_sensitive permission (the sensitive_viewer or stream_admin roles). See Roles & permissions.

Performance on very large streams

  • Use the Index message source rather than reading live from NATS.
  • Run on PostgreSQL and tune the indexer batch size and concurrency.
  • Exclude streams you never search from the payload index.

Security & data handling

Stream Trace is self-hosted. Message data is read from NATS; indexes, throughput samples, DLQ cases, users, and the audit log are persisted in your database. Optional OIDC and SMTP integrations connect to configured endpoints. Stream Trace manages its own indexing and advisory consumers, can optionally manage its advisory stream, and publishes new messages for authorized publish, replay, and retry actions. It does not administer application-owned stream or consumer configuration.

Collecting diagnostics for support

There is no automatic crash upload or support-bundle command yet. Collect a bounded window of logs and both health responses locally:

docker compose logs --since=30m stream-trace
curl --fail http://127.0.0.1:4000/api/health/live
curl --fail http://127.0.0.1:4000/api/health/ready

For Kubernetes, use kubectl logs --since=30m deployment/<name> and the same health endpoints through a port-forward. Review output before sharing it. Logs can include request URIs, and search query values may contain sensitive message data. Never send environment files, NATS or database URLs, JWT keys, cookies, payloads, headers, or raw database files unless support explicitly requests a specific redacted artifact through an agreed secure channel.