interesting SpiceDB #spicedb

interesting

Andrii

06/28/2025, 9:41 PM

interesting for variables like SPICEDB_OTEL_PROVIDER, SPICEDB_OTEL_ENDPOINT, SPICEDB_OTEL_INSECURE, SPICEDB_OTEL_SERVICE_NAME it all works well with

SPICEDB

prefix I'll check OTEL prefix for sample ratio and will let you know here

yetitwo

06/28/2025, 9:42 PM

yeah that's because there are a few flags for otel that are defined in our code:

Copy code

Observability Flags:
      --otel-endpoint string           OpenTelemetry collector endpoint - the endpoint can also be set by using enviroment variables
      --otel-insecure                  connect to the OpenTelemetry collector in plaintext
      --otel-provider string           OpenTelemetry provider for tracing ("none", "otlphttp", "otlpgrpc") (default "none")
      --otel-sample-ratio float        ratio of traces that are sampled (default 0.01)
      --otel-service-name string       service name for trace data (default "spicedb")
      --otel-trace-propagator string   OpenTelemetry trace propagation format ("b3", "w3c", "ottrace"). Add multiple propagators separated by comma. (default "w3c")

yetitwo

06/28/2025, 9:42 PM

it's a little messy 😓

Andrii

06/28/2025, 9:43 PM

yeah 🙂 that's why I wondered why SPICEDB_OTEL_SAMPLE_RATIO is so special :)))

Andrii

06/28/2025, 11:40 PM

@yetitwo using the documentation https://opentelemetry.io/docs/languages/sdk-configuration/general/ I've tried to set 2 new env vars : OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG they are ignored, just like SPICEDB_OTEL_SAMPLE_RATIO SPICEDB_OTEL_SAMPLE_RATIO is set to 0.01 by default (I can see that in logs) even though I'm getting 100% of traces

Andrii

06/28/2025, 11:42 PM

also, in this file that you shared - only 1 otel variable is not set with SPICEDB_ prefix. Another one is still there - https://github.com/authzed/examples/blob/main/tracing/docker-compose.yaml#L21

yetitwo

06/29/2025, 12:40 AM

that sounds like a bug

yetitwo

06/29/2025, 12:40 AM

would you mind opening an issue? i can have a look at it this week

Andrii

06/29/2025, 10:58 AM

thank you, @yetitwo ! I'll try local setup too and then will open an issue

Ben Simpson

06/30/2025, 4:54 AM

Coincidentally I think we just noticed this issue too. Using the default of 0.01 our traces are through the roof. I'm just rolling out this change to see if it makes any difference but based on this thread I don't expect it to:

Copy code

SPICEDB_OTEL_SAMPLE_RATIO = 0.001
OTEL_TRACES_SAMPLER = "parentbased_traceidratio"
OTEL_TRACES_SAMPLER_ARG = 0.001

Andrii

06/30/2025, 7:01 PM

@Ben Simpson yeap! Glad it is reproducible somewhere else. What version of SpiceDB are you using and where is it deployed?

Ben Simpson

06/30/2025, 10:12 PM

Having done some testing with the tracing example you linked above,

SPICEDB_OTEL_SAMPLE_RATIO

definitely seems to be working. I think in our case we must be doing 100% sampling elsewhere which is causing the

ParentBasedSampler

to sample everything

Andrii

06/30/2025, 11:03 PM

@Ben Simpson I've tried too, locally works. But in CloudRun(GCP) it seems to be ignored. I'm trying to figure out where

Ben Simpson

07/01/2025, 3:47 AM

Is it possible you have a parent that's tracing 100% like we do? Or are you seeing it even with direct calls to SpiceDB bypassing any other instrumented services?

Andrii

07/01/2025, 11:16 AM

@Ben Simpson I'm making direct calls from zed CLI. I'll try to build some similar flow locally. But so far looks very weird. I bet it might be something to do with GCP infra

9 Views

Previous Next