Trace Collection Through OpenTelemetry SpiceDB #spicedb

Trace Collection Through OpenTelemetry

Jake

07/24/2023, 2:43 PM

I don't know anything offhand, but this is what Google brought me to: https://docs.datadoghq.com/opentelemetry/otel_tracing/

dorien2806

07/24/2023, 2:56 PM

I did try that, but it seems that the opentracer package mentioned is in maintenance mode and not really supported anymore (long live up to date documentation :S). I did not get it working alas.

yetitwo

07/24/2023, 3:09 PM

@dorien2806 we've got it working in our system

yetitwo

07/24/2023, 3:10 PM

the big thing is that you need to set up spicedb to send traces to datadog, and you need to set the trace propagation method in spicedb to b3, which is one of the kinds of propagation that datadog supports

yetitwo

07/24/2023, 3:11 PM

Copy code

// SpiceDB has OTEL tracing built into it; these values hook it up to
    // our datadog sidecar.
    { "name" : "SPICEDB_OTEL_PROVIDER", "value" : "otlpgrpc" },
    // Datadog uses a proprietary trace propagation format, but it also supports B3. This is what
    // allows us to associate traces between our app and SpiceDB.
    { "name" : "SPICEDB_OTEL_TRACE_PROPAGATOR", "value" : "b3" },
    { "name" : "SPICEDB_OTEL_ENDPOINT", "value" : "localhost:4317" },
    { "name" : "OTEL_EXPORTER_OTLP_ENDPOINT", "value" : "http://localhost:4317" }

yetitwo

07/24/2023, 3:12 PM

we use the datadog container sidecar to collect the traces

yetitwo

07/24/2023, 3:12 PM

and we found that we needed both the

OTEL_ENDPOINT

and the

EXPORTER_OTLP_ENDPOINT

to get things working for whatever reason

dorien2806

07/25/2023, 11:41 AM

@yetitwo thanks for the response and example, it does seem to do something now. Did you have to add anything to your traces or logs to ensure datadog picks it up? https://docs.datadoghq.com/tracing/other_telemetry/connect_logs_and_traces/opentelemetry/?tab=go talks about translating the trace_id and span_id into the datadog format (and also adds dd.service, dd.env, dd.version attributes). Did you have to do anything like this or does it work out of the box? Nb. now I knew what to look for I also found the example in this pull request (though it uses jaeger instead of datadog) https://github.com/authzed/spicedb/pull/1165

yetitwo

07/25/2023, 1:44 PM

once I got the correct endpoints and trace propagation mechanism it seems to have just worked

yetitwo

07/25/2023, 1:45 PM

there wasn't any manual rewriting of things

yetitwo

07/25/2023, 1:45 PM

i think in our setup it's the sidecar that sets the dd tags

dorien2806

07/25/2023, 3:00 PM

Ah, what image/config did you use for the sidecar?

yetitwo

07/25/2023, 9:48 PM

https://docs.datadoghq.com/containers/docker/?tab=standard and then the configuration is more specific to your company and its infra than it is to spicedb

dorien2806

07/26/2023, 12:24 PM

Thanks for the information, I finally got it working on a plain spicedb instance (after some tweaking of environment variables and such). I made an example docker compose file in case anyone else has the same problem in the future https://github.com/dorienmc/spicedb-datadog-tracing.

dorien2806

07/27/2023, 3:05 PM

One last question; is it also possible to tell spicedb that it should add the traceid and spanid to its logs? (So logs and traces can be connected https://docs.datadoghq.com/tracing/other_telemetry/connect_logs_and_traces/opentelemetry/?tab=go)

brandon.craig

08/11/2023, 4:30 PM

@dorien2806 did you find a way to add taceid and spanid?

21 Views

Previous Next