https://authzed.com logo
Join Discord
Powered by
tuning multi-region SpiceDB
# spicedb
v
Is there any way to get more granular tracing than this? Otherwise it seems like it seems to do nothing for... a while

https://cdn.discordapp.com/attachments/844600078948630559/1130975757498318959/image.png▾

This is testing against the CRDB dedicated cloud. Using the following operator deployment of spicedb:
Copy code
yaml
apiVersion: authzed.com/v1alpha1
kind: SpiceDBCluster
metadata:
  name: spicedb-dev
spec:
  channel: stable
  version: v1.18.0
  config:
    logLevel: debug
    datastoreEngine: cockroachdb
    datastoreTxOverlapStrategy: insecure
    replicas: 1
    otelEndpoint: opentelemetry-collector.monitoring.svc:4317
    otelInsecure: "true"
    otelProvider: otlpgrpc
    otelSampleRatio: "1"
  secretName: dev-spicedb-config
j
are SpiceDB and CRDB in the same region?
v
No, CRDB is multi-region, but normal queries take at most 100ms
j
are normal queries opening transactions though?
SpiceDB has to do so to type validate writes
so its opening a transaction, reading schema, type checking, then writing the relationship(s)
we can look into adding more granular tracing for writes
v
the one by uptrace is quite nice 
github.com/uptrace/opentelemetry-go-extra
the other writes are not transactions, but do make a full row lock
v
>the one by uptrace is quite nice github.com/uptrace/opentelemetry-go-extra looks nice although does not seem to support pgx. I found https://github.com/exaring/otelpgx
new relic folks also have one: https://github.com/newrelic/go-agent/pull/592
just implemented it, seems to be working fine with CRDB

https://cdn.discordapp.com/attachments/1130975757737398382/1131194708564115566/image.png▾

Once it lands, you can grab the Docker container from the nightly builds: https://hub.docker.com/r/authzed/spicedb-git
v
Built it manually as wanted to get to the bottom of this asap... the tracing is there, but it doesn't seem like it's going to help much. I think it might be actually waiting on either 1. Opening a connection to CRDB or 2. Waiting for an open connection to start a TX?

https://cdn.discordapp.com/attachments/1130975757737398382/1131370553140330628/image.png▾

thank you btw for the quick response
j
looks like it
which is odd
SpiceDB pre-fills a pool of connections to ensure it doesn't have to wait
v
yep, it takes a few secs to become healthy on boot
right now at most I can push ~70 write tx/s
v
It almost feels like something related with connection pooling. Just to rule it out, could you run with SpiceDB 1.20? I believe we introduced the new CRDB aware balancer in 1.21, so it would allow us to rule out a problem with the connection pool and Multi-Region clusters
j
could also just be something with CRDB serverless; it has to scale itself behind the scenes, is my understanding
v
They are testing with dedicated now, afaiu
j
ah
v
The fact y'all mentioned it took a hot minute to boot reminded me of a regression we had recently with the connection pool, which is fixed in main. So if it's taking long to boot it could mean it's taking long to pre populate the Conn pools
v
I built from the 
add-pgx-otel-tracing
branch which I assume is as new as it can be
j
I wondering if this is a gRPC issue
because the time between opening the transaction and starting work is quite small
how are you invoking this?
v
It's a custom bench tool for our own services
j
hmmm
do you see the same delays when writing rels via 
zed
?
if not, how are you instantiating the client in the bench tool?
v
I have not attempted that, I would have to write some tooling around it then
Copy code
go
    client, err := authzed.NewClient(
        viper.GetString(configSpiceDBEndpoint),
        grpc.WithTransportCredentials(insecure.NewCredentials()),
        grpcutil.WithInsecureBearerToken(viper.GetString(configSpiceDBToken)),
        grpc.WithUnaryInterceptor(otelgrpc.UnaryClientInterceptor()),
        grpc.WithStreamInterceptor(otelgrpc.StreamClientInterceptor()),
    )
j
and the endpoint is a real gRPC server?
not a buffcon?
v
yes, it's a real connection
j
do you see the same delays when writing relationships not at scale?
and against how many SpiceDB nodes is this?
v
this is testing against a single SpiceDB instance
j
okay
you're likely exhausting the number of DB connections available
if you're running 70 QPS of writes
v
that's what I was thinking
j
there is a flag for increasing them
v
will try that
what is the default? 10?
j
yeah
Copy code
--datastore-conn-pool-write-max-open int                        number of concurrent connections open in a remote datastore's connection pool (default 10)
      --datastore-conn-pool-write-min-open int                        number of minimum concurrent connections open in a remote datastore's connection pool (default 10)
v
Neither increasing the SpiceDB instance count to 5, or increasing connection count to 100 made any difference it seems, which can't be right...
j
no, that doesn't seem right at all
granted, if your tester is only making a single grpc connection
it will only be talking to a single SpiceDB
v
it talks to another service which makes multiple requests
(draco in the screenshots)
and as expected starting a second bench run in parallel halves the throughput
j
is draco opening a single connection
v
it shouldn't be but I will investigate that, as that seems the case at the moment
I guess it was, just wrote this quick and dirty pool https://gist.github.com/Vilsol/f54223bfcf899101d92b183a491a8d60 And I made sure I initialize it in both the bench and the calling service, which now initializes like this:
Copy code
go
endpoint := viper.GetString(configSpiceDBEndpoint)
options := []grpc.DialOption{grpc.WithTransportCredentials(insecure.NewCredentials()),
    grpcutil.WithInsecureBearerToken(viper.GetString(configSpiceDBToken)),
    grpc.WithUnaryInterceptor(otelgrpc.UnaryClientInterceptor()),
    grpc.WithStreamInterceptor(otelgrpc.StreamClientInterceptor()),
}

clientPool := grpc2.NewClientPool(100, endpoint, options...)
client := &authzed.Client{
    SchemaServiceClient:      v1.NewSchemaServiceClient(clientPool),
    PermissionsServiceClient: v1.NewPermissionsServiceClient(clientPool),
    WatchServiceClient:       v1.NewWatchServiceClient(clientPool),
}
But all of that made no difference
j
might have to add some travcing to the WriteRels call
and see when the actual function gets hit
v
@Vilsol circling back here, we run some tests in multi-region, and reproduced the latencies you were observing. This is related to the logic put in place in SpiceDB to address the enemy-problem due to Cockroach limitations (Spanner does not have the same problem). If you set the datastore overlap strategy to insecure, you should see a reasonable throughput. If you want to learn more about the overlap strategy, check this document out https://github.com/authzed/spicedb/blob/7977eb6366ca73275322a93a96aefcc2568a0575/e2e/newenemy/README.md
we run some more tests and did some further tweaks, and managed to obtain single-region performance in a multi-region setup. Please note this the implication that it weakens the new-enemy problem-addressing properties of SpiceDB. - enable insecure overlap - disable datastore stats - set the 
relation_tuple
table with locality 
REGIONAL BY ROW
- set the 
namespace_config
table with locality 
GLOBAL
We obtained <20 p95 read and <30ms p95 writes Again, I would like to emphasize that this setup does not address the new enemy problem, but maybe you are cool with that
17 Views
PreviousNext
Powered by