https://authzed.com logo
Join Discord
Powered by
It looks like you re experiencing a
# spicedb
c
It looks like you're experiencing a timeout for dispatch. Perhaps because SPICEDB_DISPATCH_UPSTREAM_ADDR is invalid.
a
SPICEDB_DISPATCH_UPSTREAM_ADDR will take name of the deployment or name of cr?
c
How many replicas of SpiceDB are you running?
a
6
c
SPICEDB_DISPATCH_UPSTREAM_ADDR should be set to the service
a
Zooming out a bit, the symptoms we see are that, over time, there is more and more dispatch happening, and requests take longer and longer until it kind of collapses after about 10-15 minutes of load, see this graph:

https://cdn.discordapp.com/attachments/1139540955012288624/1139567572925304862/image.png▾

This is running on eks with the operator, ec2 nodes, nlb load balancer (w/ cross zone load balancing)
some other metrics correlate, cache hit rate goes down:

https://cdn.discordapp.com/attachments/1139540955012288624/1139569206917083206/image.png▾

Dispatch latency goes up

https://cdn.discordapp.com/attachments/1139540955012288624/1139569414795165696/image.png▾

c
Is dispatch going through the NLB?
a
no, dispatch is configured to use the kubernetes endpoint lookup by the operator as far as i can tell
Copy code
- name: SPICEDB_DISPATCH_UPSTREAM_ADDR
          value: kubernetes:///spicedb-cr.spicedb:dispatch
spicedb-cr is the name of the service provisioned by the operator of type ClusterIP
Copy code
apiVersion: v1
kind: Service
metadata:
  name: spicedb-cr
  namespace: spicedb
  ownerReferences:
  - apiVersion: authzed.com/v1alpha1
    kind: SpiceDBCluster
    name: spicedb-cr
    uid: 1daa5968-75bc-481e-a7cf-06721082419d
a
@corkrean operator is setting name and namespaces on the DISPATCH_UPSTREAM_ADDR https://github.com/authzed/spicedb-operator/blob/main/pkg/config/config.go#L453
c
Yep, I'm not concerned about the URL anymore.
Is the client request load reasonably constant?
Ahh nm, I see that it is
e
What can you tell us about the requests that you're sending to spicedb? The traffic doesn't look constant based on the PermissionService RPS. And what datastore are you using as a backend? Without knowing more, you could just be hitting the limits of your backing database. If it doesn't doesn't have enough capacity to handle the uncached request load, dispatches won't be able to get answers from the DB fast enough and start timing out like you're seeing. And request patterns will determine how much of the load makes it to the DB itself.
a
We are using RDS with db.r6g.large
e
Postgres I assume?
And are you doing load tests or is this application traffic?
a
Yes postgres. And load tests.
A little about the traffic / schema: - it is deeply nested. we're testing the effects of hierarchies, and so this test is about 8+ levels of subproblems. - total qps was 800 in that example, but we've seen the behavior (gradually increasing dispatch calls until timeouts) even with low qps (e.g. 150) - load is distributed over mostly check calls (85%), some LS (5%) and some LR (10%) traversing the hierarchy.
Here is a graph of a long running test at 500qps. This one took about 2 hours until instability.

https://cdn.discordapp.com/attachments/1139540955012288624/1139601694016147536/image.png▾

image is ghcr.io/authzed/spicedb:v1.23.1
e
Are you writing data during the tests or is the dataset static?
a
Dataset is static. We're testing with a small subset of the data that is written currently
I believe right now we should have a few million relations, and we're probably exercising just like <0.1% of those if my math is right
e
And you don't see anything unusual on the RDS metrics?
I would also check the spicedb logs to see if there's any obvious connection issues, but I can't think of why that would change over time if the load is relatively static
a
Haven't looked at RDS to be honest, thanks for the suggestion. Will look there.
j
@Alec do you have hedging turned on?
If the amount and kinds traffic are both constant but dispatches are rising, that could be why: if things are slowing down, hedging might be hitting
a
We have very minimal config, so probably whatever defaults are set to; lemme see
j
if it is on, try turning it off
d
could someone tell me the config flag to disable hedging? it might really be the problem here, as the db logs are showing 
ERROR: canceling statement due to user request
- as per https://authzed.com/blog/spicedb-architecture it may likely be the hedging proxy, which i assume would be the "user" in the pg logs.
found it, nvm
4 Views
PreviousNext
Powered by