SpiceDB is an open source, Google Zanzibar-inspired, database system for creating and managing security-critical application permissions.

SpiceDB

from spicedb. We've removed pgbouncer for now and just trying to figure out where the bottle neck is

after rolling the kubernetes deployment, it'll start getting some SQL errors and then some it'll hit max connections on PG.

I'll scale spicedb to 1 pod, let it run for a bit, then scale it back up. it seems to do better there

sure, a cold cache will require more requests

but the question is: why would pgbouncer result in query timeouts

unless it is trying to reuse the connections and waiting on them

oh, sure, we may have a wonky setup. we'll look at that next

but if we just let our stress test run, spicedb slowly gets better over time with out changing anything else

the caches should only cover a few seconds

CPU usage is an inverse curve of that. as is PG access locks

if your stress test is hitting the cache, it could result in the CPU lowering

or Postgres is possibly getting getter indexes, but that seems unlikely

What is impact if not using zedtoken at all, we are running a facade layer on top of spicedb as user to group mapping lives in some different system, this mapping Can't be moved to slicedb, in spicedb we  have group to permission mapping inside spicedb, this works very well for us but not able to use zedtoken.

if you don't use a zedtoken at all, there is a chance when you issue a check (when at `minimize_latency`) that you get backed a cached result

which means if someone just removed access to something, you might say they still have access for a few seconds

So what do you suggest in such cases instead of minimize_latency ?

No, as the system that holds this it does a lot of auditing and group assignment workflows etc. Due to compliance reason can't move it .

Ya, restarting spicedb really seems to tank performance

then you need to investigate what your tests are doing, exactly

it will take a short period of time to acquire new connections, but that's measured in seconds

and the cache will take some time to fill, but that's not going to apply after a few seconds

unless your quantization window is extremely long

we're just sending requests as fast as we can

you're probably overloading your cluster or your DB

the cache depends on *what* you're requesting

the db is pretty fine. Its a sustained test that does the same work in a loop. The only thing that changed was we restarted spicedb. The test never stopped.

if you're just firing off the same check over and over

its a fixed data set that we iterate over

then its likely due to the caches being empty

when you start up, we have to retrieve everything

which is likely taking some time and then you're piling up requests

which still doesn't explain why it gets *slightly* faster over time

don't throw a wall of traffic onto a cold cluster

I don't think that is practical in a kubernetes environment. things restart

not the entire cluster unless there was a full outage

a deployment entirely failing means your entire cluster is gone

anything that changes the pod spec of a deployment will restart everything.

if that's the case, you're showing that the cache is handling traffic overhead solely because your load happens to be widely cachable

real world traffic will likely have less caching

and that means you'll need more nodes to handle traffic

how many checks per second would you expect a single pod to be able to handle?

it depends entirely on the size of the nodes

a check could be a single relationship load

```
definition access_token {}

definition account {
  relation participant: user#member | service_account#member
  permission member = participant
}

definition user {
  relation inheritor: user | access_token
  relation access_token: access_token
  permission member = inheritor
  permission token = access_token
}

definition service_account {
  relation inheritor: service_account | access_token
  permission member = inheritor
}

definition role {
  relation inheritor: user#member | user#token | service_account#member
  relation parent_role: role#member
  relation account_role: account
  permission member = inheritor + parent_role
  permission parent = parent_role
  permission account = account_role
}

definition resource {
  relation reader: user#token | role#member | service_account#member
  relation writer: user#token | role#member | service_account#member
  relation manager: user#token | role#member | service_account#member
  permission manage = manager
  permission write = manage + writer
  permission read = write + reader
}
```

that's a recursive definition - it could be N levels deep

We check to see if a given token has read, write or manage permission on a given resource

which, in turn, can have many parent roles

so that could spread out to many dispatches

so it really depends on how deep those are

we arent doing any deep role testing in this.

if it is fairly flat, then you're likely simply overloading the nodes

The test data only goes 1 level of role inheritence. The tests check

what, exactly, is the data you're writing and testing

I'm not sure what kind of an answer you are looking for

the relationships you're writing and the check you're perform, concretely

but either way, I'm nearly certain you're simply overloading your nodes

if you're firing off 20-50RPS, and you have 4 CPU available, that means you're likely saturating the number of goroutines available for us to use

and since the dispatch sends requests for the same checks to the same nodes

what is the number of goroutines available?

(edited for clarify) concurrently *executing* goroutines will be limited by the number threads, which will be based on the CPU count

and as I said, I suspect your load test, by hitting the same checks, is creating a hotspot

the reason it gets better over time is because you're just hitting the cache more and more

I guess I'm looking a bit of operational guidance around tuning and sizing.

you'll likely want bigger nodes with more CPU cores, to start

and you'll likely want to change your load tester to not check the same things over and over; or at least add a bit more variance/randomness

is a shared cache on the table? a lot of the distributed DBs we've used in recent history have a disc cache or something. We'd run it on NVMe drives, etc.

we have some ideas around a shared L2 cache

we also have ideas around making it so the in-process cache can persist between restarts

whats the standard setup when you deploy a managed instance? what do you provision? Someone has an idea of what is "pretty good" for most production work loads

we usually work with the customer on that, but I'll get the node sizes we use

for now, I recommend trying nodes with more (v)CPUs

I still havent gotten back around to turning on the cluster dispatch. Is that something that helps in this kind of a situation?

do you have a load balancer in front of your cluster too?

then you need to either accept some lag on consistency or send `fully_consistent`, which will bypass the caching entirely and cause a performance impact

I will go with same, cache is non negotiable and our usecases are not that can't afford few seconds on consistency lag.