which version of SpiceDB are you running
# spicedb
y
which version of SpiceDB are you running and how many permissions are you checking in `CheckBulkPermission`s at once?
k
Spicedb version is 1.38.1. 12 permissions in first call and 6 in second
y
hmm. there have been some performance improvements in intervening versions, but the specific thing I was thinking of probably isn't in play. how deep is the hierarchy in the average permission check?
k
In the current load testing we are testing only first level. So, first level must resolve true. But 5-10% of test payload resolves false forcing SpiceDB to walk the full hierarchy. In this case we usually have 2-3 levels nesting
y
we haven't tested aurora serverless as a backing store to my knowledge. most of the deployments i've seen use provisioned instances - it's what we run in our own infrastructure and what i've seen from most community users. my understanding is that serverless is designed for variable workloads, and SpiceDB isn't really a variable workload. have you compared against AWS RDS or one of the provisioned aurora offerings?
which is to say that nothing you're doing from a SpiceDB load standpoint seems particularly out of the ordinary, the results you're getting are unexpected, and the thing that's sticking out to me right now as being different is the use of aurora serverless
k
We actually started with AWS RDS, but then switched to Aurora because of its capability to scale dynamically. Aurora lets to configure minimum and maximum capacity, by setting min capacity we can mimic AWS RDS fixed instance size. According to AWS documentation there is no difference between AWS RDS and Aurora in terms of performance and we kinda proved this on our syntethic tests. We didn't run the currect load testing on AWS RDS, but if you think it's make sense, we can do this. If we hit database difference, based on your experience are there any difference in performance between Spanner, Cockroach and Postgres? Does it make sense to consider to switching off Postgress to something else to gain more performance (of course paying the same cost)?
j
please update to the latest version of SpiceDB and try again
there are fixes in Bulk checking
y
> but the specific thing I was thinking of probably isn't in play i was wrong on this point
k
You mean this commit?https://github.com/authzed/spicedb/commit/5c799828458c881d1bd3b40f0709d72d1da9ab25 But I don't see that commit in the latest release? https://github.com/authzed/spicedb/releases/tag/v1.41.0 Should I still upgrade to v1.41.0?
j
its not released yet
you'll need to use HEAD
I should have been more specific when I said "latest version"
y
and to use head you'll use the
spicedb-git
image: https://hub.docker.com/r/authzed/spicedb-git/tags
(we also publish to quay and gcr depending on where you're pulling from)
(and you can override the
imageName
in the operator if that's how you're using it)
k
I see. Thanks. Let me try
I switched to AWS RDS and one advantage that I found is consistent response due to predictable and fixed instance size. WIth aurora many queries were timed out due to exhausting connections pool, but after several running Aurora scales the instance and gains more connections. I conduct my load testing using different spicedb tags, my results and opinion I posted in this comment https://discord.com/channels/844600078504951838/1350114521594986567/1350809224564834377. So, following my conclusion I think the problem in SpiceDB schema and how many quries we must send to resolve all permissions. Or maybe you conducted load testing against different database, such as Spanner or CockroachDB? I'm wondering maybe queries to Spanner or CockroachDB are way faster and optimized to handle more concurrent requests? Do you have any opinion on this?
j
you need to give a connection pool large enough to handle your queries
it doesn't matter which datastore is used
if SpiceDB has to wait for a connection, it will take longer to process
generally we recommend scaling the number of connections in the pool until the wait time metric goes to 0
k
Should I change any settings in spicedb? I found only 4 parameters that manage datastore pool connections
datastoreConnPoolReadMaxOpen
,
datastoreConnPoolReadMinOpen
,
datastoreConnPoolWriteMaxOpen
and
datastoreConnPoolWriteMinOpen
. Are any other parameters that help me decreasing the wait time to 0?
j
There are others for read replicas if you use them
8 Views