https://authzed.com logo
Join Discord
Powered by
Hi team! Could you help me to find which
# spicedb
k
Hi team! Could you help me to find which parameters have an impact on throughput? I need to stream 50+ mln relationships. I improved IOPS for RDS, but it grows no more than 1.5 IOPS and CPU seems also okay and grows no more than 30%. The next assumption is spicedb or my relay service. Starting from the latter, I'm almost sure that rely service which is taken messages from Kafka can make requests to SpiceDB on the rate ~1k RPS. When I increased number of replicas of SpiceDB seems like I managed to increase throughput, but I don't want to grow only horizontally. Could you help me to find which parameters can improve throughput? So far I found only number of connections in the pool for read and write operations, but maybe there are other factors? Thanks!
v
I'm going to assume, becuase the description is not clear, that you mean writes, not reads. Which API are you using to ingest the 50m relationships?
Also which version of SpiceDB are you using how much capacity do you have allocated to RDS, and how many replicas of SpiceDB and CPUs allocated to it
k
> Which API are you using to ingest the 50m relationships? In most cases 
WriteRelationships
API > Also which version of SpiceDB are you using 1.38.1 > how much capacity do you have allocated to RDS That's an open question, I know the much is better, but we still didn't reach the higher percent of resource utilization. Rught now it's running t3.medium with gp3 storage type on Postgres > how many replicas of SpiceDB I experimented with 2, 5 and 10, but I'm open to suggestion here. Right now CPU is 50m for requests and 200 for limits. Does CPU has significant impact on throughput?
v
I'd suggest using BulkImport for seeding
200 millicores limit?
k
Could you clarify please? You mean switch WriteRelationship API to BulkImport API? I think it's not an option to us, since our relay service is configured in a way to replicate old records from the database and stream new changes via CDC. So, we want to use the same service with the same functionality for both cases. Any chance to improve throughput of SpiceDB by another way? As I understand, increasing CPU might help us, but does SpiceDB have any other config? Thanks!
v
I don't understand why you can use one API, but not the other. They are functionally very similar, but bulk API allows for higher throughput.
I'd say you should be able to ingest a lot with 1 or 2 vCPU per replica, 200millicores seems very low
you should also keep an eye at write connection pools, make sure they are not starved
there is no "setting to increase throughput". SpiceDB scales with compute, but writes mainly scale through the underlying's datastore throughput
the recommendation is scale vertically, and then when you reach a certain point, scale horizontally
a 3-replica cluster is the very starting point if you are running in a multi-AZ environment, assuming 3 different AZs
make requests = limits to eliminate throttling
and also add memory requests and limits
(that helps SpiceDB calculate an optimal number for GOMAXPROCS and GOMEMLIMIT)
k
thanks for such insights
21 Views
PreviousNext
Powered by