Thank you for the quick response
# spicedbt
tilting_irons
10/02/2023, 5:03 PMThank you for the quick response!
Single request performance is good, up until 300-500 RPS we see latency increasing dramatically.
We are suspecting that latency comes from the OTEL collector due to dropped traces due to high load, wondering if it is possible you could share the monitoring setup/configurations used in your load testing please
v
vroldanbet
10/02/2023, 5:32 PMwhat database are you using? in either case you want to tune the connection jitter. As soon as connection max lifetime is hit, they will be closed and cause contention. Jitter helps making sure this does not happen all at the same time.
vroldanbet
10/02/2023, 5:33 PMI doubt otel is a problem, we've run higher load tests with OTEL sampling at 1%
vroldanbet
10/02/2023, 5:33 PMit may start to become a problem a very high RPS, when you probably will need to dial down the sampling rate even further
t
tilting_irons
10/02/2023, 5:47 PMThank you for the answer, really appreciate it!
We are using Postgres, with the default
SPICEDB_DATASTORE_CONN_MAX_OPENtilting_irons
10/02/2023, 5:52 PMBy the way, we see max connections to the RDS, so if we have 8 SpiceDB pods at 50 connections each then it should come to 400, however, the RDS (for this compute instance) caps at ~240
https://cdn.discordapp.com/attachments/1158449143698686052/1158461586504167535/image.png?ex=651c54ed&is=651b036d&hm=1c8faf1ae97d72ac70abc443cff7ff9269b793432d08149aedc475c80caf2262&
v
vroldanbet
10/02/2023, 6:03 PMmax connections means that SpiceDB may open up to that number. If you really want it to have all the connections all the time, you have to set it with the
min_connectionsvroldanbet
10/02/2023, 6:04 PMjitter depends on lifetime, if you haven't changed lifetime it should be by default 30 minutes, so I suggest to set jitter to 30 minutes
vroldanbet
10/02/2023, 6:05 PMthere is nothing really worth about our OTel setup other than we are using Jaeger and we use the default 1% sampling defined in SpiceDB
2 Views