Hey guys, we've been using SpiceDB in production f...
# spicedb
s
Hey guys, we've been using SpiceDB in production for a few months now at https://www.veed.io and have gradually been migrating our authorization data to it. We've attached a Postgres datasource, which now has a size of about 7GB and we've run into a bit of an unexpected problem. The easiest way to explain it is by showing some metrics. As our data size grew from about 2-3 million datapoints to >10 million, we started noticing significant slowdowns on some user accounts, but not on others. Following the documentation, we had implemented ZedToken caching and use
atLeastAsFresh
consistency for a number of our users - which we found to be the exact users having issues with significant slowdowns. Our investigation into the issue resulted in the discovery that making use of ZedTokens actually slowed down requests by a factor of 100. We've now set everything to be fully consistent, and as you can see in the metrics, slowdowns have stopped, but this isn't the behaviour I'd expect. Is this something you've run into before? I'd be happy to create an issue on Github and/or provide more metrics to help you gain insights here. I'd expect full consistency to start causing problems if our data set grows by another 50 million data points