Further on this, do you have any useful metrics/benchmarks on spicedb performance when running it with postgres?

You want to look at the health of connection pools, making sure your cluster is not spending too much time waiting on connections to become available. The upcoming 1.28.0 should include further improvements on request deduplication so that can give you extra headroom in a busy cluster. You also want to play around with

datastoreRevisionQuantizationMaxStalenessPercent

parameter. Increasing will also help in situations where there is contention on the connection pool. We also recommend running with PG >= 15, as it is necessary to get the query planner to select some of the new indexes. We've also observed some minor performance improvement by moving to it. We also definitely recommend making the jitter basically same or close to max connection lifetime, as it helps amortize the impact of connections being removed from the pool due to reaching max lifetime. There is exhaustive connection pool metrics, please refer to the

/metrics

endpoint to see the documentation around each metric

I am also running SpiceDB with postgres and here's my insight on metrics: 1) regarding pgxpool metrics, there are no histogram for acquire duration time. the closest you can get is average (with

rate(pgxpool_acquire_duration_ns{pool_usage="read"}[$__rate_interval]) / rate(pgxpool_acquire_count{pool_usage="read"}[$__rate_interval])

). keep that in mind when analyzing

spicedb_datastore_query_latency

which is a histogram 2) there seem to be a 3x difference between

sum(spicedb_datastore_query_latency_count{operation="QueryRelationships"})

and

sum(pgxpool_acquire_count)

which I can't get my head around. QueryRelationships doesn't use cache (does it), so whenever

sum(spicedb_datastore_query_latency_count{operation="QueryRelationships"})

is incremented -> we actually issued a datastore query -> we acquired and then returned connection from/to the pool. so I assume the above sum should be equal to (or at least be close to, taking into account other db operation)

sum(pgxpool_acquire_count)

. but with my local docker-compose installation I get like x3.3 difference (with

sum(pgxpool_acquire_count)

being smaller).

I meant more benchmarks like https://authzed.com/blog/google-scale-authorization - but for postgres and across all of the methods that you can make. I will separately take a look at the metrics.

@Will Thornton no, we don't have such numbers