Hi,
I'm running load testing on a SpiceDB instance in my system, 700 RPS of CheckPermission requests using Thumper.
The observation so far:
p99 latency (spicedb.grpc_server_handling_seconds) gradually increases over time.
Node CPU usage slowly climbs from 3% to 5% during the test but nothing critical.
Datastore CPU usage grows from 2% to 5.5%, which I believe also drives up SpiceDB datastore query latency (spicedb.spicedb_datastore_query_latency)
It feels like there is some sort of resource leakage, though nothing crashes or fails obviously.
One suspicious thing I noticed: bytes received/sent on the SpiceDB instance keep increasing, whereas I would expect them to remain more or less constant.
Does this behavior look familiar to anyone? Any ideas on what I should look into