what datastore are you using?
# spicedb
j
what datastore are you using?
r
postgres (aws)
v1.44.4 -> v1.45.1 Latency Start happening v1.44.0 -> v1.44.4 Latency Stop happening v1.39.1 -> v1.44.0 Latency Start happening
always every 10m
j
what is the GC run interval?
r
We did not attempted to tune it. Are you refering to datastore-gc-interval? Or somethign native to go?
y
the first one
r
I believe this is it (default): GCInterval: 3 * time.Minute,
j
and what does the API request graph look like?
thats a trace for production. Every request we have a max of 3 retries, so that is the OTEL for the 3 retries
for a failed request that happened at 9:30 (its super predictable, every 10 mins to the clock we encounter this timeouts)
9:30 - 9:40 - 9:50 and so on. seconds very given the metric scrape is every 10 secs or so
btw the way. This latency is the MAX. the p99 & p95 is "perfect"
so this is not serious, but I would like to understand the rule out anything that might be getting worse and worse
j
sounds like its either something hitting your cluster or something odd in your datastore
what is the datastore CPU when the 10 minute mark is reached?
r
If thats nothing comes up imeatly on your side (like cache timeout or something) then I will rule out spiceDb for now 👍 Thank you for the support
j
> what is the datastore CPU when the 10 minute mark is reached?