what datastore are you using?
# spicedbj
Joey
08/25/2025, 4:32 PMwhat datastore are you using?
r
Rodolfo
08/25/2025, 4:40 PMpostgres (aws)
Rodolfo
08/25/2025, 4:52 PMv1.44.4 -> v1.45.1 Latency Start happening
v1.44.0 -> v1.44.4 Latency Stop happening
v1.39.1 -> v1.44.0 Latency Start happening
Rodolfo
08/25/2025, 4:53 PMalways every 10m
j
Joey
08/25/2025, 4:53 PMwhat is the GC run interval?
r
Rodolfo
08/25/2025, 4:56 PMWe did not attempted to tune it.
Are you refering to datastore-gc-interval? Or somethign native to go?
y
yetitwo
08/25/2025, 5:02 PMthe first one
r
Rodolfo
08/25/2025, 5:10 PMI believe this is it (default):
GCInterval: 3 * time.Minute,
j
Joey
08/25/2025, 5:10 PMand what does the API request graph look like?
r
Rodolfo
08/25/2025, 5:17 PMhttps://cdn.discordapp.com/attachments/1409576136093995039/1409587602868142222/Screenshot_2025-08-25_at_18.16.12.png?ex=68adec41&is=68ac9ac1&hm=b0b87f0db43f555fd3011201596610054ba793f6395a0c1fc379cd4b9ea94995&
https://cdn.discordapp.com/attachments/1409576136093995039/1409587604919160944/Screenshot_2025-08-25_at_18.16.40.png?ex=68adec41&is=68ac9ac1&hm=decb361b5265bcaa078284c09363e6f2ea7c140cd1888ca186aec36712f086bd&
https://cdn.discordapp.com/attachments/1409576136093995039/1409587607138209812/Screenshot_2025-08-25_at_18.16.58.png?ex=68adec42&is=68ac9ac2&hm=ca64ce64ff562469e29833753fcae91cf6b3748bc8530289f61ba07a2d6002d4&
Rodolfo
08/25/2025, 5:18 PMthats a trace for production.
Every request we have a max of 3 retries,
so that is the OTEL for the 3 retries
Rodolfo
08/25/2025, 5:19 PMfor a failed request that happened at 9:30
(its super predictable, every 10 mins to the clock we encounter this timeouts)
Rodolfo
08/25/2025, 5:20 PM9:30 - 9:40 - 9:50 and so on.
seconds very given the metric scrape is every 10 secs or so
Rodolfo
08/25/2025, 5:36 PMbtw the way. This latency is the MAX.
the p99 & p95 is "perfect"
Rodolfo
08/25/2025, 5:37 PMso this is not serious, but I would like to understand the rule out anything that might be getting worse and worse
j
Joey
08/25/2025, 7:12 PMsounds like its either something hitting your cluster or something odd in your datastore
Joey
08/25/2025, 7:12 PMwhat is the datastore CPU when the 10 minute mark is reached?
r
Rodolfo
08/26/2025, 8:50 AMIf thats nothing comes up imeatly on your side (like cache timeout or something) then I will rule out spiceDb for now 👍
Thank you for the support
j
Joey
08/26/2025, 3:05 PM> what is the datastore CPU when the 10 minute mark is reached?