CRDB timestamp issue
# spicedb
c
When ever the record was last updated, some going back 2-3 months
j
what is the GC window configured on your CRDB?
c
4h, we have spicedb set to 2h now. Initially they were equal but I changed it to see if it would resolve this issue, however it still happend.
j
by that you mean SpiceDB's quantization window is set to 2h?
c
No quantization window is 10s, with max staleness 1800.
j
ah
c
(probably unrelated, but I'm going to go back to the defaults for those, we're seeing some weird cache performance, where hit rates drop over time)
j
its unusual to have staleness that large
c
Yeah, we were fighting some db performance issues so I was trying to get our cache hit rate as high as possible. We fixed the db perf issues now, so I'm planning to go back to defaults.
j
what was the previous default?
its possible setting a max staleness that large is the cause
especially because restarting SpiceDB would reset the revision off which its calculating the staleness
c
I'm not actually sure, it just wasn't set. I think the internal default is 10%.
j
yeah, its much smaller
I'd suggest trying to set it back to the default and seeing if that solves the issue
if it does, we'll adjust the error messaging accordingly
c
I wouldn't mind it being slightly longer like 100%. I'm a little confused looking at the code as to the format of that configuration (datastore-revision-quantization-max-staleness-percent) Should that be configured to "1" or "100"?
j
100% is
1
we'll clarify the value
c
Ahh.. that's probably contributing to my issues
Thanks!
j
what did you set it to?
c
I'm setting it to 1
j
ok
c
It was 1800... which would have put it out past the gc window
j
oh, yeah!
that'd do it!
I'll add a check for that
c
Thanks!
for the description change
c
Looks great to me!
this should prevent the problem in the future
it also logs a warning if it is in excess