I enabled enable-experimental-watchable-
# spicedbc
croemmich
02/08/2024, 7:59 PMI enabled enable-experimental-watchable-schema-cache as recommended by @vroldanbet but it seems to be causing some weird caching behavior. Turning it off immediately resolves the issue. The hit ratio is high for the duration of quantization window and then falls over several hours until it flatlines around 10%.
In the "Reboots" chart each jump in cache hit ratio was caused by a reboot of the spicedb pods. It seems to stay high for about the length of the quantization window but then falls.
In the "On->Off" graph enable-experimental-watchable-schema-cache was changed from true to false.
croemmich
02/08/2024, 8:04 PMhttps://cdn.discordapp.com/attachments/1205241511973879869/1205242668221857862/reboots.png?ex=65d7a8b4&is=65c533b4&hm=908b224318f688a672c3133acccab7cfcb013c97483208fd509a5a63cf3f6164&
https://cdn.discordapp.com/attachments/1205241511973879869/1205242668431704074/on_-_off.png?ex=65d7a8b4&is=65c533b4&hm=d2149c9145da6db47b1d1cd137a59419c5a96af4f3d1b60ec22475f5f721e059&
v
vroldanbet
02/08/2024, 8:09 PMHey that does not seem right, can you share all your SpiceDB flags?
c
croemmich
02/08/2024, 8:37 PMRevisionQuantization and MaxRevisionStalenessPercent are exagerated for testing, but I was seeing it on small values as well. It was just harder to see the relationship between cache hit and quantization window so I increased them.
https://cdn.discordapp.com/attachments/1205241511973879869/1205250986717741096/configuration.json?ex=65d7b073&is=65c53b73&hm=aba5d30490e1077a533d4288a30c7febb52bb3b889c7029c763025b1b7b256f7&
croemmich
02/08/2024, 8:43 PMWe were also seeing some weird performance issues at about 2x the quantization window. All of latency numbers were roughly doubling. Disabling enable-experimental-watchable-schema-cache fixed that as well.
j
Joey
02/08/2024, 9:27 PMyou sure you're filtering the caching metrics?
Joey
02/08/2024, 9:27 PMthe new schema cache also has its own caching metrics
Joey
02/08/2024, 9:27 PMso its possible that your prom metric names are matching both
Joey
02/08/2024, 9:27 PMI doubt that's the root cause, but worth checking
j
Joey
02/08/2024, 9:44 PMokay, those should be distinct then
Joey
02/08/2024, 9:45 PMand the dispatch cache should not be influenced by the schema cache at all
Joey
02/08/2024, 9:45 PMexcept in total memory used
Joey
02/08/2024, 9:45 PMits possible you're getting less caching due to faster eviction
Joey
02/08/2024, 9:45 PMbut that seems unlikely unless your nodes are VERY memory limited
Joey
02/08/2024, 9:46 PMlikely worth tracking these metrics too: https://github.com/authzed/spicedb/blob/main/internal/datastore/proxy/schemacaching/watchingcache.go#L43
Joey
02/08/2024, 9:46 PMsee what your hit rate is on the schema cache
v
vroldanbet
02/09/2024, 8:32 AMristretto (cache library used by SpiceDB) has some internal metrics which I believe can be exposed with an additional flag. It would be interesting to plot those and see what's happening when the schema cache is on/off. Maybe it's hitting the default cache capacity (which is inferred from the resource requests and is known to make inaccurate decisions) and thus causing eviction.
7 Views