Hey it s me again with more in depth
# spicedbw
wscalf
08/21/2023, 9:31 PMHey, it's me again with more in-depth performance questions!
wscalf
08/21/2023, 9:31 PMWe were doing some testing with a large schema (lots of definitions, lots of inherited properties, total of about 250k of schema) and potentially a lot of data for the RDS size (around 400M relation tuples on a db.r6g.xlarge) and noticed some interesting behavior: if all of our requests are checkaccess or lookupsubjects, we get excellent cache utilization (>95%) and excellent performance (P99 < 10ms)- but if we include the occasional lookupresources in the mix (approx 10% of requests), it's a totally different picture: cache utilization close to 0% and nearly every request timing out, with our RDS instance totally overwhelmed. I'm curious if folks could help us understand how that might happen- especially why the cache utilization would be impacted so much.
j
Joey
08/21/2023, 10:03 PMthe cache is shared
Joey
08/21/2023, 10:03 PMpresumably, your LR is returning lots of data and pushing other stuff out of the cache
w
wscalf
08/21/2023, 10:22 PMOkay. It's ultimately only returning one item, but there are some steps in the path where there would be a lot of items. If that could push other things out of the cache, that likely explains it.
j
Joey
08/21/2023, 10:50 PMyes
Joey
08/21/2023, 10:50 PMwe cache the intermediate steps
Joey
08/22/2023, 4:21 AMI'd see if increasing the memory available helps
Joey
08/22/2023, 4:21 AMif it does, we can look into providing a flag for potentially breaking out those caches
y
yetitwo
08/22/2023, 3:27 PMrelated question: I know that
readRelationshipsreadRelationshipsj
Joey
08/22/2023, 3:45 PMpretty much
w
wscalf
08/22/2023, 7:02 PMFollow-up question: we're currently using spicedb_cache_cost_added_bytes and spicedb_cache_cost_evicted_bytes to track cache churn, but is there a metric we could tune to get to how much cache is currently in use?
j
Joey
08/22/2023, 11:06 PMJoey
08/22/2023, 11:06 PMare the defined metrics