not inappropriate at all, but we don't have a firm date

its in active development though

and as @User said, there are definitely alternatives for now

we ourselves do post filtering

and use lookupresources in other places

@User thank for such a detailed answer! It definitely looks reasonable. I can’t test it now, but will check tomorrow, it seems like 3rd approach solves the problem

@User yep, I was thinking about post filtering, it just doesn’t look as elegant as I’d like it to be. If my query returns a page of 50 results, and on the postfiltering I find that 1 of these 50 is not authorized, what is the correct behavior? Just skip it? Query one more until I get an authorized? Return a bad request? Or 403? Not clear to me

generally speaking: you'd have an iterator from your database, and you'd pass that into a filtering iterator

so as your reading results, they are being checked and filtered from the top-level iterator

this only works if you expect your input set of objects to mostly be visible; otherwise, you could load N's of records for a single output record

which wouldn't be as efficient (and in that case, pre filtering is better)

this is why we have LookupResources in that case, and the other proposals coming down the pipeline

the "ultimate" fast method will be Tiger cache: get a bitmap, use as a filter, post-check the few items that may have had their permission changed since the bitmap was updated

but it will require object IDs that are 64-bit ints

It totally makes sense. Thanks for your detailed answer. I think we will test both postfiltering and the approach described by @User

Given that SpiceDB relies a lot on in-memory caching for performance, how do you handle updates to the service? Eg when

v1.4.1

comes out, I'll want to roll it out: it will mean a new set of pods replacing the

v1.4.0

ones (we use k8s) and therefore losing all the cache and probably seeing huge response time spikes. Is it "simply" a matter of making it a slow rolling update? How slow?

Related question: in your experience, is it better to have many small instances of SpiceDB (in CPU/RAM), or a few big ones?

right now every time the quantization period resets it will be like the caches are empty, and it hasn't been too big of a deal in practice

if you've got a highly complicated schema or data that's deeply nested, it might be a bigger impact

and yeah, my suggestion would be a slow rolling update against a large fleet, if you set the period to match the configured quantization period (5s by default) there shouldn't be any noticeable effect

w.r.t. instance size, we recently ran into an issue with scheduler contention with a large number of goroutines, which can be helped by having more cores, but we don't have any formal guidance

we recently moved production from 2 core to 8 core nodes

> the quantization period What's that? 😄 is there documentation around this? Maybe I need to dive into the Zanzibar white paper, I've been putting it off > if you set the period to match the configured quantization period (5s by default) there shouldn't be any noticeable effect Am I understanding correctly that you're suggesting replacing a fraction of the fleet (let's say ~20%) every ~5s? > we recently moved production from 2 core to 8 core nodes Not sure I get it, but it might be because we have very different scales (obviously, as it's your whole business)! For us, running a couple of SpiceDB instance on a couple of pods having 1CPU / 1GB RAM each is probably more than enough until we scale as a company. It's not very resilient though, and might not be taking advantage of the dispatch clustering, so I'm considering making it something like 5 pods with lower resources instead. Does that make sense?

best "documentation" are my talks about zanzibar or spicedb

yeah, i'm suggesting if you're not replacing a pod more frequently than the quantization period, you shouldn't be affecting the cache much

if your nodes are going to share a physical CPU, then I think it's probably best to just have one pod per CPU, otherwise you could end up with some really bad scheduler contention and priority inversion

> For us, running a couple of SpiceDB instance on a couple of pods having 1CPU / 1GB RAM each is probably more than enough until we scale as a company. whether this works for you or not is going to be highly dependent on your traffic pattern the contention that happens is the go runtime; if you have really bursty traffic you'll have so many more goroutines than cores that the runtime can't get them enough cpu time to finish before client-side timeouts start kicking in. but that's something that should be fairly easy to measure once you're up and running

> best "documentation" are my talks about zanzibar or spicedb Will check that out, danke! I see what you both mean about CPU sharing and contention. That's good advice, thanks 👍

Update: this is more complicated than I initially anticipated. I had a quick and dirty hack in place to allow large imports, which got the job done for me, but as I started to formalize the hack into more robust code, I found a number of things that should be considered: 1. The gRPC server has a limit on message sizes. This can easily be solved by batching the requests 2. Batching is tricky, serially executing each batch works to an extent, but it's not very scalable. This could be improved by executing each batch in a goroutine. Of course, this comes with some risk, as there's roughly a 9k batch size limit (see 3). If a

zed import

is trying to import 6 million+ rows, that's about 650 connections each trying to shove through 9k tuples. 3. Postgres and MySQL both have a limit on how many placeholders you can have in a single query. This appears to be 65535 for Postgres and MySQL (https://stackoverflow.com/a/49379324, https://stackoverflow.com/a/24447922). This roughly translating to a maximum of 9362 relationship tuple writes (assuming each is going to require 7 placeholders). I didn't initially hit this limitation because I was testing with an in memory database. The conclusion is, I think augmenting

zed import

to support large scale relationship sets will require a bit more design and planning than my initial hacky batching solution offers and might extend to making changes to SpiceDB, not just

zed

so it's possible to batch the batches (this is really just turtles all the way down 🐢 ).