@User yep, we have an internal design I put together for exactly that stream of changes, and then coupling it to a cache that can be fed to something like ElasticSearch

Cool! If a relationship to one resource kind changes that affects another type of resource's permission(s), will the Watch API still stream that change? Or will it only stream permission changes for the type of resource that was mutated? Not sure I fully understand the design there, but it sounds like you guys are starting this work. Do you have an open RFC doc or something I can refer to to read up on the proposed design?

not yet; I wrote this a few months ago and then it was set aside while we worked on other things (like open sourcing)

For sure

well, we'd run it as a public proposal in the Github issues to allow for public discussion

Yeah, that sounds great. That's all I meant.. If there's an open proposal then I can start looking at it and I'd be happy to be assigned to it and tackle it.

okay; I'll update the proposal and bring it over into an issue

> are you guys using ElasticSearch or another indexing system We're using a lot of different systems. ElasticSearch is the primary engine backing our app search engines. But we also index data into mongodb and others for other use cases like analytics and reporting etc... > what level of consistency are you hoping to have at any particular moment? I envision the client watching for changes to have to also respect the zookie protocol. If they do not, then they can't guarantee consistent evaluations. That also gives the client the ability to choose what level of consistency is actually required. Our reporting engine, for example, can accept a little higher consistency window than our app search engine. Our platform's app search engine should be consistent within a couple of seconds in general. Does that answer your question? I can provide more detail, but that's a summary.

yeah, makes sense

👍

the one caveat is if you gave a zedtoken "ahead" of the cache, it would simply not be able to return results (because a bunch of things could have changed)

I think that's an appropriate compromise. I need to think about it a little more though. I do think that it's fair to block/wait or fail fast if the zookie provided is ahead of the latest cached entry. In that case it would be the clients responsibility to reattempt the request.

yep

Cool, yeah we're on the same page. And, in general, these edge cases will likely not be encountered by our integrations. We don't have super tight requirements of consistent checks for our application. Some applications will require more strict constraints, but most will be quite lenient.

well, the design proposal is, in theory, massively scalable

Cool! I look forward to reading the proposal. I can commit company time to work on it as well, so you have my word that at least myself or my team can pitch in on the effort.

good to know. I'd love to get them engaged in here too once the proposal is up, so we can all discuss the design 🙂

Joey, did you open an issue for this?

@User not yet; I've been polishing up the proposal. Will likely post tomorrow

👍

getting the proposal reviewed now

Sweet. I don't know if that's too rushed, but if it worked out to fit this work into our garage week I could get one or two additional engineers potentially to help out if needs be.

@User proposal posted: https://github.com/authzed/spicedb/issues/207

Provided some feedback 🙂

@User responded

Sounds good. I like the discussion going on. Good stuff 👍 I responded back 🙂

just responded to your response

Sweet! Back at ya 😉

cool