Hi We are experiencing some strange things, regarding check permission and zedTokens. Our case is the following. We simultaneously create relations for 6 documents (ids doc1 - doc6) with the same owner relation (owner1). During the creation of ex doc3, the relation for doc1 gets deleted. When we then want to check permission isOwner after creation of doc3 with the zedToken we got from the write, we get permission denied. If we then try to loop with the same check with 500ms delay, then after 3 - 5 secs, we get permission allowed. Or if we check right after with fully_consistent we also get permission allowed. Are there something we don't understand regarding the zedToken? 🙂

> We simultaneously create meaning in the same

WriteRelationships

call? my best guess is that the revision in the token you're using isn't pointing at the revision you think it's pointing at, given the behavior you're describing

No, not in the same write, but in multiple threads. Each write gets its own zedTokan which we store together with the document.

zedtokens are not necessarily casually ordered

Why do we then get permission denied on the check permission for the specific write with its zedToken, and why does the response change if we keep trying for 5 secs?

you shouldn't so long as the write reflects all the relationships for that check

specifically, if the zedtoken you get for the document writes is functionally earlier than the zedtoken you get for the owner write, and you're checking using the zedtoken for the document write, the request may not reflect the change from the owner write

what datastore are you using?

this blog post goes into depth on how quantization windows work if you're curious: https://authzed.com/blog/hotspot-caching-in-google-zanzibar-and-spicedb

We are using postgresql

which has non-timestamp based revisions

Does that mean if we make more writes to the owner relationship, for the same owner at the same time, but manage to delete one of the first relationships, before the later is writen to the database, the token for later ones can be unusable?

it means if you're making changes to the same relationships in parallel

We will not have a delete for a speciific doc ex doc1 before it is written. But we can have a delete for doc1 while ex doc3 is written.

what is "ex" doc vs normal doc?

Sorry while doc3 is writen.

if you write a relationship for doc3 and then check for doc3 (assuming it doesn't rely on doc1 at all) with at_least_as_fresh with the zedtoken from the write, it should reflect that write

We will never delete a relation without insure we get a zedtoken for that relation write.

right, but you need to remember the zedtokens may not overlap

This is where we experience the faulty check permission.

it shouldn't do so, so you might need to put together a repro case

And if we retry for 5 seconds, it works

because at that point the window has moved on

And to make this even more strange, it only happens sometimes.

well yeah

But could it be because of cache window move this happens?

no

We use at_least_as_fresh for checks

I know - but its easy to accidentally send the wrong zedtoken

We experience this even if we do a check in the code line just after the write, with the zedtoken from the write.

> We experience this even if we do a check in the code line just after the write, with the zedtoken from the write. then please put together a repro and we'll see if I can identify the cause

Is it okay to store each write zedtoken together with the doc, and use it for checks for that document, or should we store the last zedtoken and use that for checks?

yes, that's what's intended

One last thing, if we syncronize the writes, so only one write can happen at the time, everything works fine.

sounds like there might be some overlap

I will. Thanks 🙂

no