https://authzed.com logo
Join Discord
Powered by
Hi dear spiceDB, I'm running into issues
# spicedb
h
Hi dear spiceDB, I'm running into issues with using a zedToken & parallel tests, where multiple updates happen on the same resource at the same time - but from time to time some read or permission checks use incorrect zedToken because of that. Also tested with fully_consistent in the tests, but also experience the same issue there. From what I've read in here then it is an expected behavior. I'm storing the zedToken in redis with a 1m timeout, so it shouldn't affect the product much, just that it makes it harder to write tests. Am I possibly doing something wrong? Or is there any magic/tips available that I could look into to make parallel tests run properly?
v
👋 Are you using postgres datastore?
h
Yes I am.
y
can you describe more how you're using and storing the zedtokens? putting them into a cache feels like a potential antipattern
as there's potential for race conditions
is this happening in write-then-read workflows or somewhere else?
j
Is this being used in a unit or integration test?
Or is this production code?
h
I'm currently running two services; group-service & authzed permission-service. Clients then call the group-service for creating & managing groups, that direct all authorization changes to the permission-service. The plan however is to add more types of authorization resources and also allow clients to directly call the permission-service for permission checks. At the moment I'm caching the zedTokens in redis per group/resource and I'm letting the permission-service check & update the zedTokens directly. Have not dived into how that will look like in the future, but would most likely require the permission-service to have direct access to them, since there will be multiple types of resources & accessed from both clients & other services alike. The test scenario are e2e tests that make grpc calls to the group-service. So basically users calling for creating new group resource & sequentially adding new users to them.
y
in our experience there isn't really a good way to abstract over zedtokens - either the end client needs to be aware of them or you need to completely elide them and accept that calls will be using 
minimize_latency
as their consistency
there isn't a good way for the fronting service to take on that responsibility
h
Here's an example print from my permission-service when I run parallel e2e tests:
Copy code
permission-service-1 12:48:40 *** WriteRequest
permission-service-1 12:48:40 resource.object_id: "user_group_1"
permission-service-1 12:48:40 relation: "admin"
permission-service-1 12:48:40 subject.object_id: "user_1"
permission-service-1 12:48:40 *** response finished for user_1 - writtenAt: zedToken_1
permission-service-1 12:48:42 *** WriteRequest
permission-service-1 12:48:42 resource.object_id: "user_group_1"
permission-service-1 12:48:42 relation: "member"
permission-service-1 12:48:42 subject.object_id: "user_2"
permission-service-1 12:48:42 *** WriteRequest
permission-service-1 12:48:42 resource.object_id: "user_group_1"
permission-service-1 12:48:42 relation: "member"
permission-service-1 12:48:42 subject.object_id: "user_3"
permission-service-1 12:48:42 *** response finished for user_3 - writtenAt: zedToken_3
permission-service-1 12:48:42 *** response finished for user_2 - writtenAt: zedToken_2
permission-service-1 12:48:43 *** CheckPermission
permission-service-1 12:48:43 req.Consistency: Consistency_FullyConsistent{FullyConsistent: true}
permission-service-1 12:48:43 resource.object_id: "user_group_1"
permission-service-1 12:48:43 subject.object_id: "user_3"
permission-service-1 12:48:43 permission: invite
permission-service-1 12:48:43 *** permissionship:PERMISSIONSHIP_NO_PERMISSION - readAt: zedToken_2
First call: create group resource & add the user as admin Second call: add new member to the group Third call: add yet another member to the group Fourth call: does member from third call have permission? Result: false (using the zedToken from second call)
I only added FullyConsistent as an environment var to see if I could get the parallel tests to work.
Like I said, I'm not worried that there will be a delay for permission checks outside the tests - just wanted to see if I had a way for them to work as well.
y
> as an environment var what do you mean by that?
h
just an env variable that I pass into the container when I run the tests as part of the CI
Copy code
if ps.cfg.FullyConsistent == "true" {
        req.Consistency = &v1.Consistency{Requirement: &v1.Consistency_FullyConsistent{FullyConsistent: true}}
}
j
the expectation here should be that there will be a slight delay in the consistency
h
Yeah I totally get that, just wanted to know if there might be any way for me to make my e2e tests a bit easier & less flaky. Like if there would be a way to completely shut down the cache.
v
We expect that using zedtoken from call 2 you won't be able to observe changes from call 3. However, I would expect that with 
full_consistency
you should be able to observe the change, so I wonder if there is something up here. Do you think you could write a repro in the SpiceDB repo?
8 Views
PreviousNext
Powered by