https://authzed.com logo
Some sort of thread about things
d

Donning the Sage of Asparagus

03/24/2023, 8:13 PM
ok, here goes (and I suppose the response could be collected into a thread if they want) - The basic question is "does spiceDB support iterating entities by type?" Motivation: I'm thinking about synchronizing users from an external authority (per https://authzed.com/blog/why-model-users/). Whenever I have an external authority I always worry about anti-entropy (missed messages due to wiring mishaps or bugs), which requires being able to compare state. For very large datasets I'd do something like a merkle tree, but my data set is pretty small, so I just want to iterate one or the other and compare. Detecting deletes is always tricky, so I might want to iterate both sets to make sure there's nothing in my data that isn't in the original.
j

Joey

03/24/2023, 8:16 PM
usually the recommendation here is to keep a hash relationship and use it as a precondition for each update
also I hope you appreciate the thread title 😛
d

Donning the Sage of Asparagus

03/24/2023, 8:16 PM
genuinely LOL'ed
not sure what "keep a hash relationship" means though
j

Joey

03/24/2023, 8:17 PM
so, let's say you're syncing, say, users
d

Donning the Sage of Asparagus

03/24/2023, 8:18 PM
indeed, let's say that
j

Joey

03/24/2023, 8:18 PM
when you go to add/remove a user, you calculate a hash of all the users', say, IDs
then you define a definition like so
d

Donning the Sage of Asparagus

03/24/2023, 8:18 PM
yeah, you're describing a one-level merkle tree
j

Joey

03/24/2023, 8:18 PM
yes
d

Donning the Sage of Asparagus

03/24/2023, 8:18 PM
hash all the entities, then you know if something is different
j

Joey

03/24/2023, 8:18 PM
and then you store that hash in SpiceDB
and when you go to perform an update
you replace it with a precondition specifying the older hash
that ensures that all updates are atomic
d

Donning the Sage of Asparagus

03/24/2023, 8:19 PM
the hash tells me "something is wrong." Now I need to know what
j

Joey

03/24/2023, 8:19 PM
well, you wouldn't have "something is wrong"
because write rels is transactional
you're basically say "move the state of the sync -> here"
d

Donning the Sage of Asparagus

03/24/2023, 8:20 PM
yeah, it's effectively an optimistic lock on the entire collection
theoretically that's sufficient, but my experience has been that entropy doesn't respect theory 🙂
that's a fine approach to reduce the risk, though, hadn't considered it
and I think the direct answer to my question is "no" - which is cool
we'll handle it with tombstone records for removed entities on the source side
j

Joey

03/24/2023, 8:24 PM
nothing precludes writing a reference to a tree you store somewhere
but probably not necessary
d

Donning the Sage of Asparagus

03/24/2023, 8:32 PM
yeah, definitely overkill here
thanks for the help
j

Joey

03/24/2023, 8:37 PM
of course 🙂
d

Donning the Sage of Asparagus

03/24/2023, 8:55 PM
hey, just occurred to me - computing a hash of all user's IDs requires iterating all users, doesn't it? If not, how do I compute that hash?
j

Joey

03/24/2023, 9:01 PM
yes, it does
unless you can use an additive one somehow
d

Donning the Sage of Asparagus

03/24/2023, 9:08 PM
ok, so the original question was "does spicedb support iterating entities by type" - I had inferred from your earlier responses that you were saying 'no' but I now think you were saying 'yes, but do it a cooler way'
so... does spicedb support iterating entities by type?
that is "all users" or "all teams"?
j

Joey

03/24/2023, 9:25 PM
yeah
ReadRelationships
d

Donning the Sage of Asparagus

03/24/2023, 9:36 PM
thx
could you point me to the documentation? I looked but couldn't figure out where it lives
j

jzelinskie

03/24/2023, 10:42 PM
We also have some folks that just schedule jobs for low-traffic times to replay events to ensure nothing got missed.
Pretty common pattern in the Kafka ecosystem if I'm not mistaken
d

Donning the Sage of Asparagus

03/24/2023, 11:21 PM
sure. there's other ways stuff can get out of sync, though. backfills, bugs, etc etc. It's useful to have a way to actually guarantee that what is effectively a durable cache stays in sync with the authoritative source.