Data Joining
# spicedb
t
Could somebody explain or share info, how to cover such case: I have some huge dataset like millions / billion records which I somehow displaying with pagination and filters. I need to call spicedb and ask what exact filters are accessible for current user, than pass it to my query and obtain final result But during expanding all this relations i potentially could get around million of ids and pass them to other query. This is kind'a overhead. *Could you advise how to solve this correctly? * I was thinking about like to mix up some groups in crdb and sync all of this, but unfortunately this is not a solution. I need to save flexibility Any help appreciated
b
If I understand your problem correctly, it reminds me of OPAs partial evaluation solution https://blog.openpolicyagent.org/write-policy-in-opa-enforce-policy-in-sql-d9d24db93bf4
It’s a common problem that we have as well: how do you join data across two services. Usually we grab the most selective side first and then join in the relevant pieces from the second service.
You want to get away from grabbing millions of IDs upfront though. You shouldn’t need all of that if you’ve got a bounded page size that you’re trying to fill
j
the general rule of thumb is that you filter on one side (or the other), and then apply the other filter. If you expect to only show, for example, a subset of results, you can filter in your search index and then issue checks to filter. Alternatively, you can call LookupResources with cursors and use that for pagination.
Longer term, this is the propsed solution: https://github.com/authzed/spicedb/issues/207
b
This is cool. The 64bit requirement is a bummer for those using UUIDs
j
see the later messages
we likely won't using roaring bitmaps as a result
especially because many people have non-int object IDs
b
Fantastic
I see a lot of large and interesting things planned for spicedb across the GitHub issues. Is there any sort of public roadmap indicating priority?
j
not currently
we're working on where and how to host it
2 Views