Hello spiced people 👋 I'm reading the docs to try and understand how to use the new paginated APIs (in particular LookupResources). I'm not seeing much, do you have an example of how to use pagination now?

pagination is not released yet - we released a new paginated

Datastore

implementation that allows us to expose the paginated APIs. The 1.21 release includes a new implementation of

ReadRelationships

that uses pagination under the hood, which means that it only reads as many tuples as the stream requests (roughly - it still over-reads, but nothing like before). The paginated versions of Lookup APIs is WIP and may land in 1.22

Right, so if I understand correctly, so far it's only useful for respecting

optional_limit

?

no, nothing provided via the API is being respected right now re: pagination. As I mentioned, it's only the (internal) Datastore API that has changed, in preparation to making applicable SpiceDB API paginable. I'm wondering if the fact that the buf API already shows pagination concepts be misleading here

Maybe I read what I wanted in the "Datastore accesses are now paginated to efficiently handle large Reads and LookUps" announcement 😄

could you expand? 🙏🏻 cursor and limit relatively standard across the industry, what part makes it confusing? The fact that its used over a streaming API?

I think the important part here is "already", not the cursor/limit concept. I also thought 1.21 already has pagination support. You are saying it has not, which is fine, but then it is confusing that the public API already holds these non-supported concepts as part of its contract, and thus tricks ppl in thinking they are already ready to be used 🙂

yeah that's is fair - it's a side effect of SpiceDB API being decoupled from SpiceDB repo itself

I'd expect a

limit

property to come with an

offset

property. I'd expect

cursor

and

limit

to be 2 unrelated concepts. I don't really understand ATM what

optional_cursor

is for: what can I use it for?

are you perhaps thinking of offset-based pagination instead of cursor-based pagination? Cursor is literally a relationship where to continue paginating from. So, in the upcoming paginated

LookupResources

, you'd be saying something like "fetch the next X elements (i.e. limit) starting from this last known result (i.e. cursor)". It's optional because it has to be retrofitted into an API that didn't have it before, but also because on the first call you will not have a cursor

1st call: optional_limit = 50 1st response: after_result_cursor = 50 => 1st 50 entries are shown. 2nd call: optional_cursor=50, limit = 50 2nd response: after_result_cursor=100 => 51th to 100th result are shown 3rd ... ... ... if I understand it correctly.

> So, in the upcoming paginated LookupResources , you'd be saying something like "fetch the next X elements (i.e. limit) starting from this last known result (i.e. cursor)". It's optional because it has to be retrofitted into an API that didn't have it before, but also because on the first call you will not have a cursor OK. But I am confused as to why I can use these for pagination, yet you're saying that pagination is "upcoming"?

right, that would be related to what @dguhr-rh said above: it's the problem of the decoupled lifecycle of the API definition and the

SpiceDB

repo

> right, that would be related to what @dguhr-rh said above: it's the problem of the decoupled lifecycle of the API definition and the SpiceDB repo I just got that indeed 😄

the API shows as if it's already there, when in reality that version of the API isn't yet consumed by a released version of SpiceDB

okay all the pieces of the puzzle fit now

it's fair feedback 💯 To be clear, you are referring to https://buf.build/authzed/api right?

Yes, https://buf.build/authzed/api/docs/main:authzed.api.v1 to be precise, which is linked from the docs website

as a side note - offset pagination does not scale, because it requires you to read all previous rows in order to determine what the n-th row is. I'm sure y'all know that already, but I always found this blogpost captures the problem well https://use-the-index-luke.com/no-offset