https://authzed.com logo
Join Discord
Powered by
Hi, I just want to sanity check my
# spicedb
j
Hi, I just want to sanity check my experience with the performance of the LookupResources API: - In a system with about 8k relationships; - Using 
LookupResources
to query a resource type with a response size of around 1.1K resources; I am seeing response times of around 300-500ms. This seems very high to me for the order of mag I am dealing with. Is this expected or does something seem wrong? I am seeing the same order of mag when running spice on our prod environment (with spanner) and locally (with postgres)
v
It will depend on what your schema looks like - some parts would be more expensive than others. Would you be able to share it? What version of SpiceDB are you using? Are you using cursors?
j
I can share portions of my schema but would prefer not to do that publically here. This is using spicedb 1.26.0 and not using cursors
v
please consider using 1.28. Cursors would allow you to get a response faster, the thing with non cursored LookupResources is that the server has to wait to get all the subproblems solved before it can proceed. By using cursors you only focus on a narrow subset of the subproblems and can start processing the batches in your client application while you are fetching the subsequent batches. If cursors is somehow not a viable option for you, then the options are: - optimizing the schema - looking into metrics to understand what needs to be tuned (e.g. connection pools in postgres, or gRPC connections in Spanner)
1.28 comes with a bunch of improvements for Spanner, including better observability
How many PUs has your Spanner allocated?
j
Ok I have some questions to understand this: To clarify, by cursors you are referring to pagination cursors? I don't see anything in the 1.28.0 release notes around cursors. Was there some change or improvement in 1.28.0 around pagination? I also don't fully understand why the pagination cursors would help - if I require the entire response in order to respond upstream wouldn't the time taken be the same + longer because of the added network overhead? I don't think the cursor is particularly complicated, I will share the relevant portions below in a moment. I will upgrade to 1.28.0 for the Spanner improvements and report back. Our spanner instance is only provisioned with 200 PU's right now, but I did not observe any impact when increasing this during testing. I am noticing weird traces like in the attached image where spicedb seems to complete work but the grpc response completes much later. Not sure how best to interpret it. https://cdn.discordapp.com/attachments/1181536779006918676/1181600574303645746/image.png?ex=6581a649&is=656f3149&hm=dfa4bf8d944f688c41d9c5ce7bc1c7fe3c85ccb86378631a6ffcb46a3b81d042&
> ... if I require the entire response in order to respond upstream ... For example because I need to include them in a DB query later.
> please consider using 1.28 I am deploying using the spicedb-operator which currently supports up to 1.26.0 - is it safe to override it's default max and go to 1.28.0?
v
>Was there some change or improvement in 1.28.0 around pagination? there were none related to cursors >if I require the entire response in order to respond upstream wouldn't the time taken be the same + longer because of the added network overhead? that is correct, but you are also putting more strain on the server, and for a largely enough response it may run out of memory. Cursors set a boundary to how much memory is required to be held during LookupResources. If all you need is the full-response, cursors will not make things faster. I made an assumption that you application could work with pages of results, but it sounds like you need the full response. >Our spanner instance is only provisioned with 200 PU's right now, Expensive LookupResources with a 200PUs Spanner will almost certainly get throttled by Spanner - I recommend looking into SpiceDB metrics to determine if Spanner is the bottleneck. >I am noticing weird traces like in the attached image where spicedb seems to complete work but the grpc response completes much later. Not sure how best to interpret it. I think this was already reported as a potential issue: https://github.com/authzed/spicedb/issues/1662 >I am deploying using the spicedb-operator which currently supports up to 1.26.0 - is it safe to override it's default max and go to 1.28.0? Yes because there are no relevant spanner migrations AFAIK but let me doublecheck. We should do a new operator release with 1.28.0. 1.27.0 should be available with the latest version of the operator IIRC.
>Yes because there are no relevant spanner migrations AFAIK but let me doublecheck. We should do a new operator release with 1.28.0. 1.27.0 should be available with the latest version of the operator IIRC. Actually no, there are migrations, a new implementation of the schema cache
Let me see if we can quickly cut a release
j
> that is correct, but you are also putting more strain on the server, and for a largely enough response it may run out of memory. Cursors set a boundary to how much memory is required to be held during LookupResources. Gotcha, so in this case I don't think it's a factor but definitely something to keep in mind as we scale up. > Expensive LookupResources with a 200PUs Spanner will almost certainly get throttled by Spanner - I recommend looking into SpiceDB metrics to determine if Spanner is the bottleneck. I did so some testing with variations of PU configurations and I don't think we are hitting spanner limitations - we are still working with relatively low numbers here. I've done a bunch of tweaking to our deployment configuration and I've improved the performance slightly, but whats interesting is that it is super inconsistent: https://cdn.discordapp.com/attachments/1181536779006918676/1181609253761450036/image.png?ex=6581ae5f&is=656f395f&hm=a03eb208a80a0f93ead856f90a7a6d73fae73452c6ec0ddce1738101bb4781e9&
And notice a lot of random outlier spikes of over 800ms too
Let me get a schema reference quick
v
are you running requests at a snapshot, or with minimize_latency
I'm going to close the operator PR for now, since 1.28.0 cannot be released until it hits stable channel
instead I recommend setting 
image
in your 
SpiceDBCluster
object
the operator will do the right thing and run 
migrate
for you
j
Using minimize_latency but I can test at a snapshot if you think that will make a difference
v
it will, since cache will be usable for longer
but depending on how ok you are with stale reads, you can increase the "quantization window"
so that the caches are usable for longer time
j
I sent you a schema privately, I am running the query 
lookup-resources location read user<id>
Ok so I guess it's reasonable to assume that the cache expiring is manifesting in the spikes?
v
most likely, but that's the normal behaviour. If you run the request with 
full_consistency
you'll see the true cost of your request, as cache won't be used
j
Out in the wild we are actually using zedtokens, but for my load testing I am not - so I guess this won't be needed right? That would affect the minimize_latency consistency setting
I went ahead and did this - hard to tell for sure but I didn't notice much impact on performance
v
well at least you know that all the latest perf improvements are in place, particularly singleflight datastore and dispatchers which can save up to 33% datastore access
using 
at_least_as_fresh
is the recommended way to call SpiceDB is you are keeping zedtokens around
j
Yea I think it's just hard to tell because I have changed so many variables over the last couple hours haha
Yup this is the current state out in the wild
I'll run some tests using that too now
v
What requests are you setting to the SpiceDB pods?
is it running with burstable or best effort QoS?
how many cores does each replica have?
specially for schemas that require a lot of fan out, having core around is important. You should be able to see this with the newly exposed Go scheduler metrics, if they are stable, it means you are doing well compute wise
I have a bunch of stuff going on today but keep me posted, I'll get back to you eventually
j
We are running on gcp c2-standard-4 instances and each spicedb pod is guaranteed at least 3.5 of the 4 cores. There are no limits on CPU, each pod is configured with 2G of memory but are only using between 200-300Mi while under load.
Interesting to know - let me go digging for the metrics and maybe try on higher core nodes. When I originally posted this here we were running on significantly reduced compute (n2-standard-4 and only 200m guaranteed). I didn't suspect this as a main cause as I thought I was reproducing the issue locally but that was actually just misinterpreting my results.
v
4 cortes should be probably fine. Something SpiceDB does not do is setting gomaxprocs. Have you set that to the number of cores allocated?
j
I haven't explicitly set it, but looing at the 
go_sched_gomaxprocs_threads
metric it seems to be reported as 4. Can I trust this?
Running this locally I get very stable performance with responses in 20-30ms. Do you think the large variance in performance on our prod is largely applicable to available compute?
v
>I haven't explicitly set it, but looing at the go_sched_gomaxprocs_threads metric it seems to be reported as 4. Can I trust this? yes, that's the right metric
>Running this locally I get very stable performance with responses in 20-30ms. Do you think the large variance in performance on our prod is largely applicable to available compute? well you are using postgres locally but spanner in GCP,
as I mentioned, I'd suggest looking into the Spanner metrics
also if you are running 1.28, I suggest enabling the experimental schema cache flag, it will make things faster, but set the heartbeat to something like 2 seconds
the less roundtrips to DB the better
j
Ok, let me give this a try
v
I don't see anything in your schema that could be optimized, at first glance. It's mostly unions, and multiple levels of nesting. It's a matter of whether you have a very wide relation (e.g. something like the parent of a resource having thousands of elements, and a permission doing parent->permission_in_the_parent)
please note this is only available in spanner and cockroach, not on postgres nor mysql
j
Ok is this the 
--enable-experimental-watchable-schema-cache
flag? and then Also in conjunction with 
--datastore-schema-watch-heartbeat
?
v
correct
j
Does this affect read performance even if there is no write load?
v
the watchable schema?
it's precisely for read workloads
basically it listens to a change feed from Spanner to determine if the schema has changed, so that read requests don't have to load the schema definitions each time
if you do change the schema with 
WriteSchema
then the cache will back off until it has caught up with the new schema revisions
j
Ok I've done quite a lot of debugging and I am starting to run into a wall. Current status: + Running 10 SpiceDB (1.28.0) pods with 4 cores, 2GB ram on c2-standard-4 nodes. + Scaled spanner to 2000 PU's + Enabled the schema cache + Running lookup query for ~1000 resources (tiny volume) + Setting a 
{at_least_as_fresh}
consistency token Seeing random frequent spikes of 5s+ (sometimes 8s+) for queries against spicedb. Example, see attached image: https://cdn.discordapp.com/attachments/1181536779006918676/1181945533691084800/image.png?ex=6582e78e&is=6570728e&hm=c29284b5290132333bd66c81f6076762bc1fc23e2b68b4d9d0fc1a18caf86854&
I'm at a complete loss as to how to proceed with debugging this
I've waaay overprovisioned my resources to completely eliminate that as a cause. Querying over very small quantities of resources. I'm sure this isn't expected behaviour, but everything points at spicedb being the cause here. Can I provide any information/metrics to help identify the issue?
v
You could do 
zed backup
and send it over to us.
What are you using to do the lookup resources? 
zed
?
j
I doubt this is anything to do with the actual data, I'm sure this is some configuration issue or issue with the environment I am running on. I'm not even entirely sure this is spicedb's fault, but the slow trace goes all the way into the traces exposed by spice
v
Please note that LookupResources is a streaming API, so the latency of the call will be as large as the stream remains open. I assume this is not your case, but its relevant because some use caess the API is fast but the API call look "longer" because the client is doing processing as it receives the values and holds the stream open
it could even be a load balacer issue
(load balancer keeping the stream open)
j
That is actually my current leading theory... but the thing that is throwing me off is that spicedb 'received' the request immediately but the client only starts receiving results many seconds later as indicated by the events on the receive span
the connection is happening directly to a kubernetes service, so no other loadbalancer in the way other than internal k8s networking
v
It could be something with the data. The computation time is proportional to both the schema and the data shape. So a specific data shape could be surfacing some unoptimized codepath
j
But if this was the case wouldn't we expect to see this issue for all requests
v
I don't see a reason why 1000 results should take that long, unless it's a pathological case that is really forcing to traverse the whole dataset
j
or at least all requests that miss the cache?
v
not necessarily, it really depends on the dataset. This is a graph traversal. For some subject/resources it may require the evaluation of a smaller dataset compared to other paths.
e.g. imagine an organization with 1,000,000 members, versus an organization with 1000 members
have you tried run 
check --explain
?
j
In our case we have a total of like 3000 relationships across all resource types
v
it will tell you visually which bits are being slow
j
And we are seeing this issue for the same request in that the exact parameters of the request are the same - so the traversal path, I would imaging, would be the same
and in some cases it resolves fine (20ms) and in others it will spike to many seconds
I'm becoming more and more convinced this is just some networking bug
v
and if you do 
fully-consistency
semantics, does it always reproduce the issue?
j
no
To be honest I am struggling to reproduce this issue in isolation
it's only appearing in the wild
Actually, let me double check this
v
well that would in theory discard the graph traversal computation on SpiceDB as the issue, but we cannot discard that some synchronization issue amon the different parts of the LookupResources code EDIT: but then I would expect more a deadlock kind of scenario, and then it would timeout rather than taking long to respond
have you been able to reproduce this locally?¿
I'm trying to see what's going on with the OTel traces locally
I don't seem to be able to reproduce the spans issue locally, at least with CockroachDB
j
But the issue does happen when just pulling from the cache and not hitting spanner, I think? Because it occurs even when specifying a zookie
But definitely easier to reproduce in this configuration
Nope, only on our cluster
Also, this is the first example where I have seen more spans in the trace
v
right, I think you don't see spans because when there is a cache kit, you will miss any datastore access
but it seems to suggest that something is going on that does not involve I/O
or if it does it is not appearing in the trace
j
I can share our spice backup with you
v
have you tried using the spanner emulator locally?
j
nope didn't know that was a thing!
v
yeah, it's deliberately slow and does not support concurrency, but it exists
you can see it being used in SpiceDB integration tests
and you can spin it up locally with the gcp CLI
j
How likely is this to be a spanner issue do you think
I'll try get it setup and see
v
I don't think it's necessarily an issue with Spanner but I cannot rule out SpiceDB's implementation of Spanner
each datastore has its own bespoke implementation
did you get the chance to look into prometheus metrics on the SpiceDB-Spanner client?
I wonder if it's running out of connections. It's not something you'll see necessarily unless scrape at a very fast interval like 1s
and yeah if you can provide a backup I think that would help
j
--datastore-conn-uri="projects/project-id/instances/instance-id/databases/database-id"
- what do I use here for local spanner?
v
so like all of that you have to create with the GCP CLI
j
Let me get you that backup
v
it will create a project for you
and then inside of that project you'll create an instance
let me see if I have some commands handy
j
I have metrics collection enabled - what metrics are you interested in viewing?
Oh, was using the docker container. I can switch to cli though
v
this is what I have in my .zprofile
Copy code
spanner() {
    docker run -d -p 9010:9010 -p 9020:9020 gcr.io/cloud-spanner-emulator/emulator
    gcloud config configurations activate emulator
    gcloud spanner instances create test-instance --config=emulator-config --description="Test Instance" --nodes=1
    gcloud spanner databases create spicedb --instance=test-instance
    gcloud config configurations activate default
}
j
sweet let me try that
Ah you have some custom gcloud config
ERROR: (gcloud.config.configurations.activate) Cannot activate configuration [emulator], it does not exist.
Can you share that too?
v
huh
check if the instructions there help
it has this
Copy code
gcloud config configurations create emulator
  gcloud config set auth/disable_credentials true
  gcloud config set project your-project-id
  gcloud config set api_endpoint_overrides/spanner http://localhost:9020/
`
definitely can't seem to reproduce it with Cockroach locally. Always within < 150ms
j
I CAN REPRODUCE IT LOCALLY!
haha with the spanner emulator
I have never been so happy to see a bug
ok hold on let me just tripple check I'm not misinterpreting anything
v
hah amazing
j
Interestingly, my local spanner has all spans properly exporting
Issue seems to present the same
v
are you running the same spicedb version?
spanner traces were introduced in 1.27
j
yup this is spanner 1.28.0 - same as what we are running in prod
Copy code
Summary:
  Total:    41.3156 secs
  Slowest:    1.9387 secs
  Fastest:    0.2805 secs
  Average:    0.4968 secs
  Requests/sec:    120.5356

  Total data:    174894300 bytes
  Size/request:    35190 bytes

Response time histogram:
  0.281 [1]    |
  0.446 [4410]    |â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– â– 
  0.612 [19]    |
  0.778 [0]    |
  0.944 [0]    |
  1.110 [0]    |
  1.275 [0]    |
  1.441 [117]    |â– 
  1.607 [85]    |â– 
  1.773 [172]    |â– â– 
  1.939 [166]    |â– â– 


Latency distribution:
  10% in 0.3342 secs
  25% in 0.3428 secs
  50% in 0.3553 secs
  75% in 0.3738 secs
  90% in 1.3484 secs
  95% in 1.7319 secs
  99% in 1.8761 secs
Load test summary - the distribution matches what I am seeing in prod. Same rough proportion of random spikes
v
just note that the emulator does not support concurrency so it may be a red herring
j
Actually, certain paths present worse in prod
hmm
fair
No I think I can reproduce this with a concurrency of 1
v
Copy code
Notable limitations:


The emulator only allows one read-write transaction or schema change at a time. Any concurrent transaction will be aborted. Transactions should always be wrapped in a retry loop. This recommendation applies to the Cloud Spanner service as well.
it's actually only for writes
which I don't think is your case
j
yea this is reproduceable even with sequential requests
I can reproduce this with 
zed
cli too
hyperfine 'parallel -j {threads} -n0 zed permission lookup-resources --skip-version-check location read user:acc_fR2nH4wdUkFmo4ZG1bdQGAjsjbDe ::: {1..{threads}}' -P threads 5 20 -D 5
Copy code
Benchmark 4: parallel -j 20 -n0 zed permission lookup-resources --skip-version-check location read user:acc_fR2nH4wdUkFmo4ZG1bdQGAjsjbDe ::: {1..20}
  Time (mean ± σ):     502.7 ms ± 657.9 ms    [User: 431.6 ms, System: 471.7 ms]
  Range (min … max):   288.6 ms … 2375.0 ms    10 runs
See the massive min->max range
v
yeah I'm just not sure how much of this is due to the Spanner emulator design
you say that it's even worse in your prod env?
j
yup it's much worse in prod
It happens much more frequently and often with 10s+ spikes
v
and you are doing full-consistency or any request would do it?
j
in these tests I am reproducing with any consistency setting
v
cool, then it sounds like we have a solid repro
and does it reproduce by just even issuing individual 
zed
calls? or it has to be a load test
because I'm certainly not reproducing this locally with CRDB. It's always <150
j
It does, but interestingly the spikes are smaller
With a concurrency of 20 I see spikes of 1-2s, with a concurrency of 1 I see spikes of 500-700ms
v
yeah, as I mentioned, the streaming of responses is part of the trace
j
and this is just with zed
v
so client is part of the equation. A "slow" client will make the request appear slower
j
everything is fine and the a spike hits
The command used:
Copy code
hyperfine 'parallel -j 20 -n0 zed permission lookup-resources --skip-version-check location read user:acc_fR2nH4wdUkFmo4ZG1bdQGAjsjbDe --consistency-at-least=GhUKEzE3MDE4NzI5MDAwMDAwMDAwMDA= ::: {1..20}' --runs 1000
hyperfine just the load tester - then just executing 
zed
cli with a configurable concurrency
and this is with a consistency token - still occurs
It also seems that my CPU spikes accross all cores when a spike occurs. That's just a non-scientific eyeball observation
v
try with full consistency
I'm seeing something similar lcally, but with 
minimize_latency
I get a 500ms spike when the quantization window elapses and SpiceDB discards the caches
this may be an issue with 
ReverseQueryRelationships
. I see the queries taking 60ms
there may be a missing index
have you looked into Spanner's profiler?
j
ah, yea, sorry you're right the 
--consistency-full
triggers it every request. I guess that means it's just the emulator being slow and not actually what I am seeing in prod?
Nope - not super familiar with spanner in general
v
the spanner profiler would show you if certain queries are suboptimal. It would show what the query planner did and where time was spent
j
The spanner emulator is burning CPU while running this locally with --consistency-full
maxed out
v
well so far I don't see anything abnormal in the trace other than queries being slow. There is room for some optimization here and there, but some specific query access patterns seem slow, and it's not clear if that's a spanner emulator thing, or a problem with how it's currently implemented
even the checks are relatively slow. If I pick one of the subjectIDs returned by that 
LookupResources
and invoke 
CheckPermission
it leads to 200ms requests, where Spanner queries are taking somewhere between 20ms up to 70ms
j
I'm going to setup a test on spanner cloud and compare with pgsql
If this reproduces on spanner but not pgsql then I'd say it's fair to assume it's an issue with the spanner data store
Ok @vroldanbet I'm able to reproduce the behaviour I am facing quite reliably in an isolated environment with both postgres and spanner as backing datastores. My configuration is as follows: - SpiceDB running on one 8core 16G VM - Cloud Spanner with 1000 PU's or; - Postgres server with 8core 32G VM - The same dataset I shared with you (querying a result set of around 1.1k) Executing the following command:
Copy code
parallel -j 1000 -n0 'hyperfine --runs 50 "zed permission lookup-resources --skip-version-check location read user:acc_fR2nH4wdUkFmo4ZG1bdQGAjsjbDe --consistency-at-least GhUKEzE3MDE5MDM4OTAwMDAwMDAwMDA="' ::: {1..10}
And adjusting the concurrency from 1..x. Concurrency of 1
Copy code
Time (mean ± σ):      46.8 ms ±  32.9 ms    [User: 27.0 ms, System: 20.5 ms]
  Range (min … max):    34.5 ms … 280.5 ms    100 runs
Concurrency of 10
Copy code
Time (mean ± σ):     133.4 ms ±  76.0 ms    [User: 28.9 ms, System: 18.9 ms]
  Range (min … max):    68.5 ms … 511.9 ms    50 runs
Concurrency of 20
Copy code
Time (mean ± σ):     257.4 ms ± 124.4 ms    [User: 34.9 ms, System: 16.7 ms]
  Range (min … max):    70.4 ms … 760.2 ms    50 runs
Concurrency of 50
Copy code
Time (mean ± σ):     707.5 ms ± 481.3 ms    [User: 36.9 ms, System: 19.8 ms]
  Range (min … max):    81.2 ms … 2365.0 ms    50 runs
Concurrency of 100
Copy code
Time (mean ± σ):      1.307 s ±  0.987 s    [User: 0.038 s, System: 0.022 s]
  Range (min … max):    0.000 s …  4.543 s    50 runs
These results were all against spanner, but using postgres shows rather similar results. Do these results seem expected or surprising? I find the rapid increase in p90 latency to be a bit surprising. I also find that the latency when using 
--consistency-full
being around 200-300ms to be surprising given the low volume of data. It seems spicedb doesn't deal with concurrency very well. Even on relatively beefy VM's. Thoughts?
v
I'm out of office, perhaps the rest of the team can help: @Joey @ecordell
j
Some load testing results for both spanner and postgres. Would love if someone can take a look and confirm if they match expectations for spicedb performance. https://cdn.discordapp.com/attachments/1181536779006918676/1182262317073842236/results.tar.gz?ex=65840e95&is=65719995&hm=c1f9f61575ba3ffe3fc613276e3ed0bb526b9af94e80e254bf01377d2ae51663&
j
@julienvincent you're overwhelming the single SpiceDB pod
LR is a heavy call and if you're spawning 100+ of them, you're likely simply running into contention
j
@Joey In the above the 100x concurrency was just for completeness. We are seeing multi-second latencies when running at a concurrency of ~20-30 spread across 4+ Pods all with 4 cores each - for sample sizes in the region of 1000 resources. If this is simply contention then I don't understand how to realistically proceed with SpiceDB as a solution for ACL-aware filtering.
The performance being observed here doesn't make sense to me for the small scale we are operating at. Hence I feel something must be wrong.
For example:
Copy code
~$ time parallel -j 2000 -n0 'curl -X POST -d "@./data.json" -H "authorization: Bearer dev" http://localhost:8443/v1/permissions/resources --silent | wc -l' ::: {1..3}
3227
3227
3227

real    0m1.463s
user    0m0.150s
sys    0m0.110s
3 concurrent requests reduces this instance to a 1.5s latency
j
that doesn't match the times you posted above though
then again above could be using cache, so I don't know what changed between your runs
j
In this particular example I trippled the number of resources being returned. 1k -> 3k
j
ah, that makes sense then
especially if you have any intersections, exclusions or caveats in your schema
j
So this is expected performance? What is the recommendation then for actually practically using this API in a real system? It's not really an option to accept multi-second latencies for simple list calls.
I'll share my schema with you in a mo so you can take a look
j
for one, use pagination
loading 3000+ elements is not going to be really "useful"
what, exactly, are you trying to do with the output?
j
ACL-aware filtering. For which in order to do we would need the entire set of resources a user has access to
I don't think pagination helps in this case - we still need to fetch that full set of ids. Pagination would just make it slower by adding network latency, unless I misunderstand what you are suggesting
j
how are you using the IDs?
because imagine it was 300K or 3M IDs instead of 3K
j
in an intersection query against the database with our resources
j
and is this used to display all resources the user has visibility into
or another part of a search?
j
we expose several API's which allow querying resources and filtering on the properties of those resources and returning the results in a paginated way.
in order to support pagination we need to know the full set of resources the person has access to up front
The alternative would be to load all resources from the DB and make check calls for each one against spice - which just seems worse
j
not if the working set is smaller
if the user searches for "foo", and that has 10 resources found, bulk checking will be faster
j
well, if the user searches foo and there are 3000 matches but they only have access to 10, and it's the last 10 sequentially, we would need to load all 3000 resources and bulk check them
j
correct, but that's likely still faster than filtering on 3K items if the user has access to that many outside of the match
j
but we have no way to determine up front if any particular user has access to all or only a subset without asking the question
j
sure
j
so the performance of each approach completely depends on each individual users level of access
j
correct
j
So.. I don't know how to solve this without something like https://github.com/authzed/spicedb/issues/207
j
that's why that issue exists 🙂
at some point, either pre-filter or post-filter reaches a point of diminishing returns
j
What do people currently do out in the wild
to work around this issue
j
post filter, mostly
using bulk check
j
damn
j
because most of the time, when issuing a search, the number of results to be checked will be lower than loading "everything this user can see"
imagine if you were at GitHub size and asked "which repos can a user view"
it could be 10s of millions
j
Yea no of course - but for our current scale I was assuming this would be good enough until #207 lands
j
3K results shouldn't take as long as you're seeing, mind you
j
But it seems not
j
but I don't have insight into your cluster to know where the slowdown is
your schema could just be very deep or very wide
or both
j
I shared the schema with you fyi in private chat
j
sure, but that doesn't show me how wide it is or deep
a cursory look suggests it could be quite on both fronts
j
Would the idea here be something like - load a page out of the db, bulk check it, use the subset to construct a response page, if the response page is still too small - load more and repeat?
j
yes
the risk, of course, is that you could have to check many "pages" to find the next result
j
Yea
j
do you see the same LR lag on both spanner and Postgres?
j
Yes
It's slightly better on postgres though
j
that implies that it is indeed deep or wide (or both)
j
Do you mean by ways of number of resources in the relationship 'path'? Other than the target resource, which is around 1.1k resources, we only have around 10-20 resource for all other types
j
sure, but how long is the path to get from the user to each of those resources?
j
The problem here though is this only works for cursor pagination. Most of our API's are offset based pagination (because they are servicing frontend tables and such)
So I'm not sure there really is a way to do post-filtering for a offset-based paginated API?
at most 4
j
hrmph - something seems wrong then
are the SpiceDB pods running on the same node?
j
nope, each have their own node with 4 cores
But, my reproduce case was on an isolated VM and backing datastore
single spice process, single VM (4 cores) single spanner (with any PU config, it made no difference).
Even low concurrency would very quickly jump the latency
j
that's not surprising - 4 cores is likely being overwhelmed
j
Fair, well I'd be happy to try out the post-filtering approach - but would need a solution here ^
j
you'd bulk check the results from the pagination from your source DB
j
But that only works for cursor pagination
Or do you mean bulk check the entire resultset?
in which case there is no point in pagination
j
I'm not following
you have some API that is paginated that is matching the search filter(s)
you get back the first "page" of results, bulk check, send forward any matching, iterate
and construct a cursor to represent where you left off last time
j
Yes, so that's cursor pagination. Our API's are mostly offset based pagination
If I made a query like: 
{fitler: {}, paginate: {offset: 100, limit: 10}}
then I would need to re-iterate over all previous results in order to respond
So I don't think there is a practical way to have an offset based paginated API if we used post-filtering
j
you'd need to wrap the API in a cursor-based API, yes
but the underlying one can still be offset based, assuming it has a stable ordering
if not, all bets are off
but then, all bets are off anyway
j
But.. how?
Not if you had the set of ids you have access to up front 😛
j
like I said above
IF the underlying API is stable
then you just bulk check until you find the 
limit
items and store the offset in the returned cursor
then when invoked again with the cursor, you resume calling the underlying API and continue bulk checking
j
Well, it's more complicated than that. You can't support the same kinds of sorting in a cursor based paginated API. Everything you are sorting over has to be unique
So the main reason we have an offset based API is to support sorting over arbitrary fields
We wouldn't be able to support that if we built it on top of an underlying cursor based paginated api
j
sort of - cursors can support arbitrary fields, they just need to have a define ordering between calls with the same cursor
j
Fair.. is that true of most DB's?
that sounds like something that would be undefined behaviour
or maybe not
I guess if they didn't that would break offset based pagination too
j
yeah
I would like to see why your LR calls are so slow - if there are indeed only 3 hops
it should be fast
j
I've got to run now - hard stop coming up, but I can be available for debugging this whenever you like
This convo was super useful, I will be thinking about this a lot will try play with this post-filter approach
thanks
v
This is slightly tangential, but since we are discussing pagination: unless you are using some database that handles offseted queries in a magical way, offset pagination does not scale because each query forces the database to load all previous rows of the offset. At certain query rate you'll hit a wall with it. See https://use-the-index-luke.com/no-offset SpiceDB itself does cursor based pagination with different ordering criterias.
j
Yea I am aware of those performance issues - we were using offset due to the misconception around sorting with cursors.
Having played with and thought about the post-filtering approach I don't think it's going to be a perfectly viable approach: + We have lots of cases where we have lots of data with users only having partial access + We need to know totals up front, which would always require a full table scan We need some way to pre filter data at the db query, but yea the performance of 
LookupResources
just makes that completely unviable. If there was some way to filter the results from SpiceDB based on relationships or properties on the resources this would help a lot to mitigate this.
v
I'm afraid I don't have the bandwidth right now to look what's up making your LookupResources calls slow. And even if we made them faster, without setting some upper boundaries on the size of your LR responses, there will always be a LR response large enough that will make this overall slow. At some point even the 
WHERE IN
clauses in your database will become very slow or cause load spikes that will affect every other workload relying on it.
255 Views
PreviousNext
Powered by