844600078504951838 #spicedb

Channels

Joey

02/15/2022, 3:07 AM

in fact, if we did support a "batch" check, it would likely just dispatch in parallel itself, to take advantage of the above

Joey

02/15/2022, 3:10 AM

as an example: let's say you have a schema like so:

definition user {}

definition document {
  relation reader: user
  relation writer: user
  permission write = writer
  permission view = reader + write
}

and you want to check if the user can read and write

Joey

02/15/2022, 3:11 AM

if you issue a

CheckPermission

for

document:somedoc#view

and

document:somedoc#write

, then SpiceDB will be dispatching the subproblems of checking

document:somedoc#reader

and

document:somedoc#writer

. Those will be cached, so when you check for both, the system will only calculate each once (assuming the same user, and the CheckPermission calls being close together)

Joey

02/15/2022, 3:12 AM

that will happen even if the two "root" API requests hit different SpiceDB nodes, since SpiceDB is designed to cache on specific nodes within the cluster for specific subproblems

mixedCase

02/15/2022, 3:17 AM

ClientServer connection reuse helps, but unless it's something smarter than just reusing the TCP connection it's probably still going to hurt a little latency-wise; the N+1s I mentioned in the disclaimer usually involved connection reuse. Then again, I don't know if gRPC clients are doing something smarter. On SpiceDB's end, I'm not sure how that relates to N+1? Does SpiceDB wait some time for more requests coming down the pipe before firing off to the store? Or do you mean when the first one goes through, the second one hits an in-memory cache if the second request involves the same subject, or at least subject:resource combo? If so, which one is it? Thanks for the detailed answer btw.

Joey

02/15/2022, 3:18 AM

> the second one hits an in-memory cache if the second request involves the same subject, or at least subject:resource combo if the second request matches the resource, permission and subject exactly (and consistency request), then we use the cached result immediately

Joey

02/15/2022, 3:18 AM

since CheckPermission requests almost always subdivide into subproblems, and most of those are shared, that cache hit rate can be quite high

Joey

02/15/2022, 3:19 AM

we use a consistent hashring to try to run the same subproblems on the same nodes, ensuring they can use the in-memory cache

Joey

02/15/2022, 3:22 AM

but yeah, the recommendation for now is to use multiple goroutines to execute the checks in parallel; this will still incur a slight overhead in the actual requestresponse, but that should be minimal

mixedCase

02/15/2022, 3:26 AM

Got it. It'll have to do. Thanks Joey!

Joey

02/15/2022, 3:26 AM

my pleasure

Joey

02/15/2022, 3:26 AM

if you do find that the overhead is too large, please file an issue and we can see about either optimizing that further or adding a batch check

vroldanbet

02/15/2022, 12:42 PM

hey y'all, what would be the best way to determine if cluster dispatching is working as intended and spicedb processes are dispatching to another process in the cluster?

vroldanbet

02/15/2022, 12:43 PM

Since this happens at the gRPC level with the resolver registered by

kuberesolver

, it seems to be somewhat opaque to SpiceDB?

vroldanbet

02/15/2022, 12:45 PM

I guess one way could be to check

grpc_method

tag in logs and determine if e.g.

DispatchCheck

. is being invoked?

user

02/15/2022, 4:29 PM

the best way is to look at

spicedb_dispatch_check_from_cache_total

which is the metric for cache hits from a remote dispatch

user

02/15/2022, 4:30 PM

I think using verbose logging should probably work too, but we have graph our cache hits from dispatch clients and servers

vroldanbet

02/15/2022, 6:24 PM

thanks @User ! a follow up question: for

kuberesolver

. to be able to talk to the Kube Server API and resolve endpoints, I assume it needs some

ClusterRole

. or the like? Is there any documentation or guidance around configuring the upstream?

ecordell

02/15/2022, 6:26 PM

the serviceaccount for spicedb just needs to have

get

list

, and

watch

for

endpoints

ecordell

02/15/2022, 6:27 PM

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: watchEndpoints
rules:
- apiGroups:
  - ""
  resources:
  - endpoints
  verbs:
  - get
  - list
  - watch

vroldanbet

02/15/2022, 6:28 PM

wonderful, thanks for the hint @User ! 🙇🏻‍♂️

vroldanbet

02/15/2022, 6:33 PM

I noticed that when cluster dispatch is incorrectly setup (e.g. watch gets 403), calls to SpiceDB (e.g. zed permission check) would hang forever and never return. What I see in the logs is the following:

log: 2022/02/15 18:19:11 ERROR: kuberesolver: watching ended with error='invalid response code 403 for service spicedb-internal in namespace spicedb-staging', will reconnect again

Also noticed that the log from kuberesolver is not doing structured logging

Joey

02/15/2022, 6:45 PM

@User ^

ecordell

02/15/2022, 6:58 PM

not sure if kuberesolver would be open to changing their logging, but we could ask the spicedb behavior seems reasonable for initial startup. we don't want to fail fast on connecting to dispatch peers (kube api is not expected to be as highly available as spicedb). we should probably have some metrics that you can hang alerts off of so that you know if the peer list isn't being updated arguably we should fall back to hitting the datastore directly if dispatch is unavailable, but I think at best that should be a flag since that could cascade a spicedb node failure into much higher than normal datastore load and potentially affect the entire spicedb cluster

ecordell

02/15/2022, 7:03 PM

maybe we could tie dispatch health into the readiness check, so that if dispatch is unavailable the pod doesn't receive traffic

vroldanbet

02/15/2022, 7:11 PM

right, I could see "dispatch hedging" happening here

vroldanbet

02/15/2022, 7:11 PM

wasn't hedging already happening at dispatch time?

ecordell

02/15/2022, 7:13 PM

datastore requests are hedged but not dispatch requests

ecordell

02/15/2022, 7:16 PM

hedging between local and remote dispatch would mess up the caching tree - we want to dispatch before hitting the database so that the node responsible for that key caches the response

ecordell

02/15/2022, 7:18 PM

but the issue here isn't really hedging, it's that grpc isn't able to find any endpoints to talk to with the config it's been given I think that if we make the node unavailable until it gets an initial dispatch peer list + some metrics you could alert on if dispatch isn't able to refresh the peer list for some period of time would fix the issue? You wouldn't see requests hang because the spicedb pods wouldn't be available until they could get the peers from the kube api, but once they have that list they use the peers they have and just make noise if they have trouble keeping up