https://authzed.com logo
Join Discord
Powered by
Hi! We are setting up SpiceDB in AWS ECS
# spicedb
a
Hi! We are setting up SpiceDB in AWS ECS and are looking at enabling Dispatch. The documentation https://authzed.com/docs/spicedb/configuring-dispatch talks about K8s a lot, and then it says: > The 
upstream-addr
should be the DNS address of the load balancer at which all SpiceDB nodes are accessible at the default dispatch port of 
:50053
. Does it mean that it should really be a balancer in a classical sense when a request is being forwarded to one of the nodes? Or should it be a fun-out so that a request would hit all the nodes?
v
I think the documentation is not accurate here. We haven't experimented with AWS ECS, as ECS has its own mechanism for peer discovery as far as I understand, so it may potential involve adding the AWS SDK to SpiceDB. I do know gRPC load balancer also supports DNS load balancing, so maybe there is an avenue there but please do note that DNS TTL can can cause dispatch disruption as the peers are not updated in a timely manner. For now we don't know of folks that have dispatch enabled in ECS. The supported strategies are: - Kubernetes by using the Server API to discover 
endpoints
of peers - XDS (I'm not aware of anybody that has used it)
The way dispatch works is that SpiceDB will do client-side load balancing by discovering other peer nodes
And just to be clear, this is a gRPC feature, not something specific to SpiceDB. Whatever gRPC supports, SpiceDB will. See https://grpc.io/docs/guides/custom-name-resolution/
The docs explain the caveats of DNS based resolution very well: >One reason why it might be particularly useful to use a custom name resolver rather than standard DNS is that this interface is reactive. Within standard DNS, a client looks up the address for a particular service at the beginning of the connection and maintains its connection to that address for the lifetime of the connection. However, custom name resolvers may be watch-based. That is, they can receive updates from the name server over time and therefore respond intelligently to backend failure as well as backend scale-ups and backend scale-downs.
This is an example of how to register a AWS CloudMap gRPC Resolver for service discovery: https://github.com/aws-samples/aws-cloudmap-grpc-name-resolver/blob/main/client/resolver.go#L43 and a library with an implementation: https://github.com/ab180/grpc-cloudmap-resolver
The library seems solid, would AWS CloudMap would be an option for y'all?
Opened https://github.com/authzed/spicedb/pull/1620 in case you want to give it a go
b
What happens when dispatch fails? It'll just behave as if it wasn't cached and run the query on the current node instead?
v
I think if dispatch fails the request fails, I'm not aware the code having any retry policies. @User ?
j
a failed dispatch returns an error
2 Views
PreviousNext
Powered by