I m trying to use Spicedb Serverless and SpiceDB #spicedb

I m trying to use Spicedb Serverless and

jure0888

08/30/2023, 10:14 AM

I'm trying to use Spicedb Serverless and I keep getting StatusCode.UNAVAILABLE when doing LookupResources, if I resend the request I get back corrent response. Is there something extra I need to be aware of? (I don't see this problem with local docker)

jure0888

08/30/2023, 10:17 AM

Copy code

for response in resp:
  File "/home/vagrant/.local/lib/python3.11/site-packages/grpc/_channel.py", line 541, in __next__
    return self._next()
           ^^^^^^^^^^^^
  File "/home/vagrant/.local/lib/python3.11/site-packages/grpc/_channel.py", line 967, in _next
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.UNAVAILABLE
    details = "Socket closed"
    debug_error_string = "UNKNOWN:Error received from peer ipv4:130.211.126.102:443 {grpc_message:"Socket closed", grpc_status:14, created_time:"2023-08-30T10:13:11.847162184+00:00"}"

Python code:

Copy code

resp = client.LookupResources(
    LookupResourcesRequest(
        resource_object_type='prefix/user',
        permission="role",
        subject=SubjectReference(
            object=ObjectReference(
                object_type='prefix/role',                     object_id=str(role_id)
            )
        ),
    )
)
for response in resp:
    users.add(int(response.resource_object_id))

vroldanbet

08/30/2023, 11:19 AM

all I see on the server side is

"rpc error: code = Unavailable desc = transport is closing"

. According to https://chromium.googlesource.com/external/github.com/grpc/grpc-go/#the-rpc-failed-with-error, this means the underlying HTTP connection was closed, so I'd suspect something related to our load balancer. How easy is for you to reproduce this? have you tried running the same request with

zed

to see if it could be client-induced somehow?

jure0888

08/30/2023, 11:45 AM

I can't consistently reproduce it. I'll let you know if I find a pattern.

vroldanbet

08/30/2023, 11:51 AM

I don't see anything obvious in the load-balancer, other than you seem to be load-testing and creating tons of connections / requests with your app. Is this a single python application with multiple threads, each one with its own spicedb client?

vroldanbet

08/30/2023, 11:57 AM

Please note there are rate limits in place, if you hit it, you should see

RESOURCE_EXHAUSTED

error

jure0888

08/30/2023, 12:03 PM

@vroldanbet hmm, it's not supposed to be load testing. I was hoping to use this pattern with actual app. Obviously rethinking this mode of usage now. Yes, I create a new client for each request. I'll experiment with using only one client.

vroldanbet

08/30/2023, 12:12 PM

That explains why so many connections - you are not using HTTP2 multiplexing. I don't know what the pattern looks like, but I'm surprised of the amount of

LookupResources

requests, it's not what we typically see, and definitely not in a local dev environment application. Wha are you using

LookupResources

for? Have you considered using pagination?

vroldanbet

08/30/2023, 12:23 PM

It also look like your

LookupResources

are leading to 1MB worth of data sent, which suggests very long responses, so definitely recommended looking into pagination

vroldanbet

08/30/2023, 12:34 PM

My suspicion is that somehow your application is inadvertedly closing the underlying HTTP connection, and reusing a connection that is already closed. I don't see metrics in our load-balancer indicating the load-balancer itself terminating the connections at the time your reported, but connections terminated by the client.

jure0888

08/31/2023, 3:53 PM

I can reproduce it and have a basic python script that crashes every time. I've sent it to support@ as it contains private serverless access data.

4 Views

Previous Next