https://authzed.com logo
Join Discord
Powered by
Watch API: stream terminated by RST_STREAM with er...
# spicedb
k
Hello guys, need help with unexpected issue.. While using WatchClient from watch_service_grpc and method 
watchClient.Recv()
from it, sometimes we get an error : 
"error":"rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: NO_ERROR"
Maybe someone knows the reasons and how to fix this behavior. We use the last version of SpiceDB Thanks 🙂
v
This typically happens when there is a gRPC stream timeout somwhere between the client and SpiceDB. Are you using a load-balancer?
k
Yes, of course
v
what load-balancer are you using?
the general recommendation is to retry on those errors. These errors emerge when connections timeout, either a normal lifetime timeout or an idle timeout, and are more commong on streaming API calls due to their long-lived nature, like SpiceDB's 
Watch
API. I ask the question about the load-balancer because I'm aware that for example Envoy does not support forwarding client-side provided timeout configuration, which gRPC client allows you to change: https://pkg.go.dev/google.golang.org/grpc/keepalive?utm_source=godoc#ClientParameters
k
if in general, we setup kubernetes load balander, or u need more detailed settings?
v
I don't understand your question
For reference, Envoy's limitation with gRPC/H2 ping https://github.com/envoyproxy/envoy/issues/6464
k
I asked our devopses about load-balancer, will waiting for an answer 🥲
v
The very least you can do is configuring the timeout in your gRPC client, specially setting idle timeout. Theoretically a connection with doing idle connection pings should never timeout. If you configure that and the timeout happens, you know it's the load-balancer.
k
I will try, thank u 🙂
we are using Istio for balancing
v
right, which uses Envoy, which has the problem I described above. Unfortuantely I'm not aware of any workaround to this other than retrying on the client side or disabling timeouts, which is not a great idea either
thus my recommendation would be to reconnect when those errors happen
j
or any errors, for that matter
k
Thanks a lot, guys!
75 Views
PreviousNext
Powered by