https://authzed.com logo
Join Discord
Powered by
Does anyone know if there is a way to
# spicedb
t
Does anyone know if there is a way to increase the 
grpc_health_probe
timeout value? We hit an issue where we had some heavy load that got us into a restart loop due to the healthcheck timing out. We're increasing our base number of replicas too but thought it would be nice to make this a little longer to prevent it from restarting. We're using the spicedb-operator to deploy.
Copy code
timeout: failed to connect service "localhost:50051" within 1s
  Warning  Unhealthy  55m (x6 over 12d)  kubelet  Readiness probe failed: command "grpc_health_probe -v -addr=localhost:50051" timed out
  Warning  Unhealthy  55m                kubelet  Readiness probe failed: parsed options:
One other odd thing, the error seemed to indicate the timeout is 1s but when I describe the pod I see 5s
Copy code
Liveness:       exec [grpc_health_probe -v -addr=localhost:50051] delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:      exec [grpc_health_probe -v -addr=localhost:50051] delay=0s timeout=5s period=10s #success=1 #failure=5
y
so i'm not sure where it's getting the 1s
it looks like there's now a way to do it natively, so that utility might not even be necessary: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-grpc-liveness-probe
so i think it's probably something where the default of the utility is being used and the additional configuration isn't actually applied
i asked and the 5s is a timeout of the call to the command used in the health probe rather than something that's plumbed through to the probe itself, so the probe is still using the 1s default for the 
grpc_health_probe
command
t
Ok got it, thanks for diving in!
y
t
Oh didn’t recognize the handle lol, I was wondering if you had started
y
hahaha yeah that was yesterday ^.^
36 Views
PreviousNext
Powered by