I have a deployment of spicedb + postgresql on kubernetes. I'm not using the operator. I noticed that after a while all pods except one become unready. Recently I upgraded to using TLS everywhere I am not sure if that is related. This is what I see in the logs of the unready pods:

{"level":"debug","datastoreReady":true,"dispatchReady":true,"time":"2022-10-17T17:14:38Z","message":"completed dispatcher and datastore readiness checks"}
{"level":"info","grpc.component":"server","grpc.method":"Check","grpc.method_type":"unary","grpc.service":"grpc.health.v1.Health","peer.address":"[::1]:56680","protocol":"grpc","requestID":"029dcbed3eb352a21117e9de95e9e401","grpc.request.deadline":"2022-10-17T17:14:39Z","grpc.start_time":"2022-10-17T17:14:38Z","grpc.code":"OK","grpc.time_ms":"0.015","time":"2022-10-17T17:14:38Z","message":"started call"}
a bunch of these and then ...
{"level":"info","time":"2022-10-17T17:16:27Z","message":"received interrupt"}
{"level":"info","time":"2022-10-17T17:16:27Z","message":"shutting down"}
{"level":"warn","error":"context canceled","time":"2022-10-17T17:16:27Z","message":"completed shutdown of postgres datastore"}
{"level":"info","addr":":50051","network":"tcp","prefix":"grpc","time":"2022-10-17T17:16:27Z","message":"grpc server stopped listening"}
{"level":"info","addr":":8080","prefix":"dashboard","time":"2022-10-17T17:16:27Z","message":"http server stopped serving"}
{"level":"info","addr":":9090","prefix":"metrics","time":"2022-10-17T17:16:27Z","message":"http server stopped serving"}
{"level":"info","addr":":50053","network":"tcp","prefix":"dispatch-cluster","time":"2022-10-17T17:16:27Z","message":"grpc server stopped listening"}

are there any events for the pods? kubernetes may be killing them / moving them

no...

"dispatch" is the name we use for the connection between peer spicedb nodes

ok, se here is my dispatcher related properties:

- name: SPICEDB_DISPATCH_UPSTREAM_ADDR
          value: 'kubernetes:///myspicedb.spicedb-test:dispatch'
        - name: SPICEDB_DISPATCH_CLUSTER_ENABLED
          value: 'true'

and for command line argument:

- '--dispatch-cluster-tls-cert-path'
        - /certs/tls.crt
        - '--dispatch-cluster-tls-key-path'
        - /certs/tls.key
        - '--dispatch-upstream-ca-path'
        - /ca/service-ca.crt

what datastore are you using?

postgresql but that seems to be fine

oh sorry you said postgres

so when the dispatcher is not ready, also the pod is not ready correct?

correct, but the check is only on startup. once it goes healthy once, it will stay healthy, we don't flip it back unhealthy if there's a network issue the exception is on the very first start of a cluster, it's possible for pods to not find any peers to connect to yet and therefore mark dispatch ready even though the connection hasn't been tested. (wrote this up here: https://github.com/authzed/spicedb/issues/814)

because I don't think I my cert has that name. it will have

myspicedb.spicedb-test.svc

yeah they need to match. I think it will work if you add

.svc

to the url, but haven't tested

You need to a certificate with name service-name.namespace in order to connect with TLS to your services. this is bad, I can's generate such certificate...

what prevents you from making such a cert?

I'm using an operator that only generates standard kube name certs

what operator? we use cert-manager with spicedb

this is an openshift feature: https://docs.openshift.com/container-platform/4.11/security/certificates/service-serving-certificate.html#add-service-certificate_service-serving-certificate

yeah it looks like it doesn't create certs without the `.svc`: https://github.com/openshift/service-ca-operator/blob/42088528ef8a6a4b8c99b0f558246b8025584056/pkg/controller/servingcert/controller/secret_creating_controller.go#L351-L366

it looks like it worked with the new certificate.

if you run with debug log level you should see slightly more detail

I fixed it. I had made tls aware only the readiness check, and not the liveness check