https://authzed.com logo
Join Discord
Powered by
Readiness issue
# spicedb
d
Hello again, We're using the SpiceDB operator to launch a 3 replica cluster, with an AWS RDS Postgres DB. It's mostly working well, but often when restarting a pod, and frequently when restarting the generated deployment, at least one pod will enter an error state. Sometimes they recover after some restarts, sometimes they stay in that state until the . Logs in the thread. Any advice would be much appreciated!
Copy code
{"level":"warn","dispatchReady":false,"time":"2023-09-01T16:37:42Z","message":"dispatcher failed readiness checks: found expected state when trying to connect to cluster: TRANSIENT_FAILURE"}
{"level":"info","requestID":"12ac564ddc7337ac3209ac7f80da58ec","protocol":"grpc","grpc.component":"server","grpc.service":"grpc.health.v1.Health","grpc.method":"Check","grpc.method_type":"unary","peer.address":"[::1]:52312","grpc.start_time":"2023-09-01T16:37:42Z","grpc.request.deadline":"2023-09-01T16:37:43Z","grpc.code":"OK","grpc.time_ms":"0.086","time":"2023-09-01T16:37:42Z","message":"finished call"}
{"level":"warn","dispatchReady":false,"time":"2023-09-01T16:37:43Z","message":"dispatcher failed readiness checks: found expected state when trying to connect to cluster: TRANSIENT_FAILURE"}
{"level":"info","requestID":"715c5bb7d8e5a190dd3df93e2820869d","protocol":"grpc","grpc.component":"server","grpc.service":"grpc.health.v1.Health","grpc.method":"Check","grpc.method_type":"unary","peer.address":"[::1]:52318","grpc.start_time":"2023-09-01T16:37:43Z","grpc.request.deadline":"2023-09-01T16:37:44Z","grpc.code":"OK","grpc.time_ms":"0.07","time":"2023-09-01T16:37:43Z","message":"finished call"}
{"level":"warn","dispatchReady":false,"time":"2023-09-01T16:37:43Z","message":"dispatcher failed readiness checks: found expected state when trying to connect to cluster: TRANSIENT_FAILURE"}
{"level":"warn","dispatchReady":false,"time":"2023-09-01T16:37:45Z","message":"dispatcher failed readiness checks: found expected state when trying to connect to cluster: TRANSIENT_FAILURE"}
{"level":"warn","dispatchReady":false,"time":"2023-09-01T16:37:46Z","message":"dispatcher failed readiness checks: found expected state when trying to connect to cluster: TRANSIENT_FAILURE"}
{"level":"warn","dispatchReady":false,"time":"2023-09-01T16:37:49Z","message":"dispatcher failed readiness checks: found expected state when trying to connect to cluster: TRANSIENT_FAILURE"}
j
this usually indicates that the pod cannot reach the other pod(s) its configured to reach for dispatch
is there typically a single pod coming into service or multiple?
d
Usually single, as there's a pbd in place, but the issue is always seen if there are two starting at the same time. Oddly when starting fresh (3 pods simultaneously), it's fine.
j
yeah, it might be hitting the other pod
which is, itself, not ready
d
I do sometimes see this when only one is restarting, or I'd be happy to agree with you.
Let me go have a look, and see if anything has resolved itself in the last few hours.
j
k
d
yup, looks stable now, after 74 restarts
but restarting a pod puts it back into the same error state

https://cdn.discordapp.com/attachments/1147211541800812609/1147287939039113368/Screenshot_2023-09-01_at_22.52.04.png▾

j
oh
I don't know if restarting an existing pod will work
its possible the networking gets confused
d
that's certainly what we're seeing here. What happens in the event of a pod failure, or rebalance event, then?
or, for example, running a helm upgrade that affects the SpiceDBCluster resource?
j
new pods will be put in place whenever the operator detects a change
a pod should never be reused
d
ok, that's promising for the future. Restarting was the wrong word, sorry.. I deleted the pod, which triggered a single new pod to be created
j
which should always work
if it doesn't, something is misconfigured
d
agreed!
we installed the operator with:
Copy code
kubectl apply --server-side -f https://github.com/authzed/spicedb-operator/releases/latest/download/bundle.yaml
and the SpiceDBCluster:
Copy code
apiVersion: authzed.com/v1alpha1
kind: SpiceDBCluster
metadata:
  name: {{ or .Release.name .Values.name }}
  namespace: {{  .Values.namespace }}
spec:
  config:
    datastoreEngine: postgres
    replicas: {{ .Values.replicas }}
    tlsSecretName: {{ .Values.tls.secretname }}
    image: {{ .Values.imageRepository }}:{{ .Values.dockerTag }}
  patches:
    - kind: Deployment
      patch: 
        spec:
          template:
            spec:
              tolerations:
                - key:  workload
                  operator: Equal
                  value: auth     
                  effect: NoSchedule
              nodeSelector:
                workload: auth
              affinity:
                podAntiAffinity:
                  requiredDuringSchedulingIgnoredDuringExecution:
                    - labelSelector:
                        matchExpressions:
                          - key: authzed.com/cluster-component
                            operator: In
                            values:
                              - spicedb
                      topologyKey: "kubernetes.io/hostname"
  secretName: {{ or .Release.name .Values.name }}-config
j
@ecordell might have more insights as well
e
You’re setting a tls secret, but not an upstreamDispatchCASecretName, which means that spicedb won’t trust its peers
You can either stick the CA in a secret and pass that in so the peer traffic is trusted
Or disable tls for peer dispatch
This doesn’t happen all the time because the pods don’t see each other on initial start, they think they don’t have any peers
Only after they’ve been running and one restarts, it will see its peers and the health check fails (the health check only runs on start and then latches open)
d
Oh, interesting, ty. I suspect that disabling tls for peer dispatch is preferable for now. Any chance you fancy saving me a dig through the docs to find the setting?
e
d
fantastic, thank you
thanks again for the help - slowly understanding things a bit. I'll no doubt be back with my next misconfiguration!
I see you reccommend tls for dispatch. Is it strongly reccommended when within an EKS cluster/node group?
e
It’s not a big deal to not use tls if that’s generally what you do for internal traffic, but dispatch is grpc and not all grpc ecosystem tools work well without tls
d
yup, I can now delete pods and restart deployments to my heart's content. Pods come straight back up. Thank you both for the explanation.
18 Views
PreviousNext
Powered by