https://authzed.com logo
Join Discord
Powered by
Debugging
# spicedb
t
then what else should I check to figure out the issue? @Joey , @ecordell
e
Do you see any errors in the logs? When you say init pod, what do you mean?
t
I mean the 
migrate-...
pod
The error I see on the operator is this:
Copy code
E0905 20:28:36.904837       1 handler.go:31] "requeueing after api error" err="Operation cannot be fulfilled on spicedbclusters.authzed.com \"spicedbprod0\": the object has been modified; please apply your changes to the latest version and try again" syncID="jtdD7" controller="spicedbclusters" obj={"name":"spicedbprod0","namespace":"prod"}
@ecordell there is no latest version. There is only one
e
what do the logs of the 
migrate-
pod say?
you can ignore the 
Operation cannot be fulfilled
message
t
Copy code
{"level":"info","format":"auto","log_level":"debug","provider":"zerolog","async":false,"time":"2024-09-23T23:02:37Z","message":"configured logging"}
{"level":"info","v":0,"provider":"none","endpoint":"","service":"spicedb","insecure":false,"sampleRatio":0.01,"time":"2024-09-23T23:02:37Z","message":"configured opentelemetry tracing"}
{"level":"warn","this-version":"v1.33.1","latest-released-version":"v1.36.2","time":"2024-09-23T23:02:37Z","message":"this version of SpiceDB is out of date. See: https://github.com/authzed/spicedb/releases/tag/v1.36.2"}
{"level":"warn","newCommand":"spicedb datastore migrate","time":"2024-09-23T23:02:37Z","message":"use of deprecated command"}
{"level":"info","time":"2024-09-23T23:02:37Z","message":"migrating postgres datastore"}
{"level":"info","targetRevision":"add-rel-by-alive-resource-relation-subject","time":"2024-09-23T23:02:37Z","message":"running migrations"}
{"level":"debug","pgx":{"args":[],"commandTag":"SELECT 1","pid":660443,"sql":"SELECT version_num from alembic_version","time":2.76518},"time":"2024-09-23T23:02:37Z","message":"Query"}
{"level":"info","targetRevision":"add-rel-by-alive-resource-relation-subject","time":"2024-09-23T23:02:37Z","message":"server already at requested revision"}
e
And then do the spicedb pods start up? Do they have any logs?
Is there status on he SpiceDBCluster object?
t
they don't
the migirate is in UNREADY state
Copy code
spicedbprod0-migrate-n654hf8h5bfh65b-57grj         1/2     NotReady   2          16h
@ecordell let me know if there is an easy way to debug this. We are trying to make a decision between SpiceDB and Permify and the operator was a huge plus for us.
j
@TasomT what does the status for the pod show?
e
could you actually share the whole pod definition and job definition? something seems off - there shouldn't be 2 containers in the migrate job. Do you have something that injects sidecars?
t
@Joey that is the status for the pod, here is the complete state: @ecordell yes, there is ISTIO sidecar getting injected
j
ah
yeah, that's likely breaking it
we've had other users report ISTIO breaks the networking between pods
t
how can I add annotation to disable the injection here?
@Joey basically where can I put this in the annotations: 
sidecar.istio.io/inject: "false"
e
yep you can put it in the 
config
of the object:
Copy code
spec:
  config:
    extraPodAnnotations:
      sidecar.istio.io/inject: "false"
or
Copy code
spec:
  config:
    extraPodAnnotations: "sidecar.istio.io/inject=false"
t
thanks, now I'm out of miggirate
e
did the spicedb pods come up healthy?
t
yes, they did. Now checking if we can restore to it or not
77 Views
PreviousNext
Powered by