Re this issue I m still hitting this with v1 11 0 I ve dug a

Question

Re this issue I m still hitting this with v1 11 0 I ve dug a little bit The `grpc health probe` is returning `NOT SERVING` screenshot 1 On SpiceDB side I m seeing this log ` level debug datastoreReady false dispatchReady true time 2022 08 08T12 20 53Z message completed dispatcher and datastore readiness checks ` That s with LOG LEVEL=trace My old service on `v1 7 1` has no problem pods report healthy and can connect to the DB Spinning up a new pod on `v 1 7 1` works as well I can t see anything wrong database side How can we know why does SpiceDB think that the datastore isn t ready

williamdclt · Answer

I can connect to the DB from a `v1 11 0` pod with `psql $DBURL ` DBURL is what s passed as ` datastore conn uri`

williamdclt · Answer

I bisected which release introduced this issue everything works fine until `v1 10 0` so it must be a commit in this release

ecordell · Answer

The datastore will report `ready=false` if the migration level it s talking to doesn t equal the latest migration when you upgraded did you run `spicedb migrate`

williamdclt · Answer

I was looking at that indeed my migration hook is a post upgrade so that makes sense What s the recommended way to run migrations Is it a `pre upgrade` hook

williamdclt · Answer

Would be great to get more information when health checks fail I had to go into the code to figure out what could be the issue and realise it might be migrations

ecordell · Answer

Yeah run `spicedb migrate head` from the version you re updating to before rolling out the new version

ecordell · Answer

Messaging could be better here for sure do you want to file an issue or would you like me to

williamdclt · Answer

I can do that quickly

ecordell · Answer

BTW we are planning to open source a kube operator that if you use it will mean you won t have to worry about migrations since it will handle all of that

Joey · Answer

The improved migrations system will also make this more apparent

Joey · Answer

I intend better messaging on that health check once it is in