https://authzed.com logo
#spicedb
Title
# spicedb
w

williamdclt

08/08/2022, 12:27 PM
Re: this issue I'm still hitting this with v1.11.0. I've dug a little bit: - The
grpc_health_probe
is returning
NOT_SERVING
(screenshot 1) - On SpiceDB side, I'm seeing this log:
{"level":"debug","datastoreReady":false,"dispatchReady":true,"time":"2022-08-08T12:20:53Z","message":"completed dispatcher and datastore readiness checks"}
. That's with LOG_LEVEL=trace. My old service on
v1.7.1
has no problem: pods report healthy and can connect to the DB. Spinning up a new pod on
v.1.7.1
works as well. I can't see anything wrong database-side. How can we know why does SpiceDB think that the datastore isn't ready?
I can connect to the DB from a
v1.11.0
pod with
psql "$DBURL"
(DBURL is what's passed as
--datastore-conn-uri
)
I bisected which release introduced this issue: everything works fine until
v1.10.0
, so it must be a commit in this release
e

ecordell

08/08/2022, 1:12 PM
The datastore will report
ready=false
if the migration level it's talking to doesn't equal the latest migration - when you upgraded, did you run
spicedb migrate
?
w

williamdclt

08/08/2022, 1:18 PM
I was looking at that indeed, my migration hook is a post-upgrade so that makes sense. What's the recommended way to run migrations? Is it a
pre-upgrade
hook?
Would be great to get more information when health checks fail! I had to go into the code to figure out what could be the issue and realise it might be migrations 🙂
e

ecordell

08/08/2022, 1:20 PM
Yeah, run
spicedb migrate head
from the version you're updating to before rolling out the new version
Messaging could be better here for sure, do you want to file an issue or would you like me to?
w

williamdclt

08/08/2022, 1:20 PM
I can do that quickly
e

ecordell

08/08/2022, 1:21 PM
BTW, we are planning to open source a kube operator that (if you use it) will mean you won't have to worry about migrations, since it will handle all of that
j

Joey

08/08/2022, 1:24 PM
The improved migrations system will also make this more apparent
I intend better messaging on that health check once it is in