m
Hi While upgrading spicedb if we face any issue with upgraded version, any documentation on how can we revert to older version of spicedb. Thanks
v
there is no documentation of reverts. Which version are you at and what do you want to revert to?
also, what is the issue?
m
We are analysing the revert options, because we are upgrading from v1.2.0 to latest manually. Our data store is postgres and we want to figure-out the rollback strategy beforehand if happen unexpected how fast can we revert to previous version. We are running spicedb in kubernetes as a stand alone deployment along with our application.
@vroldanbet QQ : Any reason why spicedb not have rollback strategy is there any specific reason?
v
You can rollback SpiceDB, so long it's not a 4-phase migration. If there is a 4-phase migration, there is no way back, because the process the data model will be changed in an incompatible way. Postgres datastore has at least one 4-phase migration that takes place in version v1.14, so that'd be the sensitive one. Here is a list of all the database migrations in the postgres datastore: https://github.com/authzed/spicedb-operator/blob/main/validated-update-graph.yaml#L255-L353
So up to 1.14 it should in theory be possible to rollback. Once you are past 1.14-phase2, there is no way back
m
Thanks @vroldanbet
@vroldanbet
spicedb migrate head
migrates the datastore to latest version of spicedb. Correct me if I am wrong. If I want to migrate to any specfic version ex v1.10.0, can I use
spicedb migrate
in that case?
v
That is correct, but perhaps to further clarify: it migrates to the latest datastore version of current spicedb version. So if you run
spicedb migrate head
with SpiceDB 1.13, it will migrate to the latest schema of that version.
if you wan to migrate to v1.10.0, the you need to use the docker container for SpiceDB v1.10.0 and run
spicedb migrate head
with it.
m
Got it
v
my recommendation would be to bite the bullet and migrate to the
spicedb-operator
instead of keeping your own deployment. That way migrations will be handled automatically for you next time.
m
With kubernetes-operator can we control the version upto which we want to update. Reason for asking this we don't want to jump from v1.2.0 to latest in single go our service manages auth and permission for multiple teams and we can't handle any downtime on this. And if we use kubernetes operator any kind of fault toleration is there? Also if there is any change on the API side and our service stuck because of that it can be a huge impact for us.
v
yes you can control the version
I understand you don't want to do a big version jump - the operator has you covered with various options like version pinning and update-channels
The operator also handles when cluster has issues while rolling and does not remove the previous version until the new cluster is healthy
re: API unless you are in v0 spicedb API, there is no API breakage. Again, the operator supports pinning to specific versions
m
Great, let us evaluate this also. > cluster has issues while rolling We are just running single container of spicedb as a separate deployment along with our application deployment I hope that is not an issue. We just need to deploy the spicedb operator in the same namespace and pin the version upto which we want to upgrade. Correct?
v
>We are just running single container of spicedb as a separate deployment along with our application deployment I hope that is not an issue. I don't see anything in this setup that would be an issue, it's the usual deployment model for SpiceDB. It's just a regular deployment using SpiceDB right? not injecting it as sidecar or anything like that? and you have more than 1 replicas I assume >We just need to deploy the spicedb operator in the same namespace and pin the version upto which we want to upgrade. Correct? IT does not have to be in the same namespace, unless you don't want to be doing cluster roles. I'd strongly recommend you run this whole procedure in an staging environment first, so you do a trial run of the various steps. The SpiceDB-Operator will create its own SpiceDB Deployment after your apply
SpiceDBCluster
custom resource to your kube cluster. Ideally you pin it to the same version you are currently running,
1.2.0
, and point it to the same database as your SpiceDB deployment. Once you know it's healthy and working, you would switch your client application to use the new spicedb-operator created
Deployment
, and delete the old adhoc
Deployment
. Now you can start upgrading via the operator.
m
>It's just a regular deployment using SpiceDB right? yes
Thanks for the detailed answer.
Hi @vroldanbet, If I am upgrading from v1.13.0 to v1.14.0 which includes 4 phase migration for postgres datastore and I can take some downtime while upgrading. My deployment for v1.13.0 is up and running, how can I stop the current running version scaling down deployment for v1.13.0 to zero is fine? And then rollout deployment with v1.14.0 which already have
spicedb migrate head
script in the deployment. Reference Release docs: https://github.com/authzed/spicedb/releases/tag/v1.14.0
v
@Manish 👋 it would be fine to scale to zero and run the migration. However, the issue with this migration is that it's a data migration, so the duration of the downtime is heavily dependent on the amount of data stored in your postgres - it has to go over all the rows in the database and perform a transformation over them. It may not be a "short" downtime, it could take from several minutes to even hours depending on the amount of data you stored.
Steps would be: - scale in your 1.13 to zero replicas - run
spicedb migrate head
with the 1.14 migration - scale 1.14 out to your desired replica count
m
Sure thanks
Also Spicedb v1.13.0 supports postgres 15?
v
I can say that version 1.14 does require version 14 at least. I don't have reasons to believe it wouldn't work with 15, specially since we made it a requirement later on.
But I can't guarantee since we only started testing with 15 in a later spicedb version
m
Okay we will test and update it here for others.