hi all, how do you fellows handle schema migrations? do you have any tool for this (like liquibase, flyway etc)?

We don't have any tool for this, internally we keep a Go code that runs schema migrations using the same code SpiceDB uses to run datastore migrations.

aha, thanks! so it seems that there is no atomicity for a bunch of permission changes (say changing bunch of `admin`s to `viewer`s), and i guess you are just retrying call in case of failure. Am i right?

SpiceDB schema migrations have similarities to database migrations, in that the same strategies are applicable (see https://engineering.ezcater.com/migrate-transform-and-backfill-data-with-zero-downtime). These things cannot be done atomically at scale without downtime. In your case, along with schema migration, there is also data migration involved, and in the case of the latter, that's rarely automatic as it is specific to your business domain. Schema migrations are also known to be difficult to scale - see companies like Planetscale having key product offerings like "database branching model" to make schema migrations easier. If you are introducing a breaking schema change in SpiceDB, you need to plan the classical multi-phase migration to do it with no downtime: - add new objects and relations to schema, keep old ones - start writing relationships to new relations along old relations (a.k.a "dual-write") - run backfill job (copy data from old relations to new relations) - update permissions to start using new data or even create new permissions (the latter allows you to switch back your client app in case something is not working as intended) - stop writing to old relations - delete old relationships - remove old relations and objects from schema