https://authzed.com logo
#spicedb
Title
# spicedb
j

Jake

10/06/2022, 3:20 PM
how many
relation_tuple_transaction
rows do you have?
w

williamdclt

10/06/2022, 3:21 PM
~150k
j

Jake

10/06/2022, 4:02 PM
Are you using a tool to orchestrate migrations in other solutions?
w

williamdclt

10/06/2022, 6:12 PM
We use Knex or TypeORM. It works just the same as Alembic, doesn't change anything. Most of our migrations are small and without locks, so we just write a migration and merge it. Migrations have to be backward-compatible: migrations are run before new code is deployed (SpiceDB does the opposite I think, recommendation was to run migrations in a post-upgrade hook). If the migration is going to be heavy (taking a lock or being very resource-intensive) but is still a DDL-only changes, our process is: - Write the migration. It has to be idempotent (eg using
IF NOT EXISTS
). - Run the SQL statements manually, out of office hours, to minimise impact. - Merge the migration. It's idempotent, shouldn't actually do anything, it's there for local/test environments. - Note: it's very rare that we do need to take a big lock, Postgres usually has ways to avoid that If the migration requires big DML that are going to be heavy (lock or resource-intensive): - First, do any required DDL change in a separate migration. Has to be backward-compatible. Get that deployed. - Write a job to do the DML changes. To update all rows of a big table for example, we'll probably chunk to avoid keeping locks for a long time - We might want to make an idempotent migration for the DML change. We might also want to clean up the schema with another DDL migration. Depends on the use-case
So overall: we do need a long-running job to migrate data slowly
j

Jake

10/06/2022, 7:00 PM
Spicedb is 100% migration first
And they are backward compatible with 1 version of code
What do you use for your chunk size?
And do you sleep between chunks, have a feedback signal, both, or neither?
w

williamdclt

10/06/2022, 8:22 PM
> What do you use for your chunk size? Depends how write-heavy the table is, usually a few thousands rows, maybe 10k to 50k. > And do you sleep between chunks, have a feedback signal, both, or neither? For us it's a one-off job that's going to be started and monitored by the engineer who wrote it, so it's fairly low-tech. Most often we don't sleep, sometimes we add some sleeping if we realise it's too heavy, and if we want to be fancy we make the sleep duration a parameter of the job
j

Jake

10/07/2022, 7:33 PM