Hi Folks, I am upgrading spicedb from version 1.13.0 to 1.14.0 manually not using operator. We are using postgres15 as datastore for spicedb. While upgrade getting this error `{"level":"error","error":"unable to migrate to

head

revision: error executing migration `add-xid-constraints`: ERROR: index \"ix_relation_tuple_living\" is not valid (SQLSTATE 55000)","time":"2024-06-12T11:46:14Z","message":"terminated with errors"}` Migration till

add_xi8_columns

is successfully completed but failed on

backfill-xid-add-indices

. Can someone help in figuring out the root issue for this. Thanks

have you checked if

ix_relation_tuple_living

index exists? It's created concurrently, so I could see it taking long to complete, and that could cause the migration to fail.

> have you checked if ix_relation_tuple_living index exists? Yes On our cloudsql instance pg_cron is enable for partitioning, is it possible that migration is failing because of cron database flags.

We don't run postgres in GCP, so I honestly don't know if that matters or not

Hi @vroldanbet We removed the database flags on cloudsql instance and did the upgrade once again. It is successful after that. But after the migration we are facing issue with the CheckPermission API, when we are creating any new resource and running CheckPermission on the same resource immediately, it is giving error. After around 15sec same CheckPermission is giving correct response. Check resources creation and the record corresponding the same resource is created immediately on spicedb. It is not taking time while creation but checkPermission API is able to verify resource permission after sometime. Any idea about this issue?

this is expected behaviour if your are issuing check permission calls with

minimize_latency

consistency, which is the default. If you need read-your-writes consistency, you need to use

at_least_as_fresh

with a zedtoken, of

fully_consistent

. Please note abusing

fully_consistent

will tank your SpiceDB performance. Please read more about SpiceDB Consistency guarantees in https://authzed.com/docs/spicedb/concepts/consistency

Thanks @vroldanbet

We've fixed a few issues, including one that affected LR. I recommend upgrading to the latest version, if once there if keeps reproducing, we can investigate.

We are in the process for upgrading this but we can't take it one go, is there any way to make it more stable. Because as of now even when we added checkPermission API fully consistent it is returning true/false for same user on the same resource. If we are making any changes in the permission of the user. For ex- I have a team

spicedb-admin

user

xyz

is part of spicedb-admin team, this team have access to

abc

resource. When I remove

xyz

user from team. CheckPermission for

xyz

sometime returning false sometime it is returning true. When we are check relation_tuple with record it is removed immediately when we are removing this user from the team. We upgraded v1.13 to v1.14 after that we are facing this issue, till the point we upgrade it to latest version can we some changes in the configuration to make this more stable?

You are running an out of maintenance version that could be subject to bugs that have been fixed ever since, so the recommendation is to upgrade. Once you've upgrade, if you observe the bug persists, then the recommendation is to create a test case the reproduces the issue so we can investigate.

Thanks

I suggest running in debug mode. Nothing stands out as unhealthy.

In debug logs also not getting any hint what is missing on this, by anychance you came across this type of case? https://cdn.discordapp.com/attachments/1250420897169543199/1255871932218212373/Screenshot_2024-06-27_at_6.34.36_PM.png?ex=667eb561&is=667d63e1&hm=d419e0a7960e85ad1beb5ec49a9793c764af95cff3a3b96902db6408b057b9fd&

The instance is closed ming up healthy and being terminated by kube. I'd say the problem is your health probes.

Okay