We are finally at the point of putting SpiceDB in production in Azure and are running into a sporadic issue we haven't seen locally. It seems that we are able to get quick, successful responses until SpiceDB ends up crashing with the following error:

{
  "level": "warn",
  "error": "database validation failed: context deadline exceeded; if you have previously run TRUNCATE, this database is no longer valid and must be remigrated. See: https://spicedb.dev/d/truncate-unsupported",
  "next-attempt-in": 174257.081448,
  "time": "2025-02-27T02:09:23Z",
  "message": "error attempting to perform garbage collection"
}

It appears that the

context deadline exceeded

is the response from the DB. We are setup using a postgres database. We are not hitting any resource limits on SpiceDB or postgres and usage remains low. The issue seems to be random, but we are able to trigger it slightly faster by querying against slightly more complex schema. Restarting SpiceDB manually fixes the issue and is not actually a migration issue and seems specific to the "context deadline exceeded" message.