We have a relatively large database (c. 25M relation tuples) running on a relatively small RDS instance (2VCore/16GB) with 6 SpiceDB pods and noticed that the database instance is running at about 100% CPU most of the time, even when there's no traffic going to SpiceDB, and there are lot of these errors in the log:

{"level":"error","module":"pgx","pgx":{"args":[{"Uint64":1875802,"Valid":true}],"err":"timeout: context deadline exceeded","pid":30194,"sql":"WITH rows AS (SELECT namespace, object_id, relation, userset_namespace, userset_object_id, userset_relation, created_xid, deleted_xid FROM relation_tuple WHERE deleted_xid < $1 LIMIT 1000)\n\t\t  DELETE FROM relation_tuple\n\t\t  WHERE (namespace, object_id, relation, userset_namespace, userset_object_id, userset_relation, created_xid, deleted_xid) IN (SELECT namespace, object_id, relation, userset_namespace, userset_object_id, userset_relation, created_xid, deleted_xid FROM rows);\n\t","time":59896.646647},"time":"2023-08-15T20:49:35Z","message":"Query"}
{"level":"warn","error":"error deleting in gc: timeout: context deadline exceeded","next-attempt-in":-0.000001,"time":"2023-08-15T20:49:35Z","message":"error attempting to perform garbage collection"}

Our understanding is this is likely related to cleaning up old snapshots, but any insights would help.

https://github.com/authzed/spicedb/pull/1445 for more details