https://authzed.com logo
Join Discord
Powered by
unable to verify DB
# spicedb
k
hi all. i'm new here, just beginning my spiceDB journey. i'm working on adding it as our platform authorization system. i've gotten everything working great locally, but i'm having some trouble when deploying to GCP. some details: - serving 
authzed/spicedb:v1.34.0
in GCP Cloud Run - datastore is a postgres database in GCP AlloyDB - changes to the docker image version prompt a Cloud Run Job to run any new migrations against the datastore my problem is this: when running 
serve
on the container in Cloud Run, i get the following error:
Copy code
error: "database validation failed: closed pool; if you have previously run `TRUNCATE`, this database is no longer valid and must be remigrated. See: https://spicedb.dev/d/truncate-unsupported"
level: "warn"
message: "could not check if the datastore was ready
the datastore is definitely migrated. the tables exist and there is seed data in some of them (relation_tuple_transaction, alembic_version, metadata). there was no manual 
TRUNCATE
run. can anyone point me in the right direction to resolve this? i've tried running with 
trace
& 
debug
log levels, but this is still the best error message i can get. i'm pretty sure the postgres connection is happening successfully because i see logs in the database noting the connection establishment. any help/guidance would be greatly appreciated. happy to provide more detail if needed. thanks!
j
Are you pointing to the right named DB in the instance?
> changes to the docker image version prompt a Cloud Run Job to run any new migrations against the datastore You can't simply run migrations async and ad-hoc and expect everything to work. The 
spicedb-operator
is purpose built to intelligently walk database upgrades and corresponding software versions.
since you're seeing the connections on the postgres cluster side, my guess is either 1) a malformed connection string that's pointing to the wrong named DB on the instance, or 2) the software you're trying to run isn't compatible with the version that the DB has been migrated to
k
thanks for the quick reply. yes, it is pointing to the correct named db. i'm using the 
SPICEDB_DATASTORE_CONN_URI
environment variables for both the migration job and the serving container, both from the same secret.
i understand this is a bit of an unorthodox approach. my company is small and would prefer not to spend time maintaining a k8s cluster if we don't have to. the migration job and the service are both on the latest image version, v1.34.0
i get that running the operator is a better option, but i thought that running the migrate job only when bumping the image would be a working compromise
j
the 
closed pool;
indicates a connection issue most likely
k
agreed. i think it must be coming from this line: https://github.com/authzed/spicedb/blob/main/internal/datastore/postgres/stats.go#L31, which appears to 
SELECT unique_id FROM metadata
. that works when i run it right on the db but for some reason is not returning properly in GCP.
i see it in the debug logs, and looks like it's executing (there's a runtime on there)
Copy code
pgx: {
args: [0]
commandTag: "SELECT 1"
pid: 1536370
sql: "SELECT unique_id FROM metadata"
time: 3.498286
}}
j
unfortunately i don't know anything about how cloud run works, is there a constrained networking environment from cloud run to database in some way? The error is "pool closed" which is coming from pgx, and should only happen when spicedb starts to shut down. Are you maybe running it as a short-lived container and it's getting a shutdown signal while it's actually still starting up?
k
i was able to resolve this. there was an issue with a sidecar proxy container i was using to connect to the datastore. for some reason, the initial connection went through, but actual queries were taking too long to return and spicedb decided the connection needed to close. that's my best theory, anyway, without knowing the internals. removing the sidecar proxy and connecting directly to the database appears to have resolved it. thanks both for your help!
47 Views
PreviousNext
Powered by