Hi We are getting these errors in production. I have an idea it is probably the database. But I would like to know, before we point at our database service provider. {"level":"error","requestID":"6a79f268968518926ab583aff1c69a98","protocol":"grpc","grpc.component":"server","grpc.service":"dispatch.v1.DispatchService","grpc.method":"DispatchCheck","grpc.method_type":"unary","peer.address":"10.42.13.82:38444","grpc.start_time":"2024-11-18T15:23:47Z","grpc.request.deadline":"2024-11-18T15:24:47Z","grpc.code":"Unknown","grpc.error":"error performing check: unable to query tuples: rows err: unexpected EOF","grpc.time_ms":"1.495","time":"2024-11-18T15:23:47Z","message":"finished call"} {"level":"error","requestID":"6a79f268968518926ab583aff1c69a98","protocol":"grpc","grpc.component":"server","grpc.service":"dispatch.v1.DispatchService","grpc.method":"DispatchCheck","grpc.method_type":"unary","peer.address":"10.42.11.194:58562","grpc.start_time":"2024-11-18T15:23:47Z","grpc.request.deadline":"2024-11-18T15:24:47Z","grpc.code":"Unknown","grpc.error":"rpc error: code = Unknown desc = error performing check: unable to query tuples: rows err: unexpected EOF","grpc.time_ms":"11.957","time":"2024-11-18T15:23:47Z","message":"finished call"} {"level":"error","requestID":"6a79f268968518926ab583aff1c69a98","error":"error performing check: unable to query tuples: rows err: unexpected EOF","time":"2024-11-18T15:23:47Z","message":"unexpected dispatch graph error"} {"level":"error","requestID":"6a79f268968518926ab583aff1c69a98","protocol":"grpc","grpc.component":"server","grpc.service":"authzed.api.v1.PermissionsService","grpc.method":"CheckPermission","grpc.method_type":"unary","peer.address":"10.42.9.161:33870","grpc.start_time":"2024-11-18T15:23:47Z","grpc.code":"Unknown","grpc.error":"rpc error: code = Unknown desc = error performing check: unable to query tuples: rows err: unexpected EOF","grpc.time_ms":"12.904","time":"2024-11-18T15:23:47Z","message":"finished call"}

assuming you are doing Postgres, the error happens here: https://github.com/authzed/spicedb/blob/5731e26b3b497edb3c6bfa2bae6c7fc14dd8242f/internal/datastore/postgres/common/pgx.go#L107-L109 PGX says the following > // Err returns any error that occurred while reading. Err must only be called after the Rows is closed (either by > // calling Close or by Next returning false). If it is called early it may return nil even if there was an error > // executing the query. So essentially SpiceDB was reading from the database, and something happened.

It is Postgres

its likely losing the connection

What is the default timeout, and can we change it?

there is no timeout to the database

But Joey wrote it could be timing out

yeah I don't think that's correct, but maybe he can clarify what he means. The datastores have a middleware that sever the context, and so deadlines are not propagated. This is to protect connection pools from draining on high error rates. This means requests will just continue to execute until something makes it return an error.