https://authzed.com logo
Join Discord
Powered by
I see some errors claiming that a
# spicedb
b
I see some errors claiming that a definition isn't found when it definitely exists (we would have major problems if 
auth/organisation
didn't exist, for example)
Copy code
rpc error: code = Unknown desc = object definition `auth/organisation` not found
I see these for 
authzed.api.v1.PermissionsService.CheckPermission
and 
dispatch.v1.DispatchService.DispatchCheck
. I assume these errors are making their way back to the client, but I have our clients configured to retry on 
Unknown
There's a tiny number of these compared to the number of incoming requests but it's kinda strange to see
y
what's your datastore and what's its topology?
j
are you using the watching namespace cache?
b
Aurora Postgres, 2 instances with SpiceDB configured to use reader as read replica. Using the default settings for NS cache (i.e. I haven't passed any 
ns-cache-
args)
j
hrmph, ok
will see if we can repro somehow
b
I was wondering if "object definition not found" was even supposed to manifest as 
UNKNOWN
but testing against a local instance querying a nonexistent definition I get a 
FAILED_PRECONDITION
. Have some logs correlated on 
requestID
and it looks like it may be dispatch related
Unfortunately even with retries enabled for 
Unknown
responses sometimes it ends up bubbling up to the client, which would seem to indicate it happens repeatedly for a given check 🤔 With a configuration like this:
Copy code
csharp
new RetryPolicy
{
    MaxAttempts = 5,
    InitialBackoff = TimeSpan.FromMilliseconds(50),
    MaxBackoff = TimeSpan.FromSeconds(5),
    BackoffMultiplier = 4,
    RetryableStatusCodes = { StatusCode.Unavailable, StatusCode.Unknown }
}
Sometimes it does return 
FailedPrecondition
... This was the only log entry for this requestID:
Copy code
json
{
    "message": "finished call",
    "grpc.component": "server",
    "grpc.service": "authzed.api.v1.PermissionsService",
    "grpc.method": "CheckPermission",
    "grpc.method_type": "unary",
    "peer.address": "10.0.2.112:53646",
    "grpc.start_time": "2025-03-28T00:34:38Z",
    "grpc.code": "FailedPrecondition",
    "grpc.time_ms": 2,
    "source": "stderr",
    "requestID": "cviut7hj2f0sco5qqur0",
    "protocol": "grpc",
    "grpc.error": "object definition `auth/user` not found",
    "time": "2025-03-28T00:34:38Z",
    "level": "warn"
}
j
FailedPrecondition will be returned if it is handled by the API layer
dispatch is another issue
it shouldn't be returned by the dispatch layer
that's considered an internal error
we haven't been able to repro, so we'll need more information
b
I have some additional info... it is somehow related to read replicas. After disabling the read replica connection the incidence of the error dropped to 0 If there's more info I can give you, please let me know what it is because I'm not sure what else I can share that I haven't already that would be useful. https://cdn.discordapp.com/attachments/1350995680151208056/1357119899268288553/image.png?ex=67ef0be6&is=67edba66&hm=7fe22fbd497a302be3eccf3003cb0738ec1f1313131ffa27114ba32cd4af8cba&
s
Sorry for necroposting, having the same problem with self-hosted PostgreSQL read replicas.
j
if you can get trace logs, it might help
s
I think I could but I'm unsure how. Does 
requestID
from SpiceDB logs correlate to span IDs in any way?
j
we write them to the spans
s
It would great immensely if you can specifiy a span attribute to filter on. We are using Grafana Tempo and traces are sampled, there is no easy way to get a trace by span id
r
Hello @sp132 . Did you find a way to mitiagate this issue ? We are facing the exact same problem with self hosted PSQL with read replicas.
b
Not apart from "don't enable read-replica". https://github.com/authzed/spicedb/issues/2525
Seems to be a bit of a tricky one to hunt down the root cause of
j
@Ben Simpson try the most recent release
there was an off-by-one issue with the PG revision handling in read replicas that may fix it
b
I think it was perhaps better but not completely fixed - https://github.com/authzed/spicedb/issues/2525#issuecomment-4357306958
(definitely not completely fixed, perhaps a bit better, writing is hard)
j
we'll need more information then
b
I'm willing to spend time assisting as much as I can to help resolve this, I just don't know what additional info I can provide that might help. I have already described everything to the best of my ability including producing thumper scripts that Maria was able to repro the issue with 😔 There are a bunch of people facing the same issue that are on vanilla postgres too, so we can probably rule out Aurora-specific weirdness
j
we need to know what the revision check/guard is returning
b
For anyone running into this issue and stumbling into this thread, this is fixed in 
v1.56.1
🎉 Big thanks @Joey and @Maria Ines Parnisari for your hard work getting to the bottom of this nasty one! 🙏
j
great!
168 Views
PreviousNext
Powered by