faisalmushtaq_
07/31/2025, 7:30 PMJoey
07/31/2025, 7:30 PMJoey
07/31/2025, 7:30 PMJoey
07/31/2025, 7:30 PMJoey
07/31/2025, 7:31 PMfaisalmushtaq_
07/31/2025, 7:32 PMfaisalmushtaq_
07/31/2025, 7:32 PMJoey
07/31/2025, 7:41 PMsmithp4ntz
08/01/2025, 10:15 PMyetitwo
08/02/2025, 12:44 AMyetitwo
08/02/2025, 12:45 AMsmithp4ntz
08/02/2025, 1:12 AMsmithp4ntz
08/02/2025, 1:14 AMyetitwo
08/02/2025, 1:36 AMkartikay
08/04/2025, 9:10 PMBen Simpson
08/05/2025, 1:03 AMdefinition user {}
definition organisation {
relation admin: user
relation teacher: user
relation deactivated_member: user
permission active_staff = admin + teacher - deactivated_member
}
// ID is Org:Subject pair
definition capability {
relation organisation: organisation // only ever one organisation here
relation capability_official_content: user:* | organisation:*
permission capability_applies = organisation->active_staff + organisation
permission official_content = (capability_official_content & capability_applies)
}
definition subject {
relation capability: capability
permission official_content = capability->official_content
}
definition activity {
relation subject: subject
relation official_content: user:* | organisation:*
permission view = subject->official_content & official_content
}
Checking activity:x#view
seems to blow up our database due to needing to evaluate thousands of capability
per subject
. An --explain
shows that it's evaluating a lot of tuples (the output was ~600kb of text)
Logic is:
Organisation is granted access to various subjects. Orgs may be granted different capabilities for each subject. A user may be in multiple organisations. Users should only be granted access to activities their current org (contextually) has access to. We're trying to avoid caveats so this involves 2 checks, one for the user and one for the org (hence the additional org wildcards):
activity:a_1#view@user:u_1
activity:a_1#view@organisation:o_1
It doesn't seem like this should be a problem for SpiceDB to handle - do we just need a beefier DB? I don't know if it's related to this issue in particular but our cache hit ratio has also tanked recently, it usually sits around 2:1 but is between 1.2-1.5:1 recently 😱yetitwo
08/05/2025, 2:35 AMdefinition activity {
relation subject: subject
relation official_content: activity
permission has_official_content = subject->official_content
permission view = official_content->has_official_content
}
with the idea being that you'd write an official_content
relation from an activity to itself to mark that boolean.
this isn't necessarily the optimization that's going to fix things, but using arrows instead of intersections means that SpiceDB doesn't have to evaluate as many tuples before it can decide that it's got a match, and you can propagate that same pattern up the treemsanchezdev
08/06/2025, 1:20 AMJoey
08/06/2025, 7:20 AMmsanchezdev
08/06/2025, 7:21 AMJoey
08/06/2025, 11:44 AMmsanchezdev
08/06/2025, 11:46 AMJoey
08/06/2025, 11:46 AMthanos_alas
08/07/2025, 9:19 AMJoey
08/07/2025, 12:52 PMspicedb datastore repair
thanos_alas
08/07/2025, 12:56 PMJoey
08/07/2025, 1:21 PMyetitwo
08/07/2025, 2:48 PMdiveangle
08/08/2025, 8:13 AMrelation_tuple
, not counting any deleted ones, but at the moment only 680k of those that were written in the last day (so others will expire eventually), so maybe something in that ballpark would be how many relationships this process has to write once a day, and it takes around 20 minutes generally. Of course we also usually have concurrent usage of SpiceDB by our application for mostly CheckBulkPermissions
requests whenever these syncs happen
Now with that context, we are seeing two types of errors in the spicedb pod logs, that I can confirm appear during these syncs, and I can reliably reproduce them by re-running the sync process. What is interesting is, the WriteRelationships
calls on the sync process side seem to all succeed, which is why we didn't notice this error for some time (we didn't have alerts for the SpiceDB pod's errors so we didn't notice them). From our side it looked like the sync was succeeding, but we were not seeing some relationships that we thought we should be seeing (we were relying on the sync to create them in that case). Will split the rest of the message with the actual errors into a second message.