For even more context, the sizes of the files aren't ludicrous, but can get to be decent size: * 10k relationships: 161K file size * 6m relationships: 361M file size

False alarm, I had the yaml format wrong. We were able to import all relationships without a problem

@User can you expand on what you had wrong? it sounds like we need a better error message 🙂

@User were you able to import so much data at once? I've had to write a script to import relationships in chunks of ~5000 and it's taking many hours 😬

Unrelatedly: can we expect an update to the docker image soon? PRs #430 and #431 are preventing us to test SpiceDB any further

@User what datastore are you using?

and we're hoping to cut a new image soon - its in final testing now

Postgres - I've managed to cut down the import time significantly by deleting the indexes first (and re-creating them after), but it's still over an hour

that seems problematic if it is taking hours to import

especially if it is due to indexes

The mistake I was making was not formatting the relationship tuples correctly underneath the

relationships

property. What I was initially doing was

yaml
relationships: >-
  document:doc1#owner@user:user1
  document:doc1#reader@user:user2
  document:doc1#reader@user:user3

I should have been adding a blank line between each tuple:

yaml
relationships: >-
  document:doc1#owner@user:user1

  document:doc1#reader@user:user2

  document:doc1#reader@user:user3

Doing it the first way was causing

zed

to read the entire block of text as a single token, which caused

bufio.Scanner

to fail.

aha

yeah, that's non-obvious

I was able to do about 50k relationships in a single

zed import

. It after that, the gRPC request gets too large and

zed

would fail with

Error: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (6989893 vs. 4194304)

To get around this, I ended up using one of the gRPC clients the Authzed folks have and I parsed and batched the relationships myself. I think the

zed

client could easily be modified to batch very large imports by adding some batching logic here: https://github.com/authzed/zed/blob/e9bfd18aee346632fdbd08ecdb6a38f3a41c6098/cmd/zed/import.go#L144-L150

@User would love if you could issue a PR for your modifications for import

Is there any tooling or thoughts around reporting on "cost" of a schema item to resolve (from the schema perspective, not necessarily in actual resolution reporting)? That is, longer graph paths take more to resolve than simple, short ones. For context, we are making load test plans and want to ensure that we are not concentrating on lower cost paths when we make our test scenarios.

zed

will report the cost when you use verbose/debug logs

Is there an example of a Grafana dashboard using the Prometheus metrics exposed by SpiceDB, somewhere? It would be very helpful if we could just import a pre-made dashboard in our Grafana and tweak from there!

there is not, we have one that we use internally but it focuses a lot on tenancy and authzed.com

it would make a great contribution for the repo though!

Hey everybody! My name is Alex, and my team is currently evaluating SpiceDB/Authzed to use it in our product. We like it very much so far, however there are some questions we need to clarify, and I would appretiate very much someone can point me to the right direction. Here's a (a part of the) scheme we have developed:

definition org {
    relation admin: user
    relation member: user
    relation partner: partner

    permission view = admin + member + partner
}

definition user {
    relation self: user
    relation org: org

    permission view = self + org->admin + org->partner
}

definition partner {}

definition transaction {
    relation org: org
    relation user: user

    permission view = user + org->admin + org->partner
    permission edit = user + org->admin
}

We are trying to answer the question "who can view transactions". The idea that the transaction can be viewed by the user who made it, but the admin of the org to which this transaction belongs, and by the partner of the org. As long as we are talking about referencing transaction by id, everything is clear, for the set of relarionships

org:foo#partner@partner:bar
transaction:baz#org@org:foo

assertion

transaction:baz#view@partner:bar

evaluates to

true

. What is not clear is, how it should work for the use case when we get multiple transactions based on some filters and pagination. I've read about the Lookup API and found this article https://authzed.com/blog/acl-filtering-in-authzed/, but it's not clear how the workflow should actually look like. Let's extend the example above with one more org-to-partner relationship, and one more transaction for this org

org:qux#partner@partner:bar
transaction:quuz#org@org:qux

Now let's say that

partner:bar

requests to view all the transactions of

org:qux

. If I use the Lookup API to get the list of all transaction that this partner has access to, I wlil receive 2 transactions id,

transaction:baz

and

transaction:quuz

.

Now (I assume) I need to query my database with the filter from request (

org=qux

) and then take an intersection of two sets, one from the database and one from the Lookup API result. If we have millions of transactions in every set, that doesn't sound reasonable. I feel like I'm missing something obvious in this use case. Is there a better way to solve the problem I've explained? Thanks in advance.

hi @User, you're correct in how it would work and where it would not

We have a proposal in progress called LookupWatch (and Tiger cache) to make these large kinds of lookups doable: https://github.com/authzed/spicedb/issues/207

you'd either (a) subscribe to LookupWatch and update a join table in your database for use in filtering or (b) call the Tiger cache to get back a roaring bitmap of all the objects the user can view, and use that as a filter in your database or ElasticSearch or other querying system

another option, if you know the number of results coming out of your DB is fairly constrained, is to simply

Check

each result in parallel

that's called post-filtering

@User I'm understanding you have 3 problems (might totally be misunderstanding): 1. You're not super-sure how to access-control a list of resources Joey pretty much answered this one. At a high level, you can either pre-filter by getting the list of accessible resources (Lookup) and use it in a filter in your DB query, or post-filter by optimistically querying the DB and then removing the resources the user doesn't have access to. 2. You're concerned about the amount of resources to intersect I don't think there's anything you're missing here, really. Either you accept that you have to manipulate large sets, or you find a way to reduce the space of resources to lookup, which leads me to... 3. You'd like to lookup resources accessible by a Partner in a specific Org I had the same problem just yesterday! In my case I had users being part of multiple orgs, and I wanted to lookup resources in a single Org. There is a solution indeed What you are missing is a new definition: something like

partner_of_org

. It's basically an object representing the relationship between a single partner and a single org, its ID could be a concatenation of the partner ID and the org ID.

Your schema can then look like:

definition partner {}
definition partner_of_org {
  partner: partner
}

definition org {
    relation admin: user
    relation member: user
    relation partner: partner_of_org#partner

    permission view = admin + member + partner
}

definition user {
    relation self: user
    relation org: org

    permission view = self + org->admin + org->partner
}

definition transaction {
    relation org: org
    relation user: user

    permission view = user + org->admin + org->partner
    permission edit = user + org->admin
}

Now, if you want to query all transactions viewable by a partner, you can lookup

transaction#viewable

for

partner:partner_id

. If you want to query transactions viewable by a partner in a given org, you can lookup

transaction#viewable

for

partner_of_org:org_id__partner_id

. That means you'll need your relations to be like

org:qux#partner@partner_of_org:qux__bar#user
partner_of_org:qux__bar#user@partner:bar
transaction:quuz#org@org:qux

@User thanks for sharing that! I assume it’s probably inappropriate to ask if there’s a release date for this feature 🙂 but I will definitely take a closer look and maybe my team can contribute at least a bit 😉