Hi all I have set up a spicedb-postgres configuration using docker-compose, and I have create a basic schema for users, groups, and applications.

definition user {}

definition group {
    relation owner: user
    relation admin: user | group#membership
    relation member: user | group#membership

    permission manage_groups = owner + admin
    permission membership = manage_groups + member
}
    
definition application {
    relation admin: user | group#membership
    relation users: user | group#membership

    permission permission_01 = admin
    permission permission_02 = admin + users
    ...
}

I conducted performance testing with 1000K relations, containing 50k groups and 1000 applications. As observed in the graph, for most of the requests, I am experiencing response times in double digits, which is not ideal for an authorization service. So wanted to know ways to fine-tune this for better performance. https://cdn.discordapp.com/attachments/844600078948630559/1179363425147691039/image.png?ex=657982c8&is=65670dc8&hm=dd5924328b6c71b59cb790dfceeaf50824ce537ad2bfdcf676a1180e63f1f934&

You are running a load test against a locally running, single node SpiceDB with Postgres running on your machine? Can you specify what does your SpiceDB configuration look like? Can you provide details of the load test? How have you configured your connection pools?

Yes, it is a single node SpiceDB with Postgres running locally on my machine (Mac/M1 Pro/32Gb). Created using this docker-compose found on the internet, have made some minor changes:

version: '3'

services:
  spicedb-datastore:
    image: postgres
    container_name: spicedb-datastore
    ports:
      - "5432:5432"
    environment:
      POSTGRES_DB: spicedb
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: password
    networks:
      - spicedb-net

  spicedb:
    image: authzed/spicedb:latest
    container_name: spicedb
    restart: "on-failure"
    command: ["serve", "--grpc-preshared-key", "mykey", "--datastore-engine=postgres", "--datastore-conn-uri=postgres://postgres:password@spicedb-datastore:5432/spicedb?sslmode=disable"]
    ports:
      - "8085:8080"
      - "9095:9090"
      - "50051:50051"
    depends_on:
      - spicedb-datastore
      - "migrate"
    networks:
      - spicedb-net

  migrate:
    image: authzed/spicedb:latest
    container_name: spicedb-migrate
    restart: "on-failure"
    command: ["migrate", "head", "--datastore-engine=postgres", "--datastore-conn-uri=postgres://postgres:password@spicedb-datastore:5432/spicedb?sslmode=disable"]
    depends_on:
      - spicedb-datastore
    networks:
      - spicedb-net

networks:
  spicedb-net:
    driver: bridge

I have generated around 1000k relations which contains 1000k users, 20% of users are directly connected to the applications, and 80% are present in 50k groups (distributed randomly) and those groups are again randomly connected to 1000 applications. For getting the performance result, I am sending 100k permission checks in loop, equally distributed in 100 threads using go routines and authzed go-client

@naveen_kr - You should make a workload that represents your actual application. Are you iterating over all the 1000K users? Because that does not represent the typical access pattern you see in production systems. You don't have "all users in the system acting at all times", or do you? That's that's basically the worst-case scenario possible. There would be some caching of intermediate subproblems, but most of the data would have to be freshly loaded from the datastore. You should validate this by looking into SpiceDB metrics and checking the caching ratio. - SpiceDB exposes a Prometheus metrics endpoint, so use whatever tool you typically use for this (Grafana, Datadog...). There is no UI embedded into SpiceDB. - Please refer to

spicedb serve help

to see all the various configuration flags. Metrics are exposed by default on

:9090

- You may need to tune SpiceDB according to the workload, and you are basically just using the defaults. My recommendation is to look into spicedb metrics to gather insight. - There is no native support for multi-tenancy. Our recommendation is to use the SpiceDB Operator in Kubernetes environments that helps spinning up new clusters easily. You can create a new cluster per tenant.

@vroldanbet - "Are you iterating over all the

1000K

users?" --> No, I am making

100K

calls with random users to random applications (and because it is random, some requests may even repeat). But sure, I will try connecting and look into the SpiceDB metrics for insights, and will come back if further help is needed. Also, I will read about the SpiceDB Operator in Kubernetes environments for multi-tenancy. Thank you!

yeah this is a random distribution, which im not sure it's your case but it's typically an unrealistic one in web applications. If you read https://authzed.com/blog/google-scale-authorization you'll see we used a pareto distribution because of this reason.

Sure, will read this blog