https://authzed.com logo
Join Discord
Powered by
Google-Scale Authorization: Getting to 1...
# spicedb
m
And can you also confirm that you are measuring the grpc_server_handled_total metric in these tests?
j
the right most, yes
but note that the size of the nodes matters quite a bit too
m
what is the cpu and memory req and limit that you defined for each pod
can you please share that
for my pods, I am doing request cpu: 500m and limit: 4. Does it matter in go runtime if I set the cpu request to 4 from the outset?
j
I don't recall offhand but it more or less should be the full resources of the node
m
do you are dedicate the full VM node to a single pod?
j
yes
m
Aha
j
for SpiceDB ones, yes
for the control plane, no
m
and did you have to play around with the GOMAXPROCS setting ?
I would believe no, since a VM node == k8 pod in your case
j
I don't believe so
m
ok this gong to be pretty tricky now
j
yeah, you don't want SpiceDB preempted
this is why we offer our Dedicated product - we do all this for you 😄
m
$$$ = peace of mind 🙂
let's see how this goes
I did a simple test with 34 million relations randomly selected and at about 3k RPS the p95 starts to become choppy and starts going up. My test was increasing the number of requests every 10 mins. So I am trying to figure out how to work through optimizing this now https://cdn.discordapp.com/attachments/1179622178824667156/1179629593989230662/Screen_Shot_2023-11-29_at_9.45.00_PM.png?ex=657a7aab&is=656805ab&hm=863769aede9b02ff93f905ccbd89f602cae1e634d523909491b900162eeeb971&
our current RBAC system has a P95 of 3ms so this might become a problem
I think it might be related to crdb
as it's latency jumps up after a certain RPS
j
you likely need to scale it too
m
PreviousNext
Powered by