https://authzed.com logo
Title
j

Joey

03/14/2023, 5:00 AM
that means the DB query took 400+ms; the query details should be in the attributes
p

Perseus

03/14/2023, 5:38 AM
this is what i'm seeing in the attributes/events present in the trace - does this mean it took ~300ms to do the
db transaction established
step?
trying to understand what that would mean/ how i can improve that
j

Joey

03/14/2023, 5:42 AM
likely means you had too much traffic on the node
and so it had to wait for a DB connection
p

Perseus

03/14/2023, 5:43 AM
too much traffic on the spicedb pod?
j

Joey

03/14/2023, 5:43 AM
yes
there is a limit to the connection pool
it can be raised, but then you need to make sure your DB supports that many connections * the number of pods
p

Perseus

03/14/2023, 5:44 AM
is that configurable through the CLI args?
j

Joey

03/14/2023, 5:44 AM
yes
p

Perseus

03/14/2023, 5:45 AM
cool, ill try to experiment with those values and see what i can work with
j

Joey

03/14/2023, 5:45 AM
but its generally preferable to deploy more pods vs increasing the connection pool count on existing pods
p

Perseus

03/14/2023, 5:45 AM
hmm. im running 5 pods right now, the average/max CPU are fairly low (sub 10%)
RPS is in the 100-200 range (read heavy,
ReadRelationships
)
j

Joey

03/14/2023, 5:45 AM
okay, then yeah, increase the pod connection pool size
ReadRels isn't a heavy op, but it is a longer running op
since it is streaming back all the rels found
so that means it can be holding connections longer
p

Perseus

03/14/2023, 5:46 AM
ahh. gotcha.
i also ran into some really weird behavior with ALBs on aws, i can create a new thread for that, maybe someone here would've experienced it with gRPC services - not sure if thats okay to post here
j

Joey

03/14/2023, 5:47 AM
sure, start a new thread
I believe the recommendation is to use NLBs because ALBs can have weird effects with gRPC
since gRPC is HTTP2, but with its own quirks
p

Perseus

03/14/2023, 5:48 AM
yeah, i'd picked ALBs because AWS advertised gRPC support in them
j

Joey

03/14/2023, 5:48 AM
oh really?
p

Perseus

03/14/2023, 5:48 AM
and it did work, just not perfectly
j

Joey

03/14/2023, 5:48 AM
ah
j

Joey

03/14/2023, 5:51 AM
good to know