Title
j
Joey
03/14/2023, 5:00 AMthat means the DB query took 400+ms; the query details should be in the attributes
p
Perseus
03/14/2023, 5:38 AMthis is what i'm seeing in the attributes/events present in the trace - does this mean it took ~300ms to do the
db transaction establishedtrying to understand what that would mean/ how i can improve that
j
Joey
03/14/2023, 5:42 AMlikely means you had too much traffic on the node
and so it had to wait for a DB connection
p
Perseus
03/14/2023, 5:43 AMtoo much traffic on the spicedb pod?
j
Joey
03/14/2023, 5:43 AMyes
there is a limit to the connection pool
it can be raised, but then you need to make sure your DB supports that many connections * the number of pods
p
Perseus
03/14/2023, 5:44 AMis that configurable through the CLI args?
j
Joey
03/14/2023, 5:44 AMyes
p
Perseus
03/14/2023, 5:45 AMcool, ill try to experiment with those values and see what i can work with
j
Joey
03/14/2023, 5:45 AMbut its generally preferable to deploy more pods vs increasing the connection pool count on existing pods
p
Perseus
03/14/2023, 5:45 AMhmm. im running 5 pods right now, the average/max CPU are fairly low (sub 10%)
RPS is in the 100-200 range (read heavy,
ReadRelationshipsj
Joey
03/14/2023, 5:45 AMokay, then yeah, increase the pod connection pool size
ReadRels isn't a heavy op, but it is a longer running op
since it is streaming back all the rels found
so that means it can be holding connections longer
p
Perseus
03/14/2023, 5:46 AMahh. gotcha.
i also ran into some really weird behavior with ALBs on aws, i can create a new thread for that, maybe someone here would've experienced it with gRPC services - not sure if thats okay to post here
j
Joey
03/14/2023, 5:47 AMsure, start a new thread
I believe the recommendation is to use NLBs because ALBs can have weird effects with gRPC
since gRPC is HTTP2, but with its own quirks
p
Perseus
03/14/2023, 5:48 AMyeah, i'd picked ALBs because AWS advertised gRPC support in them
j
Joey
03/14/2023, 5:48 AMoh really?
p
Perseus
03/14/2023, 5:48 AMand it did work, just not perfectly
j
Joey
03/14/2023, 5:48 AMah
p
Perseus
03/14/2023, 5:48 AMj
Joey
03/14/2023, 5:51 AMgood to know