Relationship write latency
# spicedbt
Tomek
06/21/2023, 1:53 PMHey everyone!
I noticed really weird behaviour in the way SpiceDB work for one of my relationship write, it does the write with a notable delay, around 5min. So far I noticed that only for one particular relationship, other relationships get written right away.
This makes me wonder if there's any write scheduling that could cause the delay 🤔 Eventually the relationship gets written.
(NodeJS version 16)
j
Jake
06/21/2023, 1:54 PMCan you reproduce it with this same relationship consistently?
t
Tomek
06/21/2023, 1:58 PMEach time I assign a new ID of the defined resource, in some cases I get the relationship written fast as expected. From what I've been seeing if I do a second write right after, then there's no delay
Tomek
06/21/2023, 1:59 PMI'll try to reproduce it with
zedTomek
06/21/2023, 2:20 PM@Jake this doesn't seem to be reproducible for the same relationship, the second attempt I did to write it was done right away. Also I couldn't reproduce it with
zedj
Jake
06/21/2023, 2:21 PMwhat does your environment look like?
Jake
06/21/2023, 2:22 PMin general writes aren't very variable
Jake
06/21/2023, 2:22 PMalthough I would recommend sending more than one write per
WriteRelationshipsRequestt
Tomek
06/21/2023, 2:25 PMWe have a K8s cluster, there are 2 replicas of NodeJS service responsible for permission writes, these are dispatched by messaging bus, regarding SpiceDB -- there are 2 replicas and we use external db instance of pg for datasource
Tomek
06/21/2023, 2:26 PMMessaging bus -> NodeJS service -> Write permission in SpiceDB
j
Jake
06/21/2023, 2:43 PMsounds really standard, which version of spicedb are you using, and are you using the spicedb-operator?
t
Tomek
06/21/2023, 3:14 PM1.16.2
Tomek
06/21/2023, 3:14 PMno operator
j
Jake
06/21/2023, 3:15 PMif you find a way to repro it please let us know, but otherwise my guess would be that your nodejs servers are overloaded/blocking
Jake
06/21/2023, 3:16 PMhow many QPS of write are you approximately sending?
t
Tomek
06/21/2023, 3:17 PMqueries to nodejs servers or SpiceDB?
Tomek
06/21/2023, 3:18 PMI actually excluded the option of nodejs servers being overloaded, we have logs before and after each SpiceDB write, there is a milliseconds of difference before these 2 logs :/
5 Views