https://authzed.com logo
Join Discord
Powered by
Is there a way to turn off the watch API through C...
# spicedb
p
Is there a way to turn off the watch API through CLI flags? I'm seeing high CPU usage by a query that seems to be run through the watch functionality (leading to constant spikes in the postgres CPU) with pretty high avg latency per call
The query that I tracked to the Watch functionality
Copy code
SELECT namespace, object_id, relation, userset_namespace, userset_object_id, userset_relation, caveat_name, caveat_context, created_xid, deleted_xid FROM relation_tuple WHERE ((created_xid <= $1 AND created_xid >= $2) OR (deleted_xid <= $3 AND deleted_xid >= $4))
j
watch API will only cause latency if it is being used
you can turn it off, but if it isn't being used, the cause it likely something else (like GC)
p
by "being used" - dyou mean that its enabled?
j
no
being used as in someone is making that API call
p
ah right, yeah. but is it expected to see this latency from watch API calls?
https://discord.com/channels/844600078504951838/844600078948630559/1046395550972526632 I had gone through this thread ^ which seemed to be similar to what I'm seeing
j
no, but watch isn't yet optimized
so its certainly possible its missing an index or is slower in certain cases
it shouldn't be 600ms slow, mind you
p
yeah, i was surprised to see that as well - it was hard to get an accurate EXPLAIN from postgres since I didn't know what values to put for the created_xid and deleted_xids, I used values off a random tuple that I found, but that took quite a long time
j
@Jake might have some additional insights on the watch side of Postgres tomorrow
p
got it. ill disable it on my end for now, till I can figure out the reason for the high latency. wanted to use it in the future
CPU spikes went down immediately after I disabled the watch API
j
how are you using the API?
p
I just simply connected to it - was ignoring the events from it till I had use for them
There was a 1.5min disconnect timer on the grpc stream, at which point I would re-establish the watch connection
v
Is it possible you are connecting with a very old revision that forces watch to stream like a ton of changes?
p
Actually yes. In that case performance would get worse as time goes on, right?
v
well yeah, you are streaming all the changeset. We can find ways to make that more performant, for sure, if that was a use-case, e.g. finding ways to throttle the operation.
I see the 
zedtoken
argument is optional. If you don't provide any, I think it will "start streaming changes from what it considers HEAD at the moment"
I'd expect that shouldn't cause those spikes you see
p
Yeah that's what would be happening. Since it reconnects every 1.5min it would get the list of changes since HEAD everytime that happens, as I'm not passing any zedtokens in. I'll try that out and see if the spikes go down
v
oh interesting, so if no zedtoken is provided, I'd expect it to do HEAD, but instead is doing beginning of time (hypothesis)
HEAD = latest commit
p
Yeah i think beginning of time would make sense to me as a default behaviour, since if i wanna start listening to these from scratch, it would make sense for me to not pass anything in
v
right, which judging by the spike, I'd assume that's the behaviour
I'd need to check the code to confirm
p
Yeah, it's become almost completely flat since i removed it, from frequent 60-99% spikes to a constant 5-7%
v
I know that @Jake redesigned the PSQL implementation for Watch API and is way more performant, but we haven't cut a release yet
well, interesting turn of events: if no zedtoken is provided, it computes the "optimized revision", which is the revision that is likely to be replicated (something close to HEAD). So no, it's not doing beginning of time. https://github.com/authzed/spicedb/blob/f2c6200443c750f1ce1ec11967762736fdc8d398/internal/services/v1/watch.go#L54-L58 So the theory of "spike because streaming lots of changes" is not what's going on here. Are you able to build yourself from 
main
? That'd allow you to test Jake's new changes. If you'd like to see the "beginning of time" behaviour, please file an issue 🙏🏻
p
ahhh okay. yeah I can build from main myself. Will give it a go
19 Views
PreviousNext
Powered by