https://authzed.com logo
Powered by
Title
p

Perseus

01/10/2023, 3:47 AM
Is there a way to turn off the watch API through CLI flags? I'm seeing high CPU usage by a query that seems to be run through the watch functionality (leading to constant spikes in the postgres CPU) with pretty high avg latency per call
The query that I tracked to the Watch functionality 
SELECT namespace, object_id, relation, userset_namespace, userset_object_id, userset_relation, caveat_name, caveat_context, created_xid, deleted_xid FROM relation_tuple WHERE ((created_xid <= $1 AND created_xid >= $2) OR (deleted_xid <= $3 AND deleted_xid >= $4))
j

Joey

01/10/2023, 3:58 AM
watch API will only cause latency if it is being used
you can turn it off, but if it isn't being used, the cause it likely something else (like GC)
p

Perseus

01/10/2023, 4:09 AM
by "being used" - dyou mean that its enabled?
j

Joey

01/10/2023, 4:09 AM
no
being used as in someone is making that API call
p

Perseus

01/10/2023, 4:09 AM
ah right, yeah. but is it expected to see this latency from watch API calls?
https://discord.com/channels/844600078504951838/844600078948630559/1046395550972526632 I had gone through this thread ^ which seemed to be similar to what I'm seeing
j

Joey

01/10/2023, 4:10 AM
no, but watch isn't yet optimized
so its certainly possible its missing an index or is slower in certain cases
it shouldn't be 600ms slow, mind you
p

Perseus

01/10/2023, 4:11 AM
yeah, i was surprised to see that as well - it was hard to get an accurate EXPLAIN from postgres since I didn't know what values to put for the created_xid and deleted_xids, I used values off a random tuple that I found, but that took quite a long time
j

Joey

01/10/2023, 4:13 AM
@Jake might have some additional insights on the watch side of Postgres tomorrow
p

Perseus

01/10/2023, 4:13 AM
got it. ill disable it on my end for now, till I can figure out the reason for the high latency. wanted to use it in the future
CPU spikes went down immediately after I disabled the watch API
j

Joey

01/10/2023, 5:08 AM
how are you using the API?
p

Perseus

01/10/2023, 8:31 AM
I just simply connected to it - was ignoring the events from it till I had use for them
There was a 1.5min disconnect timer on the grpc stream, at which point I would re-establish the watch connection
v

vroldanbet

01/10/2023, 9:15 AM
Is it possible you are connecting with a very old revision that forces watch to stream like a ton of changes?
p

Perseus

01/10/2023, 10:27 AM
Actually yes. In that case performance would get worse as time goes on, right?
v

vroldanbet

01/10/2023, 10:37 AM
well yeah, you are streaming all the changeset. We can find ways to make that more performant, for sure, if that was a use-case, e.g. finding ways to throttle the operation.
I see the 
zedtoken
argument is optional. If you don't provide any, I think it will "start streaming changes from what it considers HEAD at the moment"
I'd expect that shouldn't cause those spikes you see
p

Perseus

01/10/2023, 10:43 AM
Yeah that's what would be happening. Since it reconnects every 1.5min it would get the list of changes since HEAD everytime that happens, as I'm not passing any zedtokens in. I'll try that out and see if the spikes go down
v

vroldanbet

01/10/2023, 10:47 AM
oh interesting, so if no zedtoken is provided, I'd expect it to do HEAD, but instead is doing beginning of time (hypothesis)
HEAD = latest commit
p

Perseus

01/10/2023, 10:48 AM
Yeah i think beginning of time would make sense to me as a default behaviour, since if i wanna start listening to these from scratch, it would make sense for me to not pass anything in
v

vroldanbet

01/10/2023, 10:48 AM
right, which judging by the spike, I'd assume that's the behaviour
I'd need to check the code to confirm
p

Perseus

01/10/2023, 10:49 AM
Yeah, it's become almost completely flat since i removed it, from frequent 60-99% spikes to a constant 5-7%
v

vroldanbet

01/10/2023, 10:49 AM
I know that @Jake redesigned the PSQL implementation for Watch API and is way more performant, but we haven't cut a release yet
https://github.com/authzed/spicedb/pull/1039
well, interesting turn of events: if no zedtoken is provided, it computes the "optimized revision", which is the revision that is likely to be replicated (something close to HEAD). So no, it's not doing beginning of time. https://github.com/authzed/spicedb/blob/f2c6200443c750f1ce1ec11967762736fdc8d398/internal/services/v1/watch.go#L54-L58 So the theory of "spike because streaming lots of changes" is not what's going on here. Are you able to build yourself from 
main
? That'd allow you to test Jake's new changes. If you'd like to see the "beginning of time" behaviour, please file an issue 🙏🏻
p

Perseus

01/10/2023, 11:08 AM
ahhh okay. yeah I can build from main myself. Will give it a go
#spicedbJoin Discord
Powered by