How does watch API scale w/ PG? If there are 10000 RPCs to the watch endpoint simultaneously streaming events, are there 10000 queries running periodically?
e
ecordell
08/07/2024, 3:42 PM
Right now each watch opens its own connection on the backend, so you wouldn't want to open that many.
The recommendation is to open a watch for all possible consumers and then distribute to individual clients on the application side.
But we've definitely floated ideas to make this simpler / easier, especially with Spanner where there are a limited number of watch slots. Might be worth writing up a GH issue to discuss some future options to simplify this.
y
yetitwo
08/07/2024, 4:39 PM
there's also been discussion of providing easier mechanisms to move events from the watch API to something like kafka, such as through a connector
a
alechenninger
08/07/2024, 5:02 PM
Thank you both!
Is there a GH issue around a "connector"? We may be interested in contributing to something like that
y
yetitwo
08/07/2024, 5:12 PM
not yet - my understanding is that there's been a couple of starts on this idea (one using Bento and one that i started at my previous company but haven't pushed up yet) but nothing written down yet