We recently started looking at the watch API to maintain some secondary indices but have run into some issues. We have a worker application that watches a single objectType using the watch service, updates a secondary index, and keeps track of its current position using the ZedToken.
We have some quite bursty activity and noticed when 15-20 relations are created in quick succession our watch API gets immediately disconnected with this error:
"rpc error: code = ResourceExhausted desc = watch disconnected: watch fell too far behind and was disconnected"
I’m assuming its because our worker is pretty slow at churning through updates, but equally I was hoping it would be able to fall a bit further behind and recover without causing much issue.. I took a quick look through the postgres datastore implementation and can see where the err is raised but couldn’t really find any obvious cause. Is there anything you guys would recommend I look at that might help narrow down the issue or is it something you’ve commonly seen before?