https://authzed.com logo
Title
w

williamdclt

12/06/2022, 12:47 PM
In addition, I found a performance issue of the GC, details in thread!
Our monthly AWS bill for Authz in staging has grown hugely since we upgraded it to 1.14.1. Screenshot 1: I quickly identified that it comes from a much-higher read IOPS usage (we use RDS Aurora, so we pay for the IOPS): about 2000x increase! Screenshot 2: zooming in to the read IOPS, it spikes every 3 minutes. That's enough to make me suspect the GC (it's configured to run every 3 min) Screenshot 3: AWS performance insights confirms that this query is very heavy on the DB, both in CPU and in DataFileRead (ie IOPS). I'll make the GC run much less often as a workaround for now, but probable there's something to be done? Missing index probably?
j

Joey

12/06/2022, 3:35 PM
GC might be missing one the same fields
created_xid
, from the index
Victor is working on a PR for adding a covering index for the check queries, but it requires more testing, as the context extends the size of the index greatly
@williamdclt try adding
created_xid
to the
includes
of the existing relation tuple index
that seemed to change most of the (non-caveated) selects back into an index-only scan
w

williamdclt

12/06/2022, 4:25 PM
I don't have much time to tinker around this week, esp given this is only staging (prod is still on 1.13.0 due to the performance problems)
j

Joey

12/06/2022, 4:26 PM
okay, then we'll likely have our PR ready next week