Unfortunately none of the above helped in resolvin...
# spicedbs
Sjaak
02/10/2023, 8:23 AMUnfortunately none of the above helped in resolving the problem, I've looked at the data size and we should be somewhere around 200 Gb.
I've identified the main source of storage (1.1 Terabytes) to be WAL (Write Ahead Logs). This may be due to the way our database instance is configured, though we don't have the same issue for other databases in our infrastructure, so likely it's also related to a detail of how SpiceDB operates. I've noticed the same problem, though to a much lesser extent due to a lower volume in data overal, appears on our dev instance - garbage collection has been working correctly on our development environment, so a backlog in data due to failing GC can't explain it.
Yet the WAL size on our dev environment is over 5 times the size of the actual data and grows even when the data doesn't
Sjaak
02/10/2023, 11:44 AMTo give an update on this, we've figured out what the problem was - it was database replication (to fivetran). We've disabled it now and things have gone back to normal
v
vroldanbet
02/10/2023, 12:08 PMYeah SpiceDB does not register itself as a replica to follow the WAL, so that was surprising. Glad you guys found out!
3 Views