Hello Guys. We are seeing some memory problems in our spiceDb related to OTEL. (image above) Has this problem been reported yet? I have seen that the latest versions of spiceDb is changing some of the OTEL libs & Middleware. We at v1.45.3. https://cdn.discordapp.com/attachments/844600078948630559/1421065745130979339/stitched_log.png?ex=68d7ae1b&is=68d65c9b&hm=b385a3535ac534c1578d0995b919607c4ccd1b3e494bd3bf5c8a02059ffbba8d&

yours is the first report. we did update the otel libs... I hope that didn't come with a regression. what's the behavior you're observing? the above is showing a nil pointer dereference during protobuf marshalling, which could be related to memory pressure but isn't necessarily directly related to otel - the lower parts of the trace are just otel span wrapping.

We already investigated the memory usage and how much Golang reserves for itself. The pod has 4 GB, and Golang seems to take around 3 GB. I’ll bump Spice to 1.45.4 and see if that helps 💪. The behavior is tricky—it just dies every X hours. I’ve seen something similar before in my Java applications, where OTEL (without a metrics endpoint—we don’t push metrics with OTEL) would eventually go OOM. If nothing else comes to mind, I’ll keep digging. Thanks!

i just saw this on the spicedb repo: https://github.com/authzed/spicedb/issues/2579

@yetitwo sorry for the delayed response. we are now at the V1.45.4 and the error is still there. I will check if the error trace changed since and post it.

sounds good. thank you!