spicedb
- w
williamdclt
03/28/2022, 1:36 PMI'm not succeeding to visualise the dump, I might be doing something wrong. I'm doing: -
-curl localhost:9090/debug/pprof/heap > memdump
But I'm gettinggo tool pprof memdump
. Here's the file (suspiciously small, I think)memdump: decompressing profile: flate: corrupt input before offset 171 - j
Jake
03/28/2022, 1:40 PMhmm, same workflow works for me locally - j
Jake
03/28/2022, 1:41 PMis it already at the level when it would be getting oom killed? - j
Jake
03/28/2022, 1:41 PMmaybe it can't allocate enough memory to dump the heap - w
williamdclt
03/28/2022, 1:46 PMI don't know what I did, here's a better dump - j
Jake
03/28/2022, 1:47 PMis this from a process that's using excess memory? i'm seeing some pretty small numbers - w
williamdclt
03/28/2022, 1:49 PMIt's a fairly random dump, so it might not show the OOM problem - w
williamdclt
03/28/2022, 1:50 PMI'm currently installing all dependencies to visualise these dumps (graphviz mostly), I'll take a few more dumps and try to find one that shows high memory usage - w
williamdclt
03/28/2022, 2:30 PMHere's a better one, taken ~1s before the pod got OOMKilled. The
on my pod is 1Gi, I'm seeing that it did reach this limit, but I'm not really seeing this in the dump.limit - j
Jake
03/28/2022, 2:32 PMyeah only 200mb in the profile - w
williamdclt
03/28/2022, 2:32 PMA "normal" dump, taken a while before the OOM (several seconds, less than a min), looks more like that - j
Jake
03/28/2022, 2:32 PMwhich datastore are you using and what does your traffic look like? - w
williamdclt
03/28/2022, 2:34 PMPostgres, ~4 req/s (will eventually be ~100x that). Mostly checks and writes, although we do have a few lookups, - j
Jake
03/28/2022, 2:34 PMwhat's the ratio to checks and writes? - j
Jake
03/28/2022, 2:35 PMactually, not sure that's going to be relevant without the schema - j
Jake
03/28/2022, 2:35 PMand how long before you see the failure at that traffic level? - w
williamdclt
03/28/2022, 2:36 PM> what's the ratio to checks and writes? Lemme turn on OTel > and how long before you see the failure at that traffic level? a couple minutes maybe? - j
Jake
03/28/2022, 2:41 PMyou should be able to get those stats out of prometheus - w
williamdclt
03/28/2022, 2:45 PMI'm seeing
but nothing for writes in prom metrics?spicedb_dispatch_check_total - j
Jake
03/28/2022, 2:45 PMtry grpc_server_handled_total - w
williamdclt
03/28/2022, 3:02 PMSimilar check/write traffic - w
williamdclt
03/28/2022, 3:03 PMI'll stop the writes, see what happens - w
williamdclt
03/28/2022, 3:04 PMStill getting OOM with 0 write traffic - j
Jake
03/28/2022, 3:05 PMok that sounds wrong - w
williamdclt
03/28/2022, 3:09 PMInterestingly, multiple pods tend to OOM at the same time, even though I disabled the dispatch clustering - w
williamdclt
03/28/2022, 3:10 PMIt's pretty empirical, it doesn't always happens and might just be coincidental - j
Jake
03/28/2022, 3:11 PMare they all starting at the same time? wouldn't that be consistent with monotonically increasing memory usage? - w
williamdclt
03/28/2022, 3:13 PMThey're starting at more-or-less the same time, but with OOM and the CrashLoopBackoff they restart at different times - j
Jake
03/28/2022, 3:13 PMare they all getting scheduled to the same node and the node is running out of memory?