Hello Team, In our spicedb setup we have
# spicedbr
rohitlal.
11/27/2023, 9:37 AMHello Team, In our spicedb setup we have only 2 spicedb nodes no client or operator. We are observing spikes in memory,cpu and disk as well and it crashes some times only in one pod. This is being observed majorly when data push is happening in to db. we are using postgresql as our db. Can you please helpe us solve this issue. Please let me know if you need our yaml details.
v
vroldanbet
11/27/2023, 12:01 PMplease provide information about the deployment setup, the client workload and the version you are using
y
yetitwo
11/27/2023, 2:35 PMthis could be SpiceDB's garbage collection mechanism, especially if 1. you haven't modified the default spicedb settings 2. your data push does writes one-by-one and 3. your datastore is undersized
r
rohitlal.
11/28/2023, 4:58 AMrohitlal.
11/28/2023, 5:05 AMwhat is client workload means?
rohitlal.
11/28/2023, 5:07 AMSpice DB version: 1.16
rohitlal.
11/28/2023, 5:11 AMCan you please suggest what settings we need to do which will solve this issue? Currently we are using default spice DB settings.
v
vroldanbet
11/28/2023, 8:36 AMwhat kind of requests are you doing to SpiceDB when it crashes
vroldanbet
11/28/2023, 8:36 AMPlease move to the most recent version 1.27.0
vroldanbet
11/28/2023, 8:38 AMThis looks OK. Are you sure dispatching is properly configured?
vroldanbet
11/28/2023, 8:39 AMDisk seems unusual - make sure you don't have debug log level enabled
r
rohitlal.
11/28/2023, 1:56 PMIn Spice DB we are storing some permission related stuff which is around 80k data
rohitlal.
11/28/2023, 1:56 PMsure
rohitlal.
11/28/2023, 1:56 PMHow we can check dispatching thing?
v
vroldanbet
11/28/2023, 2:28 PMyou can check that the SpiceDB DispatchCheck API is being called
r
rohitlal.
11/28/2023, 3:25 PMbut how we can configure this properly?
v
vroldanbet
11/28/2023, 3:30 PMit should be something like this
Copy code
- name: SPICEDB_DISPATCH_UPSTREAM_ADDR
value: kubernetes:///<your_service_name>.<your_service_kube_namespace>:<your_service_dispatch_port_name>y
yetitwo
11/28/2023, 3:50 PMare you making the writes in batches or are you making those writes individually?
r
rohitlal.
11/29/2023, 4:49 AMFrom kafka we are pushing the data one by one into spice DB.
rohitlal.
11/29/2023, 5:00 AMThanks can you please help me where it needs to be configure?
rohitlal.
11/29/2023, 5:06 AMCan you please confirm what is spiceDB setting we need to configure to fix this issue?
rohitlal.
11/29/2023, 6:34 AMDo you recommend to use batching to push the data into spiceDB?
v
vroldanbet
11/29/2023, 3:03 PMthis would be in your kubernetes Deployment. This is why we recommend using the operator so you don't have to deal with these bits and automate the upgrades - we strongly recommend moving to the operator
y
yetitwo
11/29/2023, 3:58 PMthat's likely a problem. are you using postgres as your datastore?
yetitwo
11/29/2023, 3:59 PMwe ran into an issue where pushing updates one-by-one through kafka and it created a new postgres snapshot for each update. by default those snapshots are retained for 24hrs and then garbage collected, but if the garbage collection can't complete within its timeout, it will thrash, which will cause high CPU usage on your datastore without any traffic on it
yetitwo
11/29/2023, 4:01 PMwe fixed the issue by reducing the snapshot retention to 1hr (the window only needs to be as large as the maximum window in which you want to call
at_exact_snapshotyetitwo
11/29/2023, 4:01 PMupsizing our database also helped
v
vroldanbet
11/29/2023, 6:12 PMthat is fixed in 1.27.0 @yetitwo - GC is now way faster
vroldanbet
11/29/2023, 6:13 PMthe problem is that the queries were very inefficient. The GC should be able to keep up, as once it encounters garbage, it should iterate fast over it
vroldanbet
11/29/2023, 6:14 PMWe added a new index to it. The problem is that new index requires Postgres 15 to work, because the query planner would refuse to select it in earlier versious - due to the xid8 type, which was relatively new to PG
vroldanbet
11/29/2023, 6:19 PMbut at least we didn't observe this causing OOMKills - it saturated RDS and made SpiceDB slow overall
r
rohitlal.
11/30/2023, 5:25 AMThank you let us try upgrade the spiceDB version i hope that will solve this CPU memory issue.
rohitlal.
11/30/2023, 5:29 AMCan you please suggest which version of postres we need to use as Datastore to fix this issue?
rohitlal.
11/30/2023, 7:45 AMIf we are upgrading spiceDB to 1.27.0 then what should the postgresql db version we need to use?
v
vroldanbet
11/30/2023, 8:45 AMPG 15
r
rohitlal.
11/30/2023, 9:34 AMDo we have any document of SpiceDB where its mentioned that to use PG 15 with SpiceDB Latest version?
rohitlal.
11/30/2023, 9:35 AMCan we use Latest version PG16?
v
vroldanbet
11/30/2023, 10:14 AMThe minimum supported version is PG 13.8. SpiceDB is being tested against 13.7, 14, 15, 16. There should be documentation in the public docs about the minimum required version,.but not about the recommended PG15
r
rohitlal.
11/30/2023, 2:10 PMWhat my question is to fix this spiceDB Spike issue We need to upgrade the version to 1.27 and we are using PG 14.7 so do we need to upgrade PG version to 15 or 16 to fix that Garbage collection issue?
v
vroldanbet
11/30/2023, 2:28 PMIm not sure your issue is related with the GC inefficiency fixed in 1.27. What I would definitely suggest is to update because there have been many many perf improvements since 1.16.
You don't need to update to PG15, it's not strictly necessary to run 1.27, but if indeed the problem was related to GC, then you need to at least run PG15 to have it fixed
r
rohitlal.
12/01/2023, 6:41 AMOk. Thank You. Let us try update spiceDB and upgrade PG to 15 then i will update you.
v
vroldanbet
12/01/2023, 9:50 AM1.28 was just released - use that
r
rohitlal.
12/01/2023, 10:07 AMSure. Thanks !!