williamdclt
10/12/2022, 11:00 AMresource
configuration:
resources:
requests:
cpu: 500m
memory: 1Gi
limits:
cpu: 1000m
memory: 2Gi
vroldanbet
10/12/2022, 11:11 AMwilliamdclt
10/12/2022, 11:25 AMvroldanbet
10/12/2022, 11:27 AMvad
10/12/2022, 1:06 PM--dispatch-cluster-cache-max-cost
(which defaults to 70%) or --dispatch-cache-max-cost
(default=30%) ?vroldanbet
10/12/2022, 1:07 PMvad
10/12/2022, 1:08 PMvroldanbet
10/12/2022, 1:08 PMvad
10/12/2022, 1:09 PMgo_memstats_mallocs_total{job="spicedb-production-metrics"} - go_memstats_frees_total{job="spicedb-production-metrics"}
vroldanbet
10/12/2022, 1:11 PMvad
10/12/2022, 1:11 PMvroldanbet
10/12/2022, 1:11 PMvad
10/12/2022, 1:12 PMvroldanbet
10/12/2022, 1:12 PMvad
10/12/2022, 1:13 PMvroldanbet
10/12/2022, 1:13 PMvad
10/12/2022, 1:14 PMvroldanbet
10/12/2022, 1:15 PMvad
10/12/2022, 1:16 PMvroldanbet
10/12/2022, 1:16 PMvad
10/12/2022, 1:18 PMjzelinskie
10/12/2022, 3:21 PMvad
10/13/2022, 10:04 AMwilliamdclt
10/13/2022, 10:58 AM--dispatch-cache-max-cost
to 750MiB
, doesn't seem to have changed anything really--dispatch-cluster-cache-max-cost
)vad
10/13/2022, 1:06 PMjzelinskie
10/13/2022, 2:36 PMwilliamdclt
10/13/2022, 3:26 PMjzelinskie
10/14/2022, 3:36 PMvad
10/15/2022, 7:05 PMjzelinskie
10/17/2022, 3:26 PMvad
10/18/2022, 2:58 PMjzelinskie
10/18/2022, 2:59 PMvad
10/19/2022, 11:54 AMvroldanbet
10/19/2022, 11:57 AMvad
10/19/2022, 2:27 PMvroldanbet
10/19/2022, 2:30 PMdispatch-cache-metrics
- dispatch-cluster-cache-metrics
once enabled, you shall see the following metrics:
spicedb_dispatch_client_cost_added_bytes
spicedb_dispatch_client_cost_evicted_bytes
spicedb_dispatch_cost_added_bytes
spicedb_dispatch_cost_evicted_bytes
vad
10/19/2022, 2:43 PMvroldanbet
10/19/2022, 2:51 PMvad
10/19/2022, 2:57 PMvroldanbet
10/19/2022, 2:58 PMjzelinskie
10/19/2022, 3:35 PMvad
10/20/2022, 7:44 AMvroldanbet
10/20/2022, 7:46 AMvad
10/20/2022, 7:50 AMvroldanbet
10/20/2022, 7:50 AMvad
10/20/2022, 8:20 AMvroldanbet
10/20/2022, 8:29 AMvad
10/20/2022, 9:09 AMvroldanbet
10/21/2022, 3:13 PMjzelinskie
10/21/2022, 4:11 PMvad
11/03/2022, 7:42 AMjzelinskie
11/03/2022, 3:12 PMvad
11/03/2022, 3:36 PMjzelinskie
11/03/2022, 3:47 PMvad
11/03/2022, 4:47 PMjzelinskie
11/03/2022, 5:16 PMvad
11/04/2022, 8:30 AMjzelinskie
11/04/2022, 3:07 PMvad
11/05/2022, 8:31 PMjzelinskie
11/05/2022, 8:32 PMvad
11/08/2022, 4:34 PMjzelinskie
11/09/2022, 4:19 PMvad
11/10/2022, 8:23 AMvroldanbet
11/15/2022, 3:51 PMvad
11/15/2022, 3:59 PMvroldanbet
11/15/2022, 4:02 PMvad
11/16/2022, 10:48 AMvroldanbet
11/16/2022, 10:50 AMvad
11/16/2022, 1:32 PMvroldanbet
11/16/2022, 2:02 PMAlexB
12/22/2022, 3:31 PMvroldanbet
12/22/2022, 5:54 PMAlexB
12/27/2022, 9:26 AMvroldanbet
12/27/2022, 9:27 AMAlexB
01/04/2023, 5:21 PMJoey
01/04/2023, 6:27 PMps
show for the memory usage of the process?jzelinskie
01/04/2023, 6:32 PMgo tool pprof http://$SPICEDB_URL:$METRICS_PORT/debug/pprof/heap
should download a tarball that you can upload to https://pprof.me to share with usAlexB
01/04/2023, 9:42 PM/ # ps -o pid,user,vsz,rss,comm
PID USER VSZ RSS COMMAND
1 65532 1.7g 1.1g spicedb
Joey
01/04/2023, 9:47 PMAlexB
01/04/2023, 9:54 PMgo tool pprof
-> web
displays Showing nodes accounting for 443.62MB 92.88% of 477.64MB total
(perhaps my expectation that this would represent the total memory used is incorrect)Joey
01/04/2023, 9:54 PM[spicedb] github.com/authzed/spicedb/internal/namespace.(*ReachabilityGraph).collectEntrypoints
jumps out on the cumulative graphAlexB
01/10/2023, 10:56 PMJoey
01/10/2023, 11:00 PMAlexB
01/19/2023, 10:17 PMps -o pid,user,vsz,rss,comm
PID USER VSZ RSS COMMAND
1 65532 3.0g 1.4g spicedb
https://pprof.me/f56c390/
ps -o pid,user,vsz,rss,comm
PID USER VSZ RSS COMMAND
1 65532 3.0g 2.2g spicedb
https://pprof.me/567b9c8/Joey
01/19/2023, 10:20 PMjzelinskie
01/19/2023, 10:26 PMAlexB
01/19/2023, 10:29 PMJoey
01/19/2023, 10:30 PMjzelinskie
01/19/2023, 10:30 PMJoey
01/19/2023, 10:31 PMAlexB
01/19/2023, 10:34 PMJoey
01/19/2023, 10:36 PMpdow
01/25/2023, 2:28 PMvroldanbet
01/25/2023, 2:31 PMpdow
01/25/2023, 2:43 PMAlexB
01/25/2023, 2:48 PMPID USER VSZ RSS COMMAND
1 65532 6.3g 4.6g spicedb
pdow
01/25/2023, 2:54 PMJoey
01/25/2023, 3:18 PMpdow
01/25/2023, 3:43 PMJoey
01/25/2023, 3:47 PMpdow
01/26/2023, 12:40 PMAlexB
01/26/2023, 2:56 PMvroldanbet
01/26/2023, 3:04 PMAlexB
01/26/2023, 3:13 PMvroldanbet
01/26/2023, 3:14 PMJoey
01/26/2023, 3:34 PMmain
AlexB
01/26/2023, 3:51 PMJoey
01/26/2023, 3:51 PMAlexB
02/07/2023, 10:39 AMvroldanbet
02/09/2023, 10:05 AM