I am currently evaluating using CRDB SpiceDB #spicedb

I am currently evaluating using CRDB

Vilsol

07/17/2023, 9:14 PM

I am currently evaluating using CRDB serverless on a multi-region (US & EU) setup, and it seems like it's unable to go past something like writing 10 relationships per second? Something seems off, could there be any chance that you could try your new benchmark suite on that kind of setup?

vroldanbet

07/18/2023, 10:02 AM

please also note we used CRDB Dedicated, not serverless

vroldanbet

07/18/2023, 10:04 AM

and consensus across regions is going to take a toll on the write throughput.

vroldanbet

07/18/2023, 10:08 AM

we used 6 x 32 vCPU CRDB cluster single region for the 1M RPS (990K RPS Reads, 10K RPS Writes)

vroldanbet

07/18/2023, 10:10 AM

I'd suggest to reach out to CockroachLabs to understand if Serverless needs a prewarming phase before doing 10K RPS writes, and ask them if it's going to be able to handle that across multiple regions in Serverless

night

07/18/2023, 2:30 PM

We switched it to crdb dedicated with 4 vCPU, 16gb memory

night

07/18/2023, 4:39 PM

i think we were getting 10-20 writes per second

night

07/18/2023, 4:39 PM

so we have a workload where we have to check if a user can add a relationship, before adding a relationship - and that took roughly 10 seconds? cc @Vilsol

night

07/18/2023, 4:48 PM

the database is setup with two of the datacenters being 25ms apart, and the 3rd is roughly 80-90ms away from A and 100ms away from B

vroldanbet

07/18/2023, 5:00 PM

so you are mixing checks and writes?

vroldanbet

07/18/2023, 5:00 PM

those numbers don't add up to me

vroldanbet

07/18/2023, 5:00 PM

@ecordell do you have any suggestions on what these folks could be looking into?

vroldanbet

07/18/2023, 5:01 PM

I think one test y'all could do is start with one region, to understand if the problem is multi-region related

night

07/18/2023, 5:13 PM

yeah, single region wasn't that slow

night

07/18/2023, 5:13 PM

so we're fairly certain it's something to do with the multi region cockroach but we're not entirely sure how

ecordell

07/18/2023, 5:22 PM

is the 10-20 writes per second with the

insecure

overlap mode that Joey mentioned? if your regions are 100ms apart, that should be about 10 writes per second if the write transactions touch the same data. Are you writing to the same set of relationships in your test, or are you randomizing the writes?

night

07/18/2023, 5:22 PM

yeah, that includes the insecure overlap mode. in terms of relationships i believe they're different sets, but @Vilsol knows more

8 Views

Previous Next