bigzofsg
11/24/2022, 8:46 PMYansong
11/25/2022, 6:45 AM14:44:26 › zed schema write --skip-version-check <(cat << EOF
definition test {}
EOF
)
14:44:37 › zed schema read --skip-version-check
definition blog/test {}
Yansong
11/25/2022, 6:46 AMblog/
Yansong
11/25/2022, 6:46 AMYansong
11/25/2022, 6:53 AM14:49:53 › zed schema read --skip-version-check
definition blog/user {}
definition blog/post {
relation reader: blog/user
relation writer: blog/user
permission read = reader + writer
permission write = writer
}
14:50:46 › zed relationship create --skip-version-check blog/post:1 writer blog/user:emilia
GgUKAzEzNQ==
14:51:16 › zed permission check --skip-version-check blog/post:1 read blog/user:emilia
false
14:51:27 › zed relationship read blog/post
blog/post:1 writer blog/user:emilia
Yansong
11/25/2022, 6:54 AMYansong
11/25/2022, 8:04 AMYansong
11/25/2022, 8:05 AMJoey
11/25/2022, 8:11 AMJoey
11/25/2022, 8:12 AMYansong
11/25/2022, 8:19 AMYansong
11/25/2022, 8:19 AMYansong
11/25/2022, 8:23 AMvroldanbet
11/25/2022, 12:20 PMvroldanbet
11/25/2022, 12:21 PMPhani
11/25/2022, 3:54 PMvroldanbet
11/25/2022, 4:45 PMPhani
11/26/2022, 12:21 AMJoey
11/26/2022, 12:22 AMJoey
11/26/2022, 12:24 AMJoey
11/26/2022, 12:24 AMMohan Kandasamy
11/26/2022, 7:37 AMvroldanbet
11/26/2022, 11:02 AMvroldanbet
11/26/2022, 11:03 AMwilliamdclt
11/27/2022, 12:02 PMadd-xid-constraints
migration, running write-both-read-new
, and I'm observing much higher response time. Initially the response time shot through the roof (>1s P95), it seemed to be a pod CPU issue rather than a database issue, so I mitigated by doubling the # of replicas. It brought back the response time within usable range, but it's still much higher than it was on 1.13.0: my P50 is 3x higher (~15ms) and my P95 is 2x higher (130ms).
I'm not seeing any particular pressure on the database, and the CPU usage of my pods is way higher than it used to be (>0.5, used to be ~0.1), so I assume something is burning CPU that wasn't before.
I have this problem with write-both-read-new
but also write-both-read-old
.
I have traces but I'm not getting any insight from them, maybe you do?
I'm concerned about whether my cluster will be able to handle the load on Monday: I'm already breaching my SLOs with these response times on a Sunday where we have low traffic 😰place
11/28/2022, 12:35 AMJoey
11/28/2022, 12:38 AMplace
11/28/2022, 12:46 AMplace
11/28/2022, 12:46 AMplace
11/28/2022, 12:48 AM