3 ms·
In the article, it states that after a certain point, parallelization hits limits. Is that due to Scylladb's architecture or other constraints within the system
by ec109685 4y ago
In the article, it states that after a certain point, parallelization hits limits. Is that due to Scylladb's architecture or other constraints within the system?
For reads (unless there is another bottleneck), by 10x'ing the parallelism, an application can compensate for 10x the average latency and still deliver the same throughput.
Scylladb talked about their IO scheduler a good deal:
https://www.scylladb.com/2021/04/06/scyllas-new-io-scheduler/ https://www.scylladb.com/2021/04/06/scyllas-new-io-scheduler...
https://www.scylladb.com/2022/08/03/implementing-a-new-io-scheduler-algorithm-for-mixed-read-write-workloads/ https://www.scylladb.com/2022/08/03/implementing-a-new-io-sc...
Naively, it seems like it should have been able to compensate for the Persistent Disk latency for read heavy workloads.
- seastarer 4y agoPersistent Disk read latency is usually good, but jitters up to 100ms. The scheduler can compensate for high latency (essentially applying Little's Law) but cannot compensate for jitter without saturating the disk.
- ec109685 4y agoThanks. Do you see the same spikes with ebs volumes on AWS?
- seastarer 4y agoNo.
- deleted 4y ago[deleted]
- seastarer 4y agoSome more data on Google PD: https://raw.githubusercontent.com/scylladb/diskplorer/master/latency-matrix-results/gcp-pd-SSD-2TB.png https://raw.githubusercontent.com/scylladb/diskplorer/master... Very odd - notice how the latency is high when the read IOPS are low. When the read IOPS climb, 95th percentile latency drops. Looks like there is a constant rate of high latency requests, and when the read IOPS climb, that constant rate moves to a higher quantile. I'd inspect the raw results but they're quite big: https://github.com/scylladb/diskplorer/blob/master/latency-matrix-results/gcp-pd-SSD-2TB.json https://github.com/scylladb/diskplorer/blob/master/latency-m... Background: https://github.com/scylladb/diskplorer/ https://github.com/scylladb/diskplorer/
- ec109685 4y agoVery interesting. I wonder if higher iops involves reading from more servers and with request hedging it can smooth latency a lot better.