4 ms·
My experience with SSDs tells me the only way to beat the system is to employ some append-only log storage structure. Potentially with segmentation done at the
by bob1029 5y ago
My experience with SSDs tells me the only way to beat the system is to employ some append-only log storage structure. Potentially with segmentation done at the device level, so that you can have a large fleet of drives in "append" mode while others are reconciling or cleaning all their blocks in anticipation of taking another full sequential fill-up. Throwing mixed workloads at individual devices is just asking for trouble if you are trying to maintain some razor-thin SLA. Dirty pages and all the weird tricks employed to hide this concern results in side effects that break more complex schemes on top.
Taking to the next level - Batching your I/O in software is how you can start saying things like "Transactions per disk I/O", not just "Fewer I/O for those transactions which now fit in fewer blocks due to compression". Batching doesn't have to mean "nightly processing". It can mean "all requests which occurred over the last 100uS". From a user's perspective, this can effectively still be a real-time RPC experience. For systems with very heavy load, this sort of micro-batching can add many orders of magnitude improvement in throughput. Also bear in mind that the more transactions you have available to compress each time means you get better odds when dealing with entropy.
I have personally developed software that can insert 2-5x the stated write IOPS figure with these sorts of tactics. On modern NVMe devices this can mean you start tickling 8 figure transactions per second if the size of each request is very modest.
- pdimitar 5y agoI would love to read a more detailed blog post about the software you wrote (if it's not confidential). Sounds like you solved very interesting problems.
- wtallis 5y agoAlong these lines, NVMe has standardized an optional Predictable Latency Mode so the host can instruct the SSD to temporarily suspend background work. It's intended to help you offer tight latency guarantees when you have intelligent load balancing across a large pool of drives.