3 ms·

Probing My SSD's Latency

58 points by zdw 5y ago

nerpderp82 5y ago
The Aerospike ACT benchmarking tool characterizes flash devices on their latency under a fixed read/write profile. It won't tell you how fast your device is, it will tell you if a workload runs under a certain latency bound. https://docs.aerospike.com/server/operations/plan/ssd/ssd_certification https://docs.aerospike.com/server/operations/plan/ssd/ssd_ce... https://github.com/aerospike/act https://github.com/aerospike/act If you set max-lag-sec: 0 Then it will continue to run even if temporarily falls behind.
IvyMike 5y ago
This use of dt, etc, is pretty cool, and I understand the point of this this blog entry was an exploration of those tools. But in the wider world of SSD performance, you're going to want more. fio (written by Jens Axboe, who's responsible for a lot of the I/O subsystem in Linux) is the standard tool used to do benchmarking, has tons of options to control I/O mix and parallelism, captures both thruput and latency numbers, with lots of switches to control exactly what data is collected. https://github.com/axboe/fio https://github.com/axboe/fio "Something weird going on with the 1K bucket landing at the bottom." The SSD's Flash Translation Layer will occasionally do both caching and garbage collection, leading to the weird extreme long latency outlier. The long spike at 1K is weird, but note that optimal internal write size of the SSD is probably something like 64K or 128K these days. Different SSDs will have different FTL algorithms, different overprovisioning, different internal parallelism, etc, so it can be very tricky predict actual real-life performance. https://en.wikipedia.org/wiki/Flash_memory_controller https://en.wikipedia.org/wiki/Flash_memory_controller
- mgerdts 5y ago
  I think the 1k bucket statement was referring to the sort order. That bucket represents about 1 to 2 microseconds, which is dubiously fast. Maybe this is possible if the IO is serviced by DRAM on the NVMe controller. Maybe the IOs in question completed without going to the drive, such as in some error condition. There are also a bunch of IOs in the 0 bucket. I’m not sure how you make it from sdstrategy to sd_buf_done with the passage of no time. Perhaps the granularity if the nsecs variable’s values is rather coarse.
- yakubin 5y ago
  Using just fio without kernel tracing, you run the risk of testing the filesystem instead of the storage medium. You could use fio to fabricate I/O operations, but the measurements coming from tracing are more trustworthy than the userspace measurements fio makes (unless fio does some in-kernel measurements I don't know about).
  - IvyMike 5y ago
    Regarding measuring to avoid profiling the filesystem: You can run fio against the raw block device to avoid this.
mgerdts 5y ago
The typical tool for measuring throughput and latency is fio. If you look at datasheets for NVMe drives, you can normally find the parameters fed to fio to arrive at the numbers on the spec sheet.
jjav 5y ago
On a HDD, let's say a file is ordered on disk in blocks 1,2,3,1000,4,1001,5,6,7. Then reading the 4th, 5th, 6th and 7th block will have larger latency than reading the 2nd, 3rd, 8th, 9th blocks because the drive heads have to move a larger distance back and forth. What I've found puzzling and still don't quite understand is why files on an SSD exhibit some (less, but observable) of this same behavior. There's no drive head to seek, so why does it take longer? I've noticed this when trying to read files quickly from disk. On a HDD it is clearly faster to read blocks 1,2,3,4,5,6,7,1000,1001 and reorder then in RAM instead of having the drive head seek back and forth. But why is it also quicker to read them that way off an SSD?
- zaarn 5y ago
  I would hazard a guess and say prefetching. SSDs have large blocks, so when you read 1, blocks 2-5 are maybe part of the block and loaded in anyway. Then the controller might also cache 6-10 for the heck of it. And 1000+ isn't, so it's not in cache, so it's slower. You might want to try a skipping read test, ie, reading sector 1, sector +10MB, sector +1GB, etc until end of disk, to avoid the controller or OS being able to prefetch data into cache.
- dataflow 5y ago
  I another effect (besides prefetching which was just mentioned, and likely many others) is that SSDs stripe data across multiple chips for parallel sequential access, but you might end up negating some of the benefit of that if you end up accessing the same chip.
lodi 5y ago
If you use zfs, you can get the same kind of histogram with `zpool iostat -w`.

58 points by zdw 5y ago

The Aerospike ACT benchmarking tool characterizes flash devices on their latency under a fixed read/write profile. It won't tell you how fast your device is, it will tell you if a workload runs under a certain latency bound. https://docs.aerospike.com/server/operations/plan/ssd/ssd_certification https://docs.aerospike.com/server/operations/plan/ssd/ssd_ce... https://github.com/aerospike/act https://github.com/aerospike/act If you set max-lag-sec: 0 Then it will continue to run even if temporarily falls behind.

IvyMike 5y ago

This use of dt, etc, is pretty cool, and I understand the point of this this blog entry was an exploration of those tools. But in the wider world of SSD performance, you're going to want more. fio (written by Jens Axboe, who's responsible for a lot of the I/O subsystem in Linux) is the standard tool used to do benchmarking, has tons of options to control I/O mix and parallelism, captures both thruput and latency numbers, with lots of switches to control exactly what data is collected. https://github.com/axboe/fio https://github.com/axboe/fio "Something weird going on with the 1K bucket landing at the bottom." The SSD's Flash Translation Layer will occasionally do both caching and garbage collection, leading to the weird extreme long latency outlier. The long spike at 1K is weird, but note that optimal internal write size of the SSD is probably something like 64K or 128K these days. Different SSDs will have different FTL algorithms, different overprovisioning, different internal parallelism, etc, so it can be very tricky predict actual real-life performance. https://en.wikipedia.org/wiki/Flash_memory_controller https://en.wikipedia.org/wiki/Flash_memory_controller

mgerdts 5y ago
I think the 1k bucket statement was referring to the sort order. That bucket represents about 1 to 2 microseconds, which is dubiously fast. Maybe this is possible if the IO is serviced by DRAM on the NVMe controller. Maybe the IOs in question completed without going to the drive, such as in some error condition. There are also a bunch of IOs in the 0 bucket. I’m not sure how you make it from sdstrategy to sd_buf_done with the passage of no time. Perhaps the granularity if the nsecs variable’s values is rather coarse.
yakubin 5y ago
Using just fio without kernel tracing, you run the risk of testing the filesystem instead of the storage medium. You could use fio to fabricate I/O operations, but the measurements coming from tracing are more trustworthy than the userspace measurements fio makes (unless fio does some in-kernel measurements I don't know about).
- IvyMike 5y ago
  Regarding measuring to avoid profiling the filesystem: You can run fio against the raw block device to avoid this.

mgerdts 5y ago

The typical tool for measuring throughput and latency is fio. If you look at datasheets for NVMe drives, you can normally find the parameters fed to fio to arrive at the numbers on the spec sheet.

jjav 5y ago

On a HDD, let's say a file is ordered on disk in blocks 1,2,3,1000,4,1001,5,6,7. Then reading the 4th, 5th, 6th and 7th block will have larger latency than reading the 2nd, 3rd, 8th, 9th blocks because the drive heads have to move a larger distance back and forth. What I've found puzzling and still don't quite understand is why files on an SSD exhibit some (less, but observable) of this same behavior. There's no drive head to seek, so why does it take longer? I've noticed this when trying to read files quickly from disk. On a HDD it is clearly faster to read blocks 1,2,3,4,5,6,7,1000,1001 and reorder then in RAM instead of having the drive head seek back and forth. But why is it also quicker to read them that way off an SSD?

zaarn 5y ago
I would hazard a guess and say prefetching. SSDs have large blocks, so when you read 1, blocks 2-5 are maybe part of the block and loaded in anyway. Then the controller might also cache 6-10 for the heck of it. And 1000+ isn't, so it's not in cache, so it's slower. You might want to try a skipping read test, ie, reading sector 1, sector +10MB, sector +1GB, etc until end of disk, to avoid the controller or OS being able to prefetch data into cache.
dataflow 5y ago
I another effect (besides prefetching which was just mentioned, and likely many others) is that SSDs stripe data across multiple chips for parallel sequential access, but you might end up negating some of the benefit of that if you end up accessing the same chip.

lodi 5y ago

If you use zfs, you can get the same kind of histogram with `zpool iostat -w`.