4 ms·

A few ballpark numbers I encountered: Sequentially reading a file on a spinny laptop disk was about 80-100 MB/s. On an SSD that went up to 400-500 MB/s for me

by mrkeen 4y ago

A few ballpark numbers I encountered: Sequentially reading a file on a spinny laptop disk was about 80-100 MB/s. On an SSD that went up to 400-500 MB/s for me. That's the sequential case! What about random access? I tried an experiment where I memory mapped a large file and started updating bytes at random. I could get the rate down to kilobytes/sec. Even though we've all heard that SSDs don't pay as much as a penalty for random access as spinny disks, it's still a huge penalty. Sequential spinny disk access is faster than SSD random access.

GrayShade 4y ago
> Sequential spinny disk access is faster than SSD random access. It is, but on both kind of drives you'll want to dispatch at least a couple of requests at once to get better performance. In the memory-mapped case, that means using multiple threads. In addition, you might also want to call madvise(MADV_RANDOM) on the mapping.
- mrkeen 4y ago
  I didn't think there'd be that much discussion on my point (seq hdd > rand sdd). > In the memory-mapped case, that means using multiple threads. My gut tells me I'd lose more to contention/false-sharing than I'd gain through multithreading - but I haven't done the experiment.
koolba 4y ago
> Sequential spinny disk access is faster than SSD random access. No it’s not. At least not with modern SSDs or NVMe storage. Even at 100 MB/s, a spinning disk in sequential mode is doing 100 x 1024 / 4 = 25,600 IOPS (assuming a standard 4K per operation). Even consumer grade NVMe hardware gets 5-10x of that for random workloads.
- deleted 4y ago
  [deleted]
- kragen 4y ago
  it is true that spinning rust is slower than most ssds including nvme ssds even in sequential mode however, a spinning disk doing a sequential access is not doing 25600 iops if the sequential access lasts 10 seconds it is doing 0.1 iops
- mrkeen 4y ago
  > Even consumer grade NVMe hardware gets 5-10x of that for random workloads. Cool, lots of IOPS! But like I said, I got it down to kilobytes/sec.
  - koolba 4y ago
    On what hardware and how much parallelization? The max IOPS numbers are only when you’re saturating the command queue. It’s a throughput number, not a single operation, completion, followed by the next one.
  - justsomehnguy 4y ago
    Because you did it the most inefficient way. This [0] comment is totally on point. Also note what a consumer SSDs can be made even with a single flash chip. A more performant ones are made of bunch of chips internally (essentially a RAID0 with some magic) so they can do a parallel operations if the data resides on the different flash blocks. Still, if your thread is only doing one operation a time with blocks < flash rewrite block size you will hit the write amplification anyway. I think if you do the same test but without a memory mapped file (ie let the OS and disk subsystem do their thing) you will get much more speed. [0] https://news.ycombinator.com/item?id=33751973 https://news.ycombinator.com/item?id=33751973
wtallis 4y ago
> I tried an experiment where I memory mapped a large file and started updating bytes at random. I could get the rate down to kilobytes/sec. Memory-mapped IO means you're only giving the SSD one request to work on at a time, because a thread can only page fault on one page at a time. An SSD can only reach its peak random IO throughput if you give it lots of requests to work on in parallel. Additionally, your test was probably doing small writes with all volatile caching disallowed, forcing the (presumably consumer rather than enterprise) SSD to perform read-modify-write cycles not just of the 4kB virtual memory pages the OS works with, but also the larger native flash memory page size (commonly 16kB). If you'd been testing only read performance, or permitted a normal degree of write caching, you would have seen far higher performance.
d_tr 4y ago
True. And main memory access can easily become slower than SSD sequential access if you do random byte accesses and your working set is larger than the CPU's caches or TLBs.

by mrkeen 4y ago

GrayShade 4y ago

> Sequential spinny disk access is faster than SSD random access. It is, but on both kind of drives you'll want to dispatch at least a couple of requests at once to get better performance. In the memory-mapped case, that means using multiple threads. In addition, you might also want to call madvise(MADV_RANDOM) on the mapping.

mrkeen 4y ago
I didn't think there'd be that much discussion on my point (seq hdd > rand sdd). > In the memory-mapped case, that means using multiple threads. My gut tells me I'd lose more to contention/false-sharing than I'd gain through multithreading - but I haven't done the experiment.

koolba 4y ago

> Sequential spinny disk access is faster than SSD random access. No it’s not. At least not with modern SSDs or NVMe storage. Even at 100 MB/s, a spinning disk in sequential mode is doing 100 x 1024 / 4 = 25,600 IOPS (assuming a standard 4K per operation). Even consumer grade NVMe hardware gets 5-10x of that for random workloads.

deleted 4y ago
[deleted]
kragen 4y ago
it is true that spinning rust is slower than most ssds including nvme ssds even in sequential mode however, a spinning disk doing a sequential access is not doing 25600 iops if the sequential access lasts 10 seconds it is doing 0.1 iops
mrkeen 4y ago
> Even consumer grade NVMe hardware gets 5-10x of that for random workloads. Cool, lots of IOPS! But like I said, I got it down to kilobytes/sec.
- koolba 4y ago
  On what hardware and how much parallelization? The max IOPS numbers are only when you’re saturating the command queue. It’s a throughput number, not a single operation, completion, followed by the next one.
- justsomehnguy 4y ago
  Because you did it the most inefficient way. This [0] comment is totally on point. Also note what a consumer SSDs can be made even with a single flash chip. A more performant ones are made of bunch of chips internally (essentially a RAID0 with some magic) so they can do a parallel operations if the data resides on the different flash blocks. Still, if your thread is only doing one operation a time with blocks < flash rewrite block size you will hit the write amplification anyway. I think if you do the same test but without a memory mapped file (ie let the OS and disk subsystem do their thing) you will get much more speed. [0] https://news.ycombinator.com/item?id=33751973 https://news.ycombinator.com/item?id=33751973

wtallis 4y ago

> I tried an experiment where I memory mapped a large file and started updating bytes at random. I could get the rate down to kilobytes/sec. Memory-mapped IO means you're only giving the SSD one request to work on at a time, because a thread can only page fault on one page at a time. An SSD can only reach its peak random IO throughput if you give it lots of requests to work on in parallel. Additionally, your test was probably doing small writes with all volatile caching disallowed, forcing the (presumably consumer rather than enterprise) SSD to perform read-modify-write cycles not just of the 4kB virtual memory pages the OS works with, but also the larger native flash memory page size (commonly 16kB). If you'd been testing only read performance, or permitted a normal degree of write caching, you would have seen far higher performance.

d_tr 4y ago

True. And main memory access can easily become slower than SSD sequential access if you do random byte accesses and your working set is larger than the CPU's caches or TLBs.