3 ms·

Looks like pci-e is not much better here than USB 3.0 attached SSD. Just tried with 250GB samsung evo + JMS578 and that gives: dd if=/dev/sda of=/dev/null ifla

by megous 6y ago

Looks like pci-e is not much better here than USB 3.0 attached SSD. Just tried with 250GB samsung evo + JMS578 and that gives: dd if=/dev/sda of=/dev/null iflag=direct bs=16M count=100 1677721600 bytes (1.7 GB, 1.6 GiB) copied, 4.17517 s, 402 MB/s And that's also going over a USB3.0 HUB in addition.

vardump 6y ago
You seemed to miss random access I/O speed (4K reads/writes), which usually matters quite a bit more than sequential access. Random reads almost doubled even with a cheap SSD and random reads nearly tripled.
- megous 6y ago
  There's no test methodology, so it doesn't matter. Is it measuring raw block device I/O or filesystem I/O, which filesystem, is it submitting I/O requests sequentially or in batches so that the OS can queue multiple requests at once, etc. etc. There's nothing to compare against, so I didn't even try running the test. USB 3.0 has no trouble with high random I/O either, anyway. Article is comparing different SSDs, not just different interfaces, so random 4k block read speeds differ. But it doesn't show what's possible with good USB 3.0 SATA controller and good SSD. I just wanted to point out that speed difference doesn't have to have much to do with the interfaces themselves.
kabdib 6y ago
I'm curious about CPU usage for the two adapter types. My experience (quite a while back...) was that USB required a bunch more involvement from the CPU than the simpler block-oriented drivers. (Hubs don't affect things much; on the order of tens of nanoseconds of additional latency when things line up properly).
- geerlingguy 6y ago
  Overclocking the Pi's CPU _does_ increase the speed of some operations, for sure. For example, in my network tests, I can get 1.9 Gbps on a 2.5G adapter with MTU 1500 at 1.5 GHz (default Pi 4 clock), or 2.2 Gbps at 2.0 GHz. Using MTU 9000 lets me get over 2.3G on that adapter.
- megous 6y ago
  For USB 3.0 at 43k IOPS load - https://megous.com/dl/tmp/caf84c98b0a67db8.png https://megous.com/dl/tmp/caf84c98b0a67db8.png It's probably not yet limited by the CPU, but close. This is much slower (than RPi4) Orange Pi 3.
geerlingguy 6y ago
Another question: is that MB/sec or MiB/sec? Note that the 400 MiB/sec benchmark is 420 MB/sec. Not a _huge_ difference, but it is a difference ;) The other major difference is anything non-sequential, especially things like 4K reads. This is where the PCIe->USB->SATA (or PCIe->USB->NVMe) adapter overhead really slows things down. Try out my benchmark script and see how it fares: https://github.com/geerlingguy/raspberry-pi-dramble/blob/master/setup/benchmarks/disk-benchmark.sh https://github.com/geerlingguy/raspberry-pi-dramble/blob/mas... (Thank the marketers for making everything confusing with MiB and MB...)
- megous 6y ago
  Dunno, looks too complicated, and our results will not be comparable, because my block devices use LUKS encryption, and this doesn't measure raw block device access. I wrote this quick benchmark tool, maybe try it. It does test raw block device 4k random reads via random batches of 128 reads submitted via io_uring: https://megous.com/dl/tmp/t.c (reduce ROUNDS to 5 or 10 if you test something non-SSD) gcc -o t t.c -luring ./t /dev/sda On my workstation: - My HDDs show 4k read iops in the range of: 126-139 - My samsung 980 pro nvme hovers around 160-180 kIOPS - Samsung 840 PRO SATA : 95 kIOPS - Samsung 860 EVO SATA : 90 kIOPS On my Orange Pi 3: - USB 3.0 attached intel SSD : 4k IOPS = 15393 - USB 3.0 attached samsung 850 EVO : 4k IOPS = 43154 - Sandisk ultra A1 uSD card 09/2018 model: 4k IOPS = 2228 - Sandisk ultra A1 uSD card 10/2019 model: 4k IOPS = 2512 (they are improving over time :)) So that's about 60MiB/s and 168MiB/s random 4k reads over USB 3.0. (controller supports UAS) Is it hitting the SSD or interface limit? Dunno. I don't have the exact same samsung ssd model connected via SATA anywhere. But it's much more than the values in the article, so the values in the article are certainly not the USB 3.0 interface limits for 4k random reads. dd is outputing non-SI units in this case, so it was around 392MiB/s
justin66 6y ago
> Looks like pci-e is not much better here than USB 3.0 attached SSD. We shouldn't expect it to be. A good USB 3.0 hub shouldn't add much in the way of latency. I probably wouldn't bother, but M.2 is still a very cool thing that some people seem to want...

by megous 6y ago

vardump 6y ago

You seemed to miss random access I/O speed (4K reads/writes), which usually matters quite a bit more than sequential access. Random reads almost doubled even with a cheap SSD and random reads nearly tripled.

megous 6y ago
There's no test methodology, so it doesn't matter. Is it measuring raw block device I/O or filesystem I/O, which filesystem, is it submitting I/O requests sequentially or in batches so that the OS can queue multiple requests at once, etc. etc. There's nothing to compare against, so I didn't even try running the test. USB 3.0 has no trouble with high random I/O either, anyway. Article is comparing different SSDs, not just different interfaces, so random 4k block read speeds differ. But it doesn't show what's possible with good USB 3.0 SATA controller and good SSD. I just wanted to point out that speed difference doesn't have to have much to do with the interfaces themselves.

kabdib 6y ago

I'm curious about CPU usage for the two adapter types. My experience (quite a while back...) was that USB required a bunch more involvement from the CPU than the simpler block-oriented drivers. (Hubs don't affect things much; on the order of tens of nanoseconds of additional latency when things line up properly).

geerlingguy 6y ago
Overclocking the Pi's CPU _does_ increase the speed of some operations, for sure. For example, in my network tests, I can get 1.9 Gbps on a 2.5G adapter with MTU 1500 at 1.5 GHz (default Pi 4 clock), or 2.2 Gbps at 2.0 GHz. Using MTU 9000 lets me get over 2.3G on that adapter.
megous 6y ago
For USB 3.0 at 43k IOPS load - https://megous.com/dl/tmp/caf84c98b0a67db8.png https://megous.com/dl/tmp/caf84c98b0a67db8.png It's probably not yet limited by the CPU, but close. This is much slower (than RPi4) Orange Pi 3.

geerlingguy 6y ago

Another question: is that MB/sec or MiB/sec? Note that the 400 MiB/sec benchmark is 420 MB/sec. Not a _huge_ difference, but it is a difference ;) The other major difference is anything non-sequential, especially things like 4K reads. This is where the PCIe->USB->SATA (or PCIe->USB->NVMe) adapter overhead really slows things down. Try out my benchmark script and see how it fares: https://github.com/geerlingguy/raspberry-pi-dramble/blob/master/setup/benchmarks/disk-benchmark.sh https://github.com/geerlingguy/raspberry-pi-dramble/blob/mas... (Thank the marketers for making everything confusing with MiB and MB...)

megous 6y ago
Dunno, looks too complicated, and our results will not be comparable, because my block devices use LUKS encryption, and this doesn't measure raw block device access. I wrote this quick benchmark tool, maybe try it. It does test raw block device 4k random reads via random batches of 128 reads submitted via io_uring: https://megous.com/dl/tmp/t.c (reduce ROUNDS to 5 or 10 if you test something non-SSD) gcc -o t t.c -luring ./t /dev/sda On my workstation: - My HDDs show 4k read iops in the range of: 126-139 - My samsung 980 pro nvme hovers around 160-180 kIOPS - Samsung 840 PRO SATA : 95 kIOPS - Samsung 860 EVO SATA : 90 kIOPS On my Orange Pi 3: - USB 3.0 attached intel SSD : 4k IOPS = 15393 - USB 3.0 attached samsung 850 EVO : 4k IOPS = 43154 - Sandisk ultra A1 uSD card 09/2018 model: 4k IOPS = 2228 - Sandisk ultra A1 uSD card 10/2019 model: 4k IOPS = 2512 (they are improving over time :)) So that's about 60MiB/s and 168MiB/s random 4k reads over USB 3.0. (controller supports UAS) Is it hitting the SSD or interface limit? Dunno. I don't have the exact same samsung ssd model connected via SATA anywhere. But it's much more than the values in the article, so the values in the article are certainly not the USB 3.0 interface limits for 4k random reads. dd is outputing non-SI units in this case, so it was around 392MiB/s

justin66 6y ago

> Looks like pci-e is not much better here than USB 3.0 attached SSD. We shouldn't expect it to be. A good USB 3.0 hub shouldn't add much in the way of latency. I probably wouldn't bother, but M.2 is still a very cool thing that some people seem to want...