3 ms·

Your example proves me right. Your drive should be capable of 1000 MB/s but O_DIRECT reduces performance to 400 MB/s. This matters in the specific use case of

by mrb 3y ago

Your example proves me right. Your drive should be capable of 1000 MB/s but O_DIRECT reduces performance to 400 MB/s. This matters in the specific use case of "netcat | gunzip | dd" as the compressed data rate on GigE will indeed be around 120 MB/s but when gunzip is decompressing unused parts of the filesystem (which compress very well), it will attempt to write 1+ GB/s or more to the pipe to dd and it would not be able to keep up with O_DIRECT. Another thing you are doing wrong: benchmarking with /dev/zero. Many NVMe do transparent compression so writing zeroes is faster than writing random data and thus not a realistic benchmark. PS: to clarify I am very well aware that not using O_DIRECT gives the impression initial writes are faster as they just fill the buffer cache. I am taking about sustained I/O performance over minutes as measured with, for example, iostat. You are talking to someone who has been doing Linux sysadmin and perf optimizations for 25 years :) PPS: verifying data integrity is easy with the dd solution. I usually run "sha1sum /dev/nvme0nX" on both source and destination. PPPS: I don't think Clonezilla is even capable of doing something similar (copying a remote disk to local disk without storing an intermediate disk image).

bayindirh 3y ago
> Your example proves me right. Your drive should be capable of 1000 MB/s but O_DIRECT reduces performance to 400 MB/s. I noted that the bus I connected the device has 500MBps bandwidth theoretical, no? To cite myself: > Target is a Samsung T7 Shield 2TB, with 1050MB/sec sustained write speed. Bus is USB 3.0 with 500MBps top speed (so I can go %50 of drive speeds). Result is 404MBps, which is fair for the bus.
- mrb 3y ago
  Yes USB3.0 is 500 MB/s but are you sure your bus is 3.0? It would imply your machine is 10+ years old. Most likely it's 3.1 or newer which is 1000 MB/s. And again, benchmarking /dev/zero is invalid anyway as I explained (transparent compression)
  - doubled112 3y ago
    TIL they have been sneaking versions of USB in while I haven't been paying attention. Even on hardware I own. Thanks for that.
  - crote 3y ago
    No, it wouldn't imply the machine is 10+ years old. Even a state-of-the-art motherboard like the Gigabyte Z790 D AX (which became available in my country today) has more USB 3 gen1 (5Gbps) ports than gen2 (10Gbps). The 5Gbps ports are just marketed as "USB 3.1" instead of "USB 3.0" these days, because USB naming is confusing and the important part is the "gen x".
    - mrb 3y ago
      I didn't mean 5 Gbps USB ports have disappeared, but rather: most machines in the last ~10 years (~8-9 years?) have some 10 Gbps ports. Therefore if he is plugging a fast SSD in a slow 5 Gbps port, my assumption was that he has no 10 Gbps port.
    - Arnavion 3y ago
      To be clear for everyone: USB 3.0, USB 3.1 gen 1, and USB 3.2 gen 1x1 are all names for the same thing, the 5Gbps speed. USB 3.1 gen 2 and USB 3.2 gen 2x1 are both names for the same thing, the 10Gbps speed. USB 3.2 gen 2x2 is the 20Gbps speed. The 3.0 / 3.1 / 3.2 are the version number of the USB specification. The 3.0 version only defined the 5Gbps speed. The 3.1 version added a 10Gbps speed, called it gen 2, and renamed the previous 5Gbps speed to gen 1. The 3.2 version added a new 20Gbps speed, called it gen 2x2, and renamed the previous 5Gbps speed to gen 1x1 and the previous 10Gbps speed to gen 2x1. There's also a 3.2 gen 1x2 10Gbps speed but I've never seen it used. The 3.2 gen 1x1 is so ubiqitous that it's also referred to as just "3.2 gen 1". And none of this is to be confused with type A vs type C ports. 3.2 gen 1x1 and 3.2 gen 2x1 can be carried by type A ports, but not 3.2 gen 2x2. 3.2 gen 1x1 and 3.2 gen 2x1 and 3.2 gen 2x2 can all be carried by type C ports. Lastly, because 3.0 and 3.1 spec versions only introduced one new speed each and because 3.2 gen 2x2 is type C-only, it's possible that a port labeled "3.1" is 3.2 gen 1x1, a type A port labeled "3.2" is 3.2 gen2x1, and a type C port labeled "3.2" is 3.2 gen 2x2. But you will have to check the manual / actual negotiation at runtime to be sure.
ufocia 3y ago
I wonder how using tee to compute the hash in parallel would affect the overall performance.
- mrb 3y ago
  On GigE or even 2.5G it shouldn't slow things down, as "sha1sum" on my 4-year-old CPU can process at ~400 MB/s (~3.2 Gbit/s). But I don't bother to use tee to compute the hash in parallel because after the disk image has been written to the destination machine, I like to re-read from the destination disk to verify the data was written with integrity. So after the copy I will run sha1sum /dev/XXX on the destination machine. And while I wait for this command to complete I might as well run the same command on the source machine, in parallel. Both commands complete in about the same time so you would not be saving wall clock time. Fun fact: "openssl sha1" on a typical x86-64 machine is actually about twice faster than "sha1sum" because their code is more optimized. Another reason I don't bother to use tee to compute the hash in parallel is that it writes with a pretty small block size by default (8 kB) so for best performance you don't want to pass /dev/nvme0nX as the argument to tee, instead you would want to use fancy >(...) shell syntax to pass a file descriptor as an argument to tee which is sha1sum's stdin, then pipe the data to dd to give it the opportunity to buffer writes in 1MB block to the nvme disk: $ nc -l -p 1234 | tee >(sha1sum >s.txt) | dd bs=1M of=/dev/XXX But rescue disks sometimes have a basic shell that doesn't support fancy >(...) syntax. So in the spirit of keeping things simple I don't use tee.
  - usr1106 3y ago
    It's over 10 years ago that I had to do such operations regularly with rather unreliable networks to Southeast Asia and/or SD cards, so calculating the checksum every time on the fly was important. Instead of the "fancy" syntax I used mkfifo /tmp/cksum sha1sum /tmp/cksum & some_reader | tee /tmp/cksum | some_writer Of course under the conditions mentioned throughputs were moderate compared to what was discussed above. So I don't know how it would perform with a more performant source and target. But the important thing is that you need to pass the data through the slow endpoint only once. Disclaimer: From memory and untested now. Not.at the keyboard.

by mrb 3y ago

bayindirh 3y ago

> Your example proves me right. Your drive should be capable of 1000 MB/s but O_DIRECT reduces performance to 400 MB/s. I noted that the bus I connected the device has 500MBps bandwidth theoretical, no? To cite myself: > Target is a Samsung T7 Shield 2TB, with 1050MB/sec sustained write speed. Bus is USB 3.0 with 500MBps top speed (so I can go %50 of drive speeds). Result is 404MBps, which is fair for the bus.

mrb 3y ago
Yes USB3.0 is 500 MB/s but are you sure your bus is 3.0? It would imply your machine is 10+ years old. Most likely it's 3.1 or newer which is 1000 MB/s. And again, benchmarking /dev/zero is invalid anyway as I explained (transparent compression)
- doubled112 3y ago
  TIL they have been sneaking versions of USB in while I haven't been paying attention. Even on hardware I own. Thanks for that.
- crote 3y ago
  No, it wouldn't imply the machine is 10+ years old. Even a state-of-the-art motherboard like the Gigabyte Z790 D AX (which became available in my country today) has more USB 3 gen1 (5Gbps) ports than gen2 (10Gbps). The 5Gbps ports are just marketed as "USB 3.1" instead of "USB 3.0" these days, because USB naming is confusing and the important part is the "gen x".
  - mrb 3y ago
    I didn't mean 5 Gbps USB ports have disappeared, but rather: most machines in the last ~10 years (~8-9 years?) have some 10 Gbps ports. Therefore if he is plugging a fast SSD in a slow 5 Gbps port, my assumption was that he has no 10 Gbps port.
  - Arnavion 3y ago
    To be clear for everyone: USB 3.0, USB 3.1 gen 1, and USB 3.2 gen 1x1 are all names for the same thing, the 5Gbps speed. USB 3.1 gen 2 and USB 3.2 gen 2x1 are both names for the same thing, the 10Gbps speed. USB 3.2 gen 2x2 is the 20Gbps speed. The 3.0 / 3.1 / 3.2 are the version number of the USB specification. The 3.0 version only defined the 5Gbps speed. The 3.1 version added a 10Gbps speed, called it gen 2, and renamed the previous 5Gbps speed to gen 1. The 3.2 version added a new 20Gbps speed, called it gen 2x2, and renamed the previous 5Gbps speed to gen 1x1 and the previous 10Gbps speed to gen 2x1. There's also a 3.2 gen 1x2 10Gbps speed but I've never seen it used. The 3.2 gen 1x1 is so ubiqitous that it's also referred to as just "3.2 gen 1". And none of this is to be confused with type A vs type C ports. 3.2 gen 1x1 and 3.2 gen 2x1 can be carried by type A ports, but not 3.2 gen 2x2. 3.2 gen 1x1 and 3.2 gen 2x1 and 3.2 gen 2x2 can all be carried by type C ports. Lastly, because 3.0 and 3.1 spec versions only introduced one new speed each and because 3.2 gen 2x2 is type C-only, it's possible that a port labeled "3.1" is 3.2 gen 1x1, a type A port labeled "3.2" is 3.2 gen2x1, and a type C port labeled "3.2" is 3.2 gen 2x2. But you will have to check the manual / actual negotiation at runtime to be sure.

ufocia 3y ago

I wonder how using tee to compute the hash in parallel would affect the overall performance.

mrb 3y ago
On GigE or even 2.5G it shouldn't slow things down, as "sha1sum" on my 4-year-old CPU can process at ~400 MB/s (~3.2 Gbit/s). But I don't bother to use tee to compute the hash in parallel because after the disk image has been written to the destination machine, I like to re-read from the destination disk to verify the data was written with integrity. So after the copy I will run sha1sum /dev/XXX on the destination machine. And while I wait for this command to complete I might as well run the same command on the source machine, in parallel. Both commands complete in about the same time so you would not be saving wall clock time. Fun fact: "openssl sha1" on a typical x86-64 machine is actually about twice faster than "sha1sum" because their code is more optimized. Another reason I don't bother to use tee to compute the hash in parallel is that it writes with a pretty small block size by default (8 kB) so for best performance you don't want to pass /dev/nvme0nX as the argument to tee, instead you would want to use fancy >(...) shell syntax to pass a file descriptor as an argument to tee which is sha1sum's stdin, then pipe the data to dd to give it the opportunity to buffer writes in 1MB block to the nvme disk: $ nc -l -p 1234 | tee >(sha1sum >s.txt) | dd bs=1M of=/dev/XXX But rescue disks sometimes have a basic shell that doesn't support fancy >(...) syntax. So in the spirit of keeping things simple I don't use tee.
- usr1106 3y ago
  It's over 10 years ago that I had to do such operations regularly with rather unreliable networks to Southeast Asia and/or SD cards, so calculating the checksum every time on the fly was important. Instead of the "fancy" syntax I used mkfifo /tmp/cksum sha1sum /tmp/cksum & some_reader | tee /tmp/cksum | some_writer Of course under the conditions mentioned throughputs were moderate compared to what was discussed above. So I don't know how it would perform with a more performant source and target. But the important thing is that you need to pass the data through the slow endpoint only once. Disclaimer: From memory and untested now. Not.at the keyboard.