4 ms·

As a sysadmin, I'd rather use NVMe TCP or Clonezilla to do a slow write rather than trying to go 5% faster with more moving parts and chance to corrupt my drive

As a sysadmin, I'd rather use NVMe TCP or Clonezilla to do a slow write rather than trying to go 5% faster with more moving parts and chance to corrupt my drive in the process. Plus, a it'd be well deserved coffee break. Considering I'd be going at GigE speeds at best, I'd add "oflag=direct" to bypass caching on the target. A bog standard NVMe can write >300MBps unhindered, so trying to cache is moot. Lastly, parted can do partition resizing, but given the user is not a power user to begin with, it's just me nitpicking. Nice post otherwise.

iforgotpassword 3y ago
I don't see how you can consider the nvme over tcp version less moving parts. dd is installed on every system, and if you don't have nc you can still use ssh and sacrifice a bit of performance. dd if=/dev/foo | ssh dest@bar "cat > /dev/moo"
- bayindirh 3y ago
  NVMe over TCP encapsulates and shows me the remote device as is. Just a block device. I just copy that block device with "dd", that's all. It's just a dumb pipe encapsulated with TCP, which is already battle tested enough. Moreover, if I have fatter pipe, I can tune dd for better performance with a single command.
  - darkwater 3y ago
    netcat encapsulates data just the same (although in a different manner), and it's even more battle-tested. NVMe over TCP use case is to actually use the remote disk over the network as it were local. If you just need to dump a whole disk like in the article, dd+netcat (or even just netcat, as someone pointed out) will work just the same.
  - iforgotpassword 3y ago
    Nvme over TCP encapsulates the entire nvme protocol in TCP, which is way more complex than just sending the raw data. It's the opposite of "a dumb pipe encapsulated in tcp", this is what the netccat approach would be. Heck if you insist on representing the drive as a block device on the remote side you could just as well use NBD, which is just about as many moving parts as nvme over tcp but still a simpler protocol.
mrb 3y ago
NVMe/TCP or Clonezilla are vastly more moving parts and chances to mess up the options, compared to dd. In fact, the author's solution exposes his NVMe to unauthenticated remote write access by any number of clients(!) By comparison, the dd on the source is read-only, and the dd on the destination only accepts the first connection (yours) and no one else on the network can write to the disk. I strongly recommend against oflag=direct as in this specific use case it will always degrade performance. Read the O_DIRECT section in open(2). Or try it. Basically using oflag=direct locks the buffer so dd will have to wait for the block to be written by the kernel to disk until it can start reading the data again to fill the buffer with the next block, thereby reducing performance.
- bayindirh 3y ago
  > the author's solution exposes his NVMe to unauthenticated remote write access by any number of clients(!) I won't be bothered in a home network. > Clonezilla are vastly more moving parts ...and one of these moving parts is image integrity and write integrity verification, allowing byte-by-byte integrity during imaging and after write. > I strongly recommend against oflag=direct as in this... [snipped for brevity] Unless you're getting a bottom of the barrel NVMe, all of them have DRAM caches and do their own write caching independent of O_DIRECT, which only bypasses OS caches. Unless the pipe you have has higher throughput than your drive, caching in the storage device's controller ensures optimal write speeds. I can hit theoretical maximum write speeds of all my SSDs (internal or external) with O_DIRECT. When the pipe is fatter or the device can't sustain that speeds, things go south, but this is why we have knobs. When you don't use O_DIRECT in these cases, you see initial speed surge maybe, but total time doesn't reduce. TL;DR: When you're getting your data at 100MBps at most, using O_DIRECT on an SSD with 1GBps write speeds doesn't affect anything. You're not saturating anything on the pipe. Just did a small test: dd if=/dev/zero of=test.file bs=1024kB count=3072 oflag=direct status=progress 2821120000 bytes (2.8 GB, 2.6 GiB) copied, 7 s, 403 MB/s 3072+0 records in 3072+0 records out 3145728000 bytes (3.1 GB, 2.9 GiB) copied, 7.79274 s, 404 MB/s Target is a Samsung T7 Shield 2TB, with 1050MB/sec sustained write speed. Bus is USB 3.0 with 500MBps top speed (so I can go %50 of drive speeds). Result is 404MBps, which is fair for the bus. If the drive didn't have its own cache, caching on the OS side would have more profound effect since I can queue more writes to device and pool them at RAM.

iforgotpassword 3y ago

I don't see how you can consider the nvme over tcp version less moving parts. dd is installed on every system, and if you don't have nc you can still use ssh and sacrifice a bit of performance. dd if=/dev/foo | ssh dest@bar "cat > /dev/moo"

bayindirh 3y ago
NVMe over TCP encapsulates and shows me the remote device as is. Just a block device. I just copy that block device with "dd", that's all. It's just a dumb pipe encapsulated with TCP, which is already battle tested enough. Moreover, if I have fatter pipe, I can tune dd for better performance with a single command.
- darkwater 3y ago
  netcat encapsulates data just the same (although in a different manner), and it's even more battle-tested. NVMe over TCP use case is to actually use the remote disk over the network as it were local. If you just need to dump a whole disk like in the article, dd+netcat (or even just netcat, as someone pointed out) will work just the same.
- iforgotpassword 3y ago
  Nvme over TCP encapsulates the entire nvme protocol in TCP, which is way more complex than just sending the raw data. It's the opposite of "a dumb pipe encapsulated in tcp", this is what the netccat approach would be. Heck if you insist on representing the drive as a block device on the remote side you could just as well use NBD, which is just about as many moving parts as nvme over tcp but still a simpler protocol.

mrb 3y ago

NVMe/TCP or Clonezilla are vastly more moving parts and chances to mess up the options, compared to dd. In fact, the author's solution exposes his NVMe to unauthenticated remote write access by any number of clients(!) By comparison, the dd on the source is read-only, and the dd on the destination only accepts the first connection (yours) and no one else on the network can write to the disk. I strongly recommend against oflag=direct as in this specific use case it will always degrade performance. Read the O_DIRECT section in open(2). Or try it. Basically using oflag=direct locks the buffer so dd will have to wait for the block to be written by the kernel to disk until it can start reading the data again to fill the buffer with the next block, thereby reducing performance.

bayindirh 3y ago
> the author's solution exposes his NVMe to unauthenticated remote write access by any number of clients(!) I won't be bothered in a home network. > Clonezilla are vastly more moving parts ...and one of these moving parts is image integrity and write integrity verification, allowing byte-by-byte integrity during imaging and after write. > I strongly recommend against oflag=direct as in this... [snipped for brevity] Unless you're getting a bottom of the barrel NVMe, all of them have DRAM caches and do their own write caching independent of O_DIRECT, which only bypasses OS caches. Unless the pipe you have has higher throughput than your drive, caching in the storage device's controller ensures optimal write speeds. I can hit theoretical maximum write speeds of all my SSDs (internal or external) with O_DIRECT. When the pipe is fatter or the device can't sustain that speeds, things go south, but this is why we have knobs. When you don't use O_DIRECT in these cases, you see initial speed surge maybe, but total time doesn't reduce. TL;DR: When you're getting your data at 100MBps at most, using O_DIRECT on an SSD with 1GBps write speeds doesn't affect anything. You're not saturating anything on the pipe. Just did a small test: dd if=/dev/zero of=test.file bs=1024kB count=3072 oflag=direct status=progress 2821120000 bytes (2.8 GB, 2.6 GiB) copied, 7 s, 403 MB/s 3072+0 records in 3072+0 records out 3145728000 bytes (3.1 GB, 2.9 GiB) copied, 7.79274 s, 404 MB/s Target is a Samsung T7 Shield 2TB, with 1050MB/sec sustained write speed. Bus is USB 3.0 with 500MBps top speed (so I can go %50 of drive speeds). Result is 404MBps, which is fair for the bus. If the drive didn't have its own cache, caching on the OS side would have more profound effect since I can queue more writes to device and pool them at RAM.