9 ms·

This use of dd may cause corruption! You need iflag=fullblock to ensure it doesn't truncate any blocks, and (at the risk of cargo-culting) conv=sync doesn't hur

This use of dd may cause corruption! You need iflag=fullblock to ensure it doesn't truncate any blocks, and (at the risk of cargo-culting) conv=sync doesn't hurt as well. I prefer to just nc -l -p 1234 > /dev/nvme0nX.

mrb 3y ago
Partial reads won't corrupt the data. Dd will issue other read() until 1MB of data is buffered. The iflag=fullblock is only useful when counting or skipping bytes or doing direct I/O. See line 1647: https://github.com/coreutils/coreutils/blob/master/src/dd.c#L1647 https://github.com/coreutils/coreutils/blob/master/src/dd.c#...
adrian_b 3y ago
According to the documentation of dd, "iflag=fullblock" is required only when dd is used with the "count=" option. Otherwise, i.e. when dd has to read the entire input file because there is no "count=" option, "iflag=fullblock" does not have any documented effect. From "info dd": "If short reads occur, as could be the case when reading from a pipe for example, ‘iflag=fullblock’ ensures that ‘count=’ counts complete input blocks rather than input read operations."
- tripflag 3y ago
  Thank you for the correction -- it is likely that I did use count= when I ran into this some 10 years ago (and been paranoid about ever since). I thought a chunk of data was missing in the middle of the output file, causing everything after that to be shifted over, but I'm probably misremembering.
  - fsckboy 3y ago
    thank you for bringing it up! i wasn't even aware of this potential problem, and I use bs= count= and skip= seek= (sk"i"p means "input") through pipes across the net aaaallll the time for decades. it pretty much seems iflag=fullblock is a requirement if you want the counts to work, even though the failure times might be rare
dezgeg 3y ago
Isn't `nc -l -p 1234 > /dev/nvme0nX` working by accident (relying on that netcat is buffering its output in multiples of disk block size)?
- mrb 3y ago
  Your exact command works reliably but is inefficient. And it works by design, not accident. For starters, the default block size in most netcat implementations is tiny like 4 kB or less. So there is a higher CPU and I/O overhead. And if netcat does a partial or small read less than 4 kB, when it writes the partial block to the nvme disk, the kernel would take care of reading a full 4kB block from the nvme disk, updating it with the partial data block, and rewriting the full 4kB block to the disk, which is what makes it work, albeit inefficiently.
- jasomill 3y ago
  No — the kernel buffers non-O_DIRECT writes to block devices to ensure correctness. Larger writes will be more efficient, however, if only due to reduced system call overhead. While not necessary when writing an image with the correct block size for the target device, even partial block overwrites work fine: # yes | head -c 512 > foo # losetup /dev/loop0 foo # echo 'Ham and jam and Spam a lot.' | dd bs=5 of=/dev/loop0 5+1 records in 5+1 records out 28 bytes copied, 0.000481667 s, 58.1 kB/s # hexdump -C /dev/loop0 00000000 48 61 6d 20 61 6e 64 20 6a 61 6d 20 61 6e 64 20 |Ham and jam and | 00000010 53 70 61 6d 20 61 20 6c 6f 74 2e 0a 79 0a 79 0a |Spam a lot..y.y.| 00000020 79 0a 79 0a 79 0a 79 0a 79 0a 79 0a 79 0a 79 0a |y.y.y.y.y.y.y.y.| * 00000200 Partial block overwrites may (= will, unless the block to be overwritten is in the kernel's buffer cache) require a read/modify/write operation, but this is transparent to the application. Finally, note that this applies to most block devices, but tape devices work differently: partial overwrites are not supported, and, in variable block mode, the size of individual write calls determines the resulting tape block sizes.
  - dezgeg 3y ago
    Somehow I had thought even in buffered mode the kernel would only accept block-aligned and sized I/O. TIL.
M95D 3y ago
I would include bs=1M and oflag=direct for some extra speed.
deleted 3y ago
[deleted]

mrb 3y ago

Partial reads won't corrupt the data. Dd will issue other read() until 1MB of data is buffered. The iflag=fullblock is only useful when counting or skipping bytes or doing direct I/O. See line 1647: https://github.com/coreutils/coreutils/blob/master/src/dd.c#L1647 https://github.com/coreutils/coreutils/blob/master/src/dd.c#...

adrian_b 3y ago

According to the documentation of dd, "iflag=fullblock" is required only when dd is used with the "count=" option. Otherwise, i.e. when dd has to read the entire input file because there is no "count=" option, "iflag=fullblock" does not have any documented effect. From "info dd": "If short reads occur, as could be the case when reading from a pipe for example, ‘iflag=fullblock’ ensures that ‘count=’ counts complete input blocks rather than input read operations."

tripflag 3y ago
Thank you for the correction -- it is likely that I did use count= when I ran into this some 10 years ago (and been paranoid about ever since). I thought a chunk of data was missing in the middle of the output file, causing everything after that to be shifted over, but I'm probably misremembering.
- fsckboy 3y ago
  thank you for bringing it up! i wasn't even aware of this potential problem, and I use bs= count= and skip= seek= (sk"i"p means "input") through pipes across the net aaaallll the time for decades. it pretty much seems iflag=fullblock is a requirement if you want the counts to work, even though the failure times might be rare

dezgeg 3y ago

Isn't `nc -l -p 1234 > /dev/nvme0nX` working by accident (relying on that netcat is buffering its output in multiples of disk block size)?

mrb 3y ago
Your exact command works reliably but is inefficient. And it works by design, not accident. For starters, the default block size in most netcat implementations is tiny like 4 kB or less. So there is a higher CPU and I/O overhead. And if netcat does a partial or small read less than 4 kB, when it writes the partial block to the nvme disk, the kernel would take care of reading a full 4kB block from the nvme disk, updating it with the partial data block, and rewriting the full 4kB block to the disk, which is what makes it work, albeit inefficiently.
jasomill 3y ago
No — the kernel buffers non-O_DIRECT writes to block devices to ensure correctness. Larger writes will be more efficient, however, if only due to reduced system call overhead. While not necessary when writing an image with the correct block size for the target device, even partial block overwrites work fine: # yes | head -c 512 > foo # losetup /dev/loop0 foo # echo 'Ham and jam and Spam a lot.' | dd bs=5 of=/dev/loop0 5+1 records in 5+1 records out 28 bytes copied, 0.000481667 s, 58.1 kB/s # hexdump -C /dev/loop0 00000000 48 61 6d 20 61 6e 64 20 6a 61 6d 20 61 6e 64 20 |Ham and jam and | 00000010 53 70 61 6d 20 61 20 6c 6f 74 2e 0a 79 0a 79 0a |Spam a lot..y.y.| 00000020 79 0a 79 0a 79 0a 79 0a 79 0a 79 0a 79 0a 79 0a |y.y.y.y.y.y.y.y.| * 00000200 Partial block overwrites may (= will, unless the block to be overwritten is in the kernel's buffer cache) require a read/modify/write operation, but this is transparent to the application. Finally, note that this applies to most block devices, but tape devices work differently: partial overwrites are not supported, and, in variable block mode, the size of individual write calls determines the resulting tape block sizes.
- dezgeg 3y ago
  Somehow I had thought even in buffered mode the kernel would only accept block-aligned and sized I/O. TIL.

M95D 3y ago

I would include bs=1M and oflag=direct for some extra speed.

deleted 3y ago

[deleted]