3 ms·

cp queries the preferred block size of the destination file in 'struct stat', and has specific tweaks for certain filesystems. As far as I can tell, dd does not

cp queries the preferred block size of the destination file in 'struct stat', and has specific tweaks for certain filesystems. As far as I can tell, dd does not do this as it calls through to 'write' directly. In any case, the tests in ioblksize.h indicate that bs=4M is far too large and may perform worse than the default for cp/cat (128KiB). There is a script there that should clear things up for more modern systems. The point about fdatasync is superfluous as you can run 'sync' yourself, or unmount the filesystem.

NoZebra120vClip 3y ago
This example was performed on the block-special device, so there is no filesystem. dd uses a default block size of 512 bytes, according to the manual page: calling write(2) directly means you need to choose a buffer of some size. "bs=" sets both input and output block sizes, which probably isn't the best idea in this case. Block sizes are a tricky subject: https://utcc.utoronto.ca/~cks/space/blog/unix/StatfsPeculiarities https://utcc.utoronto.ca/~cks/space/blog/unix/StatfsPeculiar... https://utcc.utoronto.ca/~cks/space/blog/tech/SSDsAnd4KSectorsII https://utcc.utoronto.ca/~cks/space/blog/tech/SSDsAnd4KSecto...
- rollcat 3y ago
  I've been re-implementing a bunch of coreutils as an exercise, and got stuck on dd input/output block sizes, AND disk/partition block sizes for a while. (As far as I understand it, for dd I need a ring buffer the size of max(ibs, obs), and then some moderately clever book-keeping to know when to trigger the next read/write, perhaps with code specific to ibs>obs, ibs<obs, etc; partitioning on the other hand is plainly stupid, there's decades of hardware and software just lying to each other and nothing makes sense.) Thank you and everyone else in this thread for the know-how and references! I would like to eventually write an article (or at least heavily commented source) to hopefully explain all this nonsense for other people like me.
mort96 3y ago
TFA mentions that you can use sync after cp to do the same thing as fdatasync. You can't "unmount the file system" because you're writing directly to the block device, the thumb drive isn't mounted.
- formerly_proven 3y ago
  Syncing as you go (ideally asynchronously) when you have to sync anyway (like when you're writing an image to a thumb drive, or write to NFS) has the big advantage that you don't end up saying "I'm done writing all data" and then blocking for five minutes waiting for the kernel to flush a couple gigs to disk.
  - NoZebra120vClip 3y ago
    For removable media, I believe that finishing up with "eject" would be the safest way to ensure that all writes were committed to the hardware.
    - mort96 3y ago
      You can't "unmount the file system" because you're writing directly to the block device, the thumb drive isn't mounted. ("Eject" is macOS's term for "unmount filesystem")
      - NoZebra120vClip 3y ago
        Distrowatch.com is a site promoting use of Linux and BSD. "eject" is a Linux command which attempts to safely disconnect removable media. https://manpages.ubuntu.com/manpages/noble/en/man1/eject.1.html https://manpages.ubuntu.com/manpages/noble/en/man1/eject.1.h... TFA doesn't mention filesystems at all, sort of jumps in where we find the block device. Things could become messy if the device were mounted while the copy is attempted.

NoZebra120vClip 3y ago

This example was performed on the block-special device, so there is no filesystem. dd uses a default block size of 512 bytes, according to the manual page: calling write(2) directly means you need to choose a buffer of some size. "bs=" sets both input and output block sizes, which probably isn't the best idea in this case. Block sizes are a tricky subject: https://utcc.utoronto.ca/~cks/space/blog/unix/StatfsPeculiarities https://utcc.utoronto.ca/~cks/space/blog/unix/StatfsPeculiar... https://utcc.utoronto.ca/~cks/space/blog/tech/SSDsAnd4KSectorsII https://utcc.utoronto.ca/~cks/space/blog/tech/SSDsAnd4KSecto...

rollcat 3y ago
I've been re-implementing a bunch of coreutils as an exercise, and got stuck on dd input/output block sizes, AND disk/partition block sizes for a while. (As far as I understand it, for dd I need a ring buffer the size of max(ibs, obs), and then some moderately clever book-keeping to know when to trigger the next read/write, perhaps with code specific to ibs>obs, ibs<obs, etc; partitioning on the other hand is plainly stupid, there's decades of hardware and software just lying to each other and nothing makes sense.) Thank you and everyone else in this thread for the know-how and references! I would like to eventually write an article (or at least heavily commented source) to hopefully explain all this nonsense for other people like me.

mort96 3y ago

TFA mentions that you can use sync after cp to do the same thing as fdatasync. You can't "unmount the file system" because you're writing directly to the block device, the thumb drive isn't mounted.

formerly_proven 3y ago
Syncing as you go (ideally asynchronously) when you have to sync anyway (like when you're writing an image to a thumb drive, or write to NFS) has the big advantage that you don't end up saying "I'm done writing all data" and then blocking for five minutes waiting for the kernel to flush a couple gigs to disk.
- NoZebra120vClip 3y ago
  For removable media, I believe that finishing up with "eject" would be the safest way to ensure that all writes were committed to the hardware.
  - mort96 3y ago
    You can't "unmount the file system" because you're writing directly to the block device, the thumb drive isn't mounted. ("Eject" is macOS's term for "unmount filesystem")
    - NoZebra120vClip 3y ago
      Distrowatch.com is a site promoting use of Linux and BSD. "eject" is a Linux command which attempts to safely disconnect removable media. https://manpages.ubuntu.com/manpages/noble/en/man1/eject.1.html https://manpages.ubuntu.com/manpages/noble/en/man1/eject.1.h... TFA doesn't mention filesystems at all, sort of jumps in where we find the block device. Things could become messy if the device were mounted while the copy is attempted.