3 ms·
...except in the absolute most-common case, as the article mentions: making or restoring backups of disks. Not because dd is needed for this case, mind you. Ra
by derefr 3y ago
...except in the absolute most-common case, as the article mentions: making or restoring backups of disks.
Not because dd is needed for this case, mind you. Rather just because disks are big, and so it's a lot faster to (be able to) specify a big block size. One much larger than the disk's own sector size; one much larger, even, than the current 128KiB default of cat(1)!
Consider: cat's design principle is that it's reading a stream (or series of streams) serially, and writing that concatenated stream serially, as a synchronous back-and-forth process. cat(1) aims to have a low, fixed memory usage: it reads data into a buffer, then writes data from that buffer to the output. It only ever uses the one buffer, and that buffer is static — part of the program's .bss section.
dd(1), meanwhile, is an asynchronous pipeline between the read and write ends. It doesn't care how much memory it uses; it'll use however much is required to do the job you've configured it to do. It doesn't mind submitting a bunch of simultaneous read requests; assembling them out-of-order into a buffer as large as you like; and then writing out the buffer with a bunch of simultaneous write requests. And it doesn't need to fully assemble the "read batch" before turning it into a "write batch" — dd(1) uses a read-write ring-buffer, so as soon as the earliest contiguous read-results are available, they get fed into the write queue.
This design means that dd(1) can take strong advantage of devices like NVMe that use high internal read/write parallelism when slammed with a high IO-queue-depth of read/write requests. Which is exactly what you're doing when you specify a `bsize` to dd(1) that's higher than the disk's sector size.
Try it out for yourself — do something like:
for p in $(seq 5 20); do
bs=$(echo 2^$p | bc)
echo "bs=$bs"
time dd if=/dev/nvme0n1 of=/dev/nvme1n1 bs=$bs
done
And, while each one is executing, do an `iostat -x` and look at the `aqu-sz` column. The speed of the copy should peak at exactly the same time that your aqu-sz hits the effective limit of the read-parallelism of the source disk or the write-pararallelism of the target disk.