4 ms·

I have also discovered that certain implementations of dd have a progress printing functionality that can be used for similar purposes. You can put a "dd status

I have also discovered that certain implementations of dd have a progress printing functionality that can be used for similar purposes. You can put a "dd status=progress" in a pipeline and it will print the amount and rate of data being piped! This dd option is not as nice as pipe viewer but it's handy for when pv isn't around for some reason.

derefr 3y ago
Even if you don't pass this argument, you can poke most implementations of dd(1) with a certain POSIX signal, and they'll respond by printing a progress line. On Linux, this is SIGUSR1, and you have to use kill(1) to send it. On BSDs (incl. macOS), though, the signal dd listens for is instead called SIGINFO (which probably makes this make a lot more sense for why a process would have this response to it.) Shells/terminal emulators on these platforms emit SIGINFO when you type Ctrl+T into them! (For a lot more useful info about this behavior: https://stuff-things.net/2016/04/06/that-one-stupid-dd-trick-and-the-ballad-of-siginfo/ https://stuff-things.net/2016/04/06/that-one-stupid-dd-trick...) Bonus fact not mentioned in the above article: dd used in the middle of a pipeline will still "hear" Ctrl+T and print progress, since signals generated by a shell (think: SIGINT from Ctrl+C) are propagated to all processes in the process group started by the command. Test it yourself: cat /dev/zero | dd count=10000000 bs=1024 | cat > /dev/null
- BlackLotus89 3y ago
  Yeah bit me in the butt once on Mac OS usr1 killed dd. When I want progress of dd and didn't define status I mostly use progress now. Also works with many pther utils like cp, xz and all the usual suspects
- cycomanic 3y ago
  When did I mentioned one should always point to https://www.vidarholen.net/contents/blog/?p=479 https://www.vidarholen.net/contents/blog/?p=479 For like >99% of cases where people used dd they would have been better of using a different tool.
  - cycomanic 3y ago
    That should have read when dd is mentioned
  - derefr 3y ago
    ...except in the absolute most-common case, as the article mentions: making or restoring backups of disks. Not because dd is needed for this case, mind you. Rather just because disks are big, and so it's a lot faster to (be able to) specify a big block size. One much larger than the disk's own sector size; one much larger, even, than the current 128KiB default of cat(1)! Consider: cat's design principle is that it's reading a stream (or series of streams) serially, and writing that concatenated stream serially, as a synchronous back-and-forth process. cat(1) aims to have a low, fixed memory usage: it reads data into a buffer, then writes data from that buffer to the output. It only ever uses the one buffer, and that buffer is static — part of the program's .bss section. dd(1), meanwhile, is an asynchronous pipeline between the read and write ends. It doesn't care how much memory it uses; it'll use however much is required to do the job you've configured it to do. It doesn't mind submitting a bunch of simultaneous read requests; assembling them out-of-order into a buffer as large as you like; and then writing out the buffer with a bunch of simultaneous write requests. And it doesn't need to fully assemble the "read batch" before turning it into a "write batch" — dd(1) uses a read-write ring-buffer, so as soon as the earliest contiguous read-results are available, they get fed into the write queue. This design means that dd(1) can take strong advantage of devices like NVMe that use high internal read/write parallelism when slammed with a high IO-queue-depth of read/write requests. Which is exactly what you're doing when you specify a `bsize` to dd(1) that's higher than the disk's sector size. Try it out for yourself — do something like: for p in $(seq 5 20); do bs=$(echo 2^$p | bc) echo "bs=$bs" time dd if=/dev/nvme0n1 of=/dev/nvme1n1 bs=$bs done And, while each one is executing, do an `iostat -x` and look at the `aqu-sz` column. The speed of the copy should peak at exactly the same time that your aqu-sz hits the effective limit of the read-parallelism of the source disk or the write-pararallelism of the target disk.
Too 3y ago
For copying disks, an even more powerful tool is ddrescue. It works exactly like dd but writes its progress into a map-file that can be used to pause and resume the operation any time. It can also skip certain sectors known to be broken. Super useful if you have a big slow disk that you have to copy under unreliable conditions. Saved my ass several times with old partially broken hard drives.

derefr 3y ago

Even if you don't pass this argument, you can poke most implementations of dd(1) with a certain POSIX signal, and they'll respond by printing a progress line. On Linux, this is SIGUSR1, and you have to use kill(1) to send it. On BSDs (incl. macOS), though, the signal dd listens for is instead called SIGINFO (which probably makes this make a lot more sense for why a process would have this response to it.) Shells/terminal emulators on these platforms emit SIGINFO when you type Ctrl+T into them! (For a lot more useful info about this behavior: https://stuff-things.net/2016/04/06/that-one-stupid-dd-trick-and-the-ballad-of-siginfo/ https://stuff-things.net/2016/04/06/that-one-stupid-dd-trick...) Bonus fact not mentioned in the above article: dd used in the middle of a pipeline will still "hear" Ctrl+T and print progress, since signals generated by a shell (think: SIGINT from Ctrl+C) are propagated to all processes in the process group started by the command. Test it yourself: cat /dev/zero | dd count=10000000 bs=1024 | cat > /dev/null

BlackLotus89 3y ago
Yeah bit me in the butt once on Mac OS usr1 killed dd. When I want progress of dd and didn't define status I mostly use progress now. Also works with many pther utils like cp, xz and all the usual suspects
cycomanic 3y ago
When did I mentioned one should always point to https://www.vidarholen.net/contents/blog/?p=479 https://www.vidarholen.net/contents/blog/?p=479 For like >99% of cases where people used dd they would have been better of using a different tool.
- cycomanic 3y ago
  That should have read when dd is mentioned
- derefr 3y ago
  ...except in the absolute most-common case, as the article mentions: making or restoring backups of disks. Not because dd is needed for this case, mind you. Rather just because disks are big, and so it's a lot faster to (be able to) specify a big block size. One much larger than the disk's own sector size; one much larger, even, than the current 128KiB default of cat(1)! Consider: cat's design principle is that it's reading a stream (or series of streams) serially, and writing that concatenated stream serially, as a synchronous back-and-forth process. cat(1) aims to have a low, fixed memory usage: it reads data into a buffer, then writes data from that buffer to the output. It only ever uses the one buffer, and that buffer is static — part of the program's .bss section. dd(1), meanwhile, is an asynchronous pipeline between the read and write ends. It doesn't care how much memory it uses; it'll use however much is required to do the job you've configured it to do. It doesn't mind submitting a bunch of simultaneous read requests; assembling them out-of-order into a buffer as large as you like; and then writing out the buffer with a bunch of simultaneous write requests. And it doesn't need to fully assemble the "read batch" before turning it into a "write batch" — dd(1) uses a read-write ring-buffer, so as soon as the earliest contiguous read-results are available, they get fed into the write queue. This design means that dd(1) can take strong advantage of devices like NVMe that use high internal read/write parallelism when slammed with a high IO-queue-depth of read/write requests. Which is exactly what you're doing when you specify a `bsize` to dd(1) that's higher than the disk's sector size. Try it out for yourself — do something like: for p in $(seq 5 20); do bs=$(echo 2^$p | bc) echo "bs=$bs" time dd if=/dev/nvme0n1 of=/dev/nvme1n1 bs=$bs done And, while each one is executing, do an `iostat -x` and look at the `aqu-sz` column. The speed of the copy should peak at exactly the same time that your aqu-sz hits the effective limit of the read-parallelism of the source disk or the write-pararallelism of the target disk.

Too 3y ago

For copying disks, an even more powerful tool is ddrescue. It works exactly like dd but writes its progress into a map-file that can be used to pause and resume the operation any time. It can also skip certain sectors known to be broken. Super useful if you have a big slow disk that you have to copy under unreliable conditions. Saved my ass several times with old partially broken hard drives.