4 ms·

Not a moreutil, but I recently discovered `pv`, the pipe viewer and it's so useful. Like tqdm (Python progressbar library) but as a Unix utility. Just put it be

Not a moreutil, but I recently discovered `pv`, the pipe viewer and it's so useful. Like tqdm (Python progressbar library) but as a Unix utility. Just put it between two pipes and it'll display rate of bytes/lines Apparently it's neither a coreutil nor a moreutil. Here's an HN discussion from 2022: https://news.ycombinator.com/item?id=33244768 https://news.ycombinator.com/item?id=33244768

chlorion 3y ago
I have also discovered that certain implementations of dd have a progress printing functionality that can be used for similar purposes. You can put a "dd status=progress" in a pipeline and it will print the amount and rate of data being piped! This dd option is not as nice as pipe viewer but it's handy for when pv isn't around for some reason.
- derefr 3y ago
  Even if you don't pass this argument, you can poke most implementations of dd(1) with a certain POSIX signal, and they'll respond by printing a progress line. On Linux, this is SIGUSR1, and you have to use kill(1) to send it. On BSDs (incl. macOS), though, the signal dd listens for is instead called SIGINFO (which probably makes this make a lot more sense for why a process would have this response to it.) Shells/terminal emulators on these platforms emit SIGINFO when you type Ctrl+T into them! (For a lot more useful info about this behavior: https://stuff-things.net/2016/04/06/that-one-stupid-dd-trick-and-the-ballad-of-siginfo/ https://stuff-things.net/2016/04/06/that-one-stupid-dd-trick...) Bonus fact not mentioned in the above article: dd used in the middle of a pipeline will still "hear" Ctrl+T and print progress, since signals generated by a shell (think: SIGINT from Ctrl+C) are propagated to all processes in the process group started by the command. Test it yourself: cat /dev/zero | dd count=10000000 bs=1024 | cat > /dev/null
  - BlackLotus89 3y ago
    Yeah bit me in the butt once on Mac OS usr1 killed dd. When I want progress of dd and didn't define status I mostly use progress now. Also works with many pther utils like cp, xz and all the usual suspects
  - cycomanic 3y ago
    When did I mentioned one should always point to https://www.vidarholen.net/contents/blog/?p=479 https://www.vidarholen.net/contents/blog/?p=479 For like >99% of cases where people used dd they would have been better of using a different tool.
    - cycomanic 3y ago
      That should have read when dd is mentioned
    - derefr 3y ago
      ...except in the absolute most-common case, as the article mentions: making or restoring backups of disks. Not because dd is needed for this case, mind you. Rather just because disks are big, and so it's a lot faster to (be able to) specify a big block size. One much larger than the disk's own sector size; one much larger, even, than the current 128KiB default of cat(1)! Consider: cat's design principle is that it's reading a stream (or series of streams) serially, and writing that concatenated stream serially, as a synchronous back-and-forth process. cat(1) aims to have a low, fixed memory usage: it reads data into a buffer, then writes data from that buffer to the output. It only ever uses the one buffer, and that buffer is static — part of the program's .bss section. dd(1), meanwhile, is an asynchronous pipeline between the read and write ends. It doesn't care how much memory it uses; it'll use however much is required to do the job you've configured it to do. It doesn't mind submitting a bunch of simultaneous read requests; assembling them out-of-order into a buffer as large as you like; and then writing out the buffer with a bunch of simultaneous write requests. And it doesn't need to fully assemble the "read batch" before turning it into a "write batch" — dd(1) uses a read-write ring-buffer, so as soon as the earliest contiguous read-results are available, they get fed into the write queue. This design means that dd(1) can take strong advantage of devices like NVMe that use high internal read/write parallelism when slammed with a high IO-queue-depth of read/write requests. Which is exactly what you're doing when you specify a `bsize` to dd(1) that's higher than the disk's sector size. Try it out for yourself — do something like: for p in $(seq 5 20); do bs=$(echo 2^$p | bc) echo "bs=$bs" time dd if=/dev/nvme0n1 of=/dev/nvme1n1 bs=$bs done And, while each one is executing, do an `iostat -x` and look at the `aqu-sz` column. The speed of the copy should peak at exactly the same time that your aqu-sz hits the effective limit of the read-parallelism of the source disk or the write-pararallelism of the target disk.
- Too 3y ago
  For copying disks, an even more powerful tool is ddrescue. It works exactly like dd but writes its progress into a map-file that can be used to pause and resume the operation any time. It can also skip certain sectors known to be broken. Super useful if you have a big slow disk that you have to copy under unreliable conditions. Saved my ass several times with old partially broken hard drives.
derefr 3y ago
You can also use pv as you would use cat, e.g. pv file.tar.gz.part1 file.tar.gz.part2 | tar -x -z Just like cat, pv used this way will stream out the concatenation of the passed-in file paths; but it will also add up the sizes of these files and use them to calculate the total stream size, i.e. the divisor for its displayed progress percentage.
matrss 3y ago
> Like tqdm (Python progressbar library) but as a Unix utility. FYI: tqdm can be used in a shell pipeline as well. It's documented (at least) in their readme: https://github.com/tqdm/tqdm#module https://github.com/tqdm/tqdm#module
genman 3y ago
It is a really incredible anxiety reducing small tool when you have to transfer large files.
LeoPanthera 3y ago
If you have a long running copy process running but forget to enable progress, you can use the "progress" utility to show the progress of something that is already running. It supports: cp mv dd tar bsdtar cat rsync scp grep fgrep egrep cut sort cksum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum adb gzip gunzip bzip2 bunzip2 xz unxz lzma unlzma 7z 7za zip unzip zcat bzcat lzcat coreutils split gpg gcp gmv
- natmaka 3y ago
  Generic way: https://github.com/ebikt/progress.pl https://github.com/ebikt/progress.pl

chlorion 3y ago

I have also discovered that certain implementations of dd have a progress printing functionality that can be used for similar purposes. You can put a "dd status=progress" in a pipeline and it will print the amount and rate of data being piped! This dd option is not as nice as pipe viewer but it's handy for when pv isn't around for some reason.

derefr 3y ago
Even if you don't pass this argument, you can poke most implementations of dd(1) with a certain POSIX signal, and they'll respond by printing a progress line. On Linux, this is SIGUSR1, and you have to use kill(1) to send it. On BSDs (incl. macOS), though, the signal dd listens for is instead called SIGINFO (which probably makes this make a lot more sense for why a process would have this response to it.) Shells/terminal emulators on these platforms emit SIGINFO when you type Ctrl+T into them! (For a lot more useful info about this behavior: https://stuff-things.net/2016/04/06/that-one-stupid-dd-trick-and-the-ballad-of-siginfo/ https://stuff-things.net/2016/04/06/that-one-stupid-dd-trick...) Bonus fact not mentioned in the above article: dd used in the middle of a pipeline will still "hear" Ctrl+T and print progress, since signals generated by a shell (think: SIGINT from Ctrl+C) are propagated to all processes in the process group started by the command. Test it yourself: cat /dev/zero | dd count=10000000 bs=1024 | cat > /dev/null
- BlackLotus89 3y ago
  Yeah bit me in the butt once on Mac OS usr1 killed dd. When I want progress of dd and didn't define status I mostly use progress now. Also works with many pther utils like cp, xz and all the usual suspects
- cycomanic 3y ago
  When did I mentioned one should always point to https://www.vidarholen.net/contents/blog/?p=479 https://www.vidarholen.net/contents/blog/?p=479 For like >99% of cases where people used dd they would have been better of using a different tool.
  - cycomanic 3y ago
    That should have read when dd is mentioned
  - derefr 3y ago
    ...except in the absolute most-common case, as the article mentions: making or restoring backups of disks. Not because dd is needed for this case, mind you. Rather just because disks are big, and so it's a lot faster to (be able to) specify a big block size. One much larger than the disk's own sector size; one much larger, even, than the current 128KiB default of cat(1)! Consider: cat's design principle is that it's reading a stream (or series of streams) serially, and writing that concatenated stream serially, as a synchronous back-and-forth process. cat(1) aims to have a low, fixed memory usage: it reads data into a buffer, then writes data from that buffer to the output. It only ever uses the one buffer, and that buffer is static — part of the program's .bss section. dd(1), meanwhile, is an asynchronous pipeline between the read and write ends. It doesn't care how much memory it uses; it'll use however much is required to do the job you've configured it to do. It doesn't mind submitting a bunch of simultaneous read requests; assembling them out-of-order into a buffer as large as you like; and then writing out the buffer with a bunch of simultaneous write requests. And it doesn't need to fully assemble the "read batch" before turning it into a "write batch" — dd(1) uses a read-write ring-buffer, so as soon as the earliest contiguous read-results are available, they get fed into the write queue. This design means that dd(1) can take strong advantage of devices like NVMe that use high internal read/write parallelism when slammed with a high IO-queue-depth of read/write requests. Which is exactly what you're doing when you specify a `bsize` to dd(1) that's higher than the disk's sector size. Try it out for yourself — do something like: for p in $(seq 5 20); do bs=$(echo 2^$p | bc) echo "bs=$bs" time dd if=/dev/nvme0n1 of=/dev/nvme1n1 bs=$bs done And, while each one is executing, do an `iostat -x` and look at the `aqu-sz` column. The speed of the copy should peak at exactly the same time that your aqu-sz hits the effective limit of the read-parallelism of the source disk or the write-pararallelism of the target disk.
Too 3y ago
For copying disks, an even more powerful tool is ddrescue. It works exactly like dd but writes its progress into a map-file that can be used to pause and resume the operation any time. It can also skip certain sectors known to be broken. Super useful if you have a big slow disk that you have to copy under unreliable conditions. Saved my ass several times with old partially broken hard drives.

derefr 3y ago

You can also use pv as you would use cat, e.g. pv file.tar.gz.part1 file.tar.gz.part2 | tar -x -z Just like cat, pv used this way will stream out the concatenation of the passed-in file paths; but it will also add up the sizes of these files and use them to calculate the total stream size, i.e. the divisor for its displayed progress percentage.

matrss 3y ago

> Like tqdm (Python progressbar library) but as a Unix utility. FYI: tqdm can be used in a shell pipeline as well. It's documented (at least) in their readme: https://github.com/tqdm/tqdm#module https://github.com/tqdm/tqdm#module

genman 3y ago

It is a really incredible anxiety reducing small tool when you have to transfer large files.

LeoPanthera 3y ago

If you have a long running copy process running but forget to enable progress, you can use the "progress" utility to show the progress of something that is already running. It supports: cp mv dd tar bsdtar cat rsync scp grep fgrep egrep cut sort cksum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum adb gzip gunzip bzip2 bunzip2 xz unxz lzma unlzma 7z 7za zip unzip zcat bzcat lzcat coreutils split gpg gcp gmv

natmaka 3y ago
Generic way: https://github.com/ebikt/progress.pl https://github.com/ebikt/progress.pl