7 ms·

Pipe Viewer

183 points by 0x45696e6172 4y ago

trabant00 4y ago
I've used it mostly to measure events per second with something like: tail -f /some/log | grep something | pv -lr > /dev/null or tcpdump expression | pv -lr > /dev/null
JayGuerette 4y ago
pv is a great tool. One of it's lesser known features is throttling; transfer a file without dominating your bandwidth: pv -L 200K < bigfile.iso | ssh somehost 'cat > bigfile.iso' Complete with a progress bar, speed, and ETA.
- smcl 4y ago
  Oh damn that's neat I never thought to use `ssh` directly when transferring a file, I always used `scp bigfile.iso name@server.org:path/in/destination`
  - fbergen 4y ago
    Also see `scp -l 200 bigfile.iso name@server.org:path/in/destination` from man page: -l limit Limits the used bandwidth, specified in Kbit/s.
    - MayeulC 4y ago
      Also you probably shouldn't use scp. rsync and sftp have mostly the same semantics. rsync --bwlimit=2OOK bigfile.iso name@server.org:path/in/destination sftp -l 200 bigfile.iso name@server.org:path/in/destination Although it seems that scp is becoming a wrapper around sftp these days: https://www.redhat.com/en/blog/openssh-scp-deprecation-rhel-9-what-you-need-know https://www.redhat.com/en/blog/openssh-scp-deprecation-rhel-... https://news.ycombinator.com/item?id=25005567 https://news.ycombinator.com/item?id=25005567
  - tyingq 4y ago
    A similar trick that's nice is piping tar through ssh. Handy if you don't have rsync or something better around. Even handy for one file, since it preserves permissions, etc. tar -cf - some/dir | ssh remote 'cd /place/to/go && tar -xvf -'
    - Twirrim 4y ago
      I love this trick. I was dealing with some old solaris boxes something like 15 years ago when I learned you could do this. I couldn't rsync, and had started off SCP'ing hundreds of thousands of files across but it was going to take an insane length of time. Asked one of the other sysadmins if they knew a better way and they pointed out you can pipe stuff in to ssh for the other side too. Every now and then this technique proves useful in unexpected ways :)
- dspillett 4y ago
  Similarly, though useful less often these days, using -B/--buffer-size to increase the amount that it can buffer. If reading data from traditional hard drives, piping that data through some process, and writing the result back to the same drives, this option can increase throughput significantly by reducing head movements. It can help on other storage systems too, but usually not so much so.
est 4y ago
pv was the tool when I discovered sometimes the VPS have only 10Gbps memory copy speed.
cwillu 4y ago
pv -d $(pidof xz):1 is great for when you realize too late that something is slow enough that you want a progress indication, and definitely do not want to restart from scratch.
- xuhu 4y ago
  How `pv -d` work ? Does it use perf probes or attach to the target PID ?
  - remram 4y ago
    It finds the file using /proc/<pid>/fd/<num> and watches its size grow. It doesn't work with pipes, devices, a file being overwritten (not appended to), or anything whose size doesn't grow.
    - Jenda_ 4y ago
      It does, even for reading (cat /dev/nvme0n1 > /dev/null). As per changelog: 1.5.2 - 10 February 2014 allow --watchfd to look at block devices You can see the position in /proc/<pid>/fdinfo/<num>
  - cwillu 4y ago
    It appears to monitor the contents of /proc/‹pid›/fdinfo/‹fd›
- dspillett 4y ago
  Another good option for that, which works in a number of other useful circumstances too, is progress: https://github.com/Xfennec/progress https://github.com/Xfennec/progress
londons_explore 4y ago
It would be nice to indicate if the upstream or the downstream is the 'limiting' factor in speed. Ie. within pv, is it the reading the input stream or the writing the output stream that is blocking most of the time?
- ketralnis 4y ago
  It's open source, be the change you want to see in the world
  - kotlin2 4y ago
    Having maintained an open source library, it’s actually really helpful to see features people want. Not everyone needs to contribute directly to the code base. User feedback is valuable, too.
  - ranger_danger 4y ago
    Unfortunately not everyone is a developer.
  - deleted 4y ago
    [deleted]
- senjin 4y ago
  This would be a genius addition
- bingaling 4y ago
  it's instantaneous, but the -T (transfer buffer % full display) is sometimes useful for that. (0% full -> source limited, 100% full -> sink limited)
  - Twirrim 4y ago
    Oh wow, I'd completely missed that -T flag. That's some useful data. Thanks for mentioning it!
- tanelpoder 4y ago
michaelmior 4y ago
Probably my favorite non-POSIX tool that I insert into my pipelines whenever anything takes more than a few second. I find it super helpful to avoid premature optimization. If I can quickly see that my hacked together pipeline will run in a few minutes and I only ever need to do that once, I'll probably just let it finish. If it's going to take a few hours, I might decide it's worth optimizing. It also helps me optimize my time. If something is going to finish in a few minutes, I probably won't context switch to another major task. However, if something is going to take a few hours then I'll probably switch to work on something different knowing approximately when I can go back and check on results.
- systems_glitch 4y ago
  Same, one of the first utilities I install on a new system.
derefr 4y ago
As a person who runs a lot of ETL-like commands at work, I never find myself using pv(1). I love the idea of it, but for the commands I most want to measure progress of, they always seem to be either: 1. things where I'd be paranoid about pv(1) itself becoming the bottleneck in the pipeline — e.g. dd(1) of large disks where I've explicitly set a large blocksize and set conv=idirect/odirect, to optimize throughput. 2. things where the program has some useful cleverness I rely on that requires being fed by a named file argument, but behaves a lot less intelligently when being fed from stdin — e.g. feeding SQL files into psql(1). 3. things where the program, even while writing to stdout, also produces useful "sampled progress" informational messages on stderr, which I'd like to see; where pv(1) and this output logging would fight each-other if both were running. 4. things where there's no clean place to insert pv(1) anyway — mostly, this comes up for any command that manages jobs itself in order to do things in parallel, e.g. any object-storage-client mass-copy, or any parallel-rsync script. (You'd think these programs would also report global progress, but they usually don't!) I could see pv(1) being fixed to address case 3 (by e.g. drawing progress while streaming stderr-logged output below it, using a TUI); but the other cases seem to be fundamental limitations. Personally, when I want to observe progress on some sort of operation that's creating files (rsync, tar/untar, etc), here's what I do instead: I run the command-line, and then, in a separate terminal connected to the machine the files are being written/unpacked onto, I run this: # for files watch -n 2 -- ls -lh $filepath # for directories watch -n 4 -- du -h -d 0 $dirpath If I'm in a tmux(1) session, I usually run the file-copying command in one pane, and then create a little three-vertical-line pane below it to run the observation command. Doing things this way doesn't give you a percentage progress, but I find that with most operations I already know what the target's goal size is going to be, so all I really need to know is the size-so-far. (And pv(1) can't tell you the target size in many cases anyway.)
- prmoustache 4y ago
  Sometimes you prefer predictability and information over sheer speed. If do a very large transfer that could take hours, I'd rather trade a bit of speed to know the progress and make sure nothing is stuck than launching in the blind and then repeat slow and expensive du commands to know where I am in the transfer or have to strace the process.
TT-392 4y ago
But... pipe-viewer was already a commandline youtube browser
- dima55 4y ago
  pv predates youtube itself
heinrich5991 4y ago
`progress` is also a nice tool to see progress of programs operating linearly on a single file. A lot of tools do that!
torgard 4y ago
There are countless times where I would have found this incredibly helpful. Just 10 minutes ago, I wanted this exact tool. Thanks!
sigmonsays 4y ago
i've consistently lost and found this tool over and over again for over 20 years
- pbhjpbhj 4y ago
  Same, `apropos $keyword` helps, but strangely in this case doesn't find `progress` from `apropos progress`.
dang 4y ago
Related: PV (Pipe Viewer) – add a progress bar to most command-line programs - https://news.ycombinator.com/item?id=23826845 https://news.ycombinator.com/item?id=23826845 - July 2020 (2 comments) A Unix Utility You Should Know About: Pipe Viewer - https://news.ycombinator.com/item?id=8761094 https://news.ycombinator.com/item?id=8761094 - Dec 2014 (1 comment) Pipe Viewer - https://news.ycombinator.com/item?id=5942115 https://news.ycombinator.com/item?id=5942115 - June 2013 (1 comment) Pipe Viewer - https://news.ycombinator.com/item?id=4020026 https://news.ycombinator.com/item?id=4020026 - May 2012 (26 comments) A Unix Utility You Should Know About: Pipe Viewer - https://news.ycombinator.com/item?id=462244 https://news.ycombinator.com/item?id=462244 - Feb 2009 (63 comments)
deleted 4y ago
[deleted]
Drybones 4y ago
Funny timing as I recently, as of yesterday, found out about pv when searching for a way to view the progress of a zfs send and receive Another utility I found out about is “progress” available at least on debian systems. It can monitor stuff like cp and mv without actually being used in the command
pkrumins 4y ago
I wrote a pv tutorial: https://catonmat.net/unix-utilities-pipe-viewer https://catonmat.net/unix-utilities-pipe-viewer
herpderperator 4y ago
Can I just say, there's something so nice about this webpage. It's information-dense and well organised. I wish we had more of this today.
Kukumber 4y ago
Can someone change the URL to https? https://www.ivarch.com/programs/pv.shtml https://www.ivarch.com/programs/pv.shtml

183 points by 0x45696e6172 4y ago

I've used it mostly to measure events per second with something like: tail -f /some/log | grep something | pv -lr > /dev/null or tcpdump expression | pv -lr > /dev/null

JayGuerette 4y ago

pv is a great tool. One of it's lesser known features is throttling; transfer a file without dominating your bandwidth: pv -L 200K < bigfile.iso | ssh somehost 'cat > bigfile.iso' Complete with a progress bar, speed, and ETA.

smcl 4y ago
Oh damn that's neat I never thought to use `ssh` directly when transferring a file, I always used `scp bigfile.iso name@server.org:path/in/destination`
- fbergen 4y ago
  Also see `scp -l 200 bigfile.iso name@server.org:path/in/destination` from man page: -l limit Limits the used bandwidth, specified in Kbit/s.
  - MayeulC 4y ago
    Also you probably shouldn't use scp. rsync and sftp have mostly the same semantics. rsync --bwlimit=2OOK bigfile.iso name@server.org:path/in/destination sftp -l 200 bigfile.iso name@server.org:path/in/destination Although it seems that scp is becoming a wrapper around sftp these days: https://www.redhat.com/en/blog/openssh-scp-deprecation-rhel-9-what-you-need-know https://www.redhat.com/en/blog/openssh-scp-deprecation-rhel-... https://news.ycombinator.com/item?id=25005567 https://news.ycombinator.com/item?id=25005567
- tyingq 4y ago
  A similar trick that's nice is piping tar through ssh. Handy if you don't have rsync or something better around. Even handy for one file, since it preserves permissions, etc. tar -cf - some/dir | ssh remote 'cd /place/to/go && tar -xvf -'
  - Twirrim 4y ago
    I love this trick. I was dealing with some old solaris boxes something like 15 years ago when I learned you could do this. I couldn't rsync, and had started off SCP'ing hundreds of thousands of files across but it was going to take an insane length of time. Asked one of the other sysadmins if they knew a better way and they pointed out you can pipe stuff in to ssh for the other side too. Every now and then this technique proves useful in unexpected ways :)
dspillett 4y ago
Similarly, though useful less often these days, using -B/--buffer-size to increase the amount that it can buffer. If reading data from traditional hard drives, piping that data through some process, and writing the result back to the same drives, this option can increase throughput significantly by reducing head movements. It can help on other storage systems too, but usually not so much so.

est 4y ago

pv was the tool when I discovered sometimes the VPS have only 10Gbps memory copy speed.

cwillu 4y ago

pv -d $(pidof xz):1 is great for when you realize too late that something is slow enough that you want a progress indication, and definitely do not want to restart from scratch.

xuhu 4y ago
How `pv -d` work ? Does it use perf probes or attach to the target PID ?
- remram 4y ago
  It finds the file using /proc/<pid>/fd/<num> and watches its size grow. It doesn't work with pipes, devices, a file being overwritten (not appended to), or anything whose size doesn't grow.
  - Jenda_ 4y ago
    It does, even for reading (cat /dev/nvme0n1 > /dev/null). As per changelog: 1.5.2 - 10 February 2014 allow --watchfd to look at block devices You can see the position in /proc/<pid>/fdinfo/<num>
- cwillu 4y ago
  It appears to monitor the contents of /proc/‹pid›/fdinfo/‹fd›
dspillett 4y ago
Another good option for that, which works in a number of other useful circumstances too, is progress: https://github.com/Xfennec/progress https://github.com/Xfennec/progress

londons_explore 4y ago

It would be nice to indicate if the upstream or the downstream is the 'limiting' factor in speed. Ie. within pv, is it the reading the input stream or the writing the output stream that is blocking most of the time?

ketralnis 4y ago
It's open source, be the change you want to see in the world
- kotlin2 4y ago
  Having maintained an open source library, it’s actually really helpful to see features people want. Not everyone needs to contribute directly to the code base. User feedback is valuable, too.
- ranger_danger 4y ago
  Unfortunately not everyone is a developer.
- deleted 4y ago
  [deleted]
senjin 4y ago
This would be a genius addition
bingaling 4y ago
it's instantaneous, but the -T (transfer buffer % full display) is sometimes useful for that. (0% full -> source limited, 100% full -> sink limited)
- Twirrim 4y ago
  Oh wow, I'd completely missed that -T flag. That's some useful data. Thanks for mentioning it!
tanelpoder 4y ago

michaelmior 4y ago

Probably my favorite non-POSIX tool that I insert into my pipelines whenever anything takes more than a few second. I find it super helpful to avoid premature optimization. If I can quickly see that my hacked together pipeline will run in a few minutes and I only ever need to do that once, I'll probably just let it finish. If it's going to take a few hours, I might decide it's worth optimizing. It also helps me optimize my time. If something is going to finish in a few minutes, I probably won't context switch to another major task. However, if something is going to take a few hours then I'll probably switch to work on something different knowing approximately when I can go back and check on results.

systems_glitch 4y ago
Same, one of the first utilities I install on a new system.

derefr 4y ago

As a person who runs a lot of ETL-like commands at work, I never find myself using pv(1). I love the idea of it, but for the commands I most want to measure progress of, they always seem to be either: 1. things where I'd be paranoid about pv(1) itself becoming the bottleneck in the pipeline — e.g. dd(1) of large disks where I've explicitly set a large blocksize and set conv=idirect/odirect, to optimize throughput. 2. things where the program has some useful cleverness I rely on that requires being fed by a named file argument, but behaves a lot less intelligently when being fed from stdin — e.g. feeding SQL files into psql(1). 3. things where the program, even while writing to stdout, also produces useful "sampled progress" informational messages on stderr, which I'd like to see; where pv(1) and this output logging would fight each-other if both were running. 4. things where there's no clean place to insert pv(1) anyway — mostly, this comes up for any command that manages jobs itself in order to do things in parallel, e.g. any object-storage-client mass-copy, or any parallel-rsync script. (You'd think these programs would also report global progress, but they usually don't!) I could see pv(1) being fixed to address case 3 (by e.g. drawing progress while streaming stderr-logged output below it, using a TUI); but the other cases seem to be fundamental limitations. Personally, when I want to observe progress on some sort of operation that's creating files (rsync, tar/untar, etc), here's what I do instead: I run the command-line, and then, in a separate terminal connected to the machine the files are being written/unpacked onto, I run this: # for files watch -n 2 -- ls -lh $filepath # for directories watch -n 4 -- du -h -d 0 $dirpath If I'm in a tmux(1) session, I usually run the file-copying command in one pane, and then create a little three-vertical-line pane below it to run the observation command. Doing things this way doesn't give you a percentage progress, but I find that with most operations I already know what the target's goal size is going to be, so all I really need to know is the size-so-far. (And pv(1) can't tell you the target size in many cases anyway.)

prmoustache 4y ago
Sometimes you prefer predictability and information over sheer speed. If do a very large transfer that could take hours, I'd rather trade a bit of speed to know the progress and make sure nothing is stuck than launching in the blind and then repeat slow and expensive du commands to know where I am in the transfer or have to strace the process.

TT-392 4y ago

But... pipe-viewer was already a commandline youtube browser

dima55 4y ago
pv predates youtube itself

heinrich5991 4y ago

`progress` is also a nice tool to see progress of programs operating linearly on a single file. A lot of tools do that!

torgard 4y ago

There are countless times where I would have found this incredibly helpful. Just 10 minutes ago, I wanted this exact tool. Thanks!

sigmonsays 4y ago

i've consistently lost and found this tool over and over again for over 20 years

pbhjpbhj 4y ago
Same, `apropos $keyword` helps, but strangely in this case doesn't find `progress` from `apropos progress`.

dang 4y ago

Related: PV (Pipe Viewer) – add a progress bar to most command-line programs - https://news.ycombinator.com/item?id=23826845 https://news.ycombinator.com/item?id=23826845 - July 2020 (2 comments) A Unix Utility You Should Know About: Pipe Viewer - https://news.ycombinator.com/item?id=8761094 https://news.ycombinator.com/item?id=8761094 - Dec 2014 (1 comment) Pipe Viewer - https://news.ycombinator.com/item?id=5942115 https://news.ycombinator.com/item?id=5942115 - June 2013 (1 comment) Pipe Viewer - https://news.ycombinator.com/item?id=4020026 https://news.ycombinator.com/item?id=4020026 - May 2012 (26 comments) A Unix Utility You Should Know About: Pipe Viewer - https://news.ycombinator.com/item?id=462244 https://news.ycombinator.com/item?id=462244 - Feb 2009 (63 comments)

deleted 4y ago

[deleted]

Drybones 4y ago

Funny timing as I recently, as of yesterday, found out about pv when searching for a way to view the progress of a zfs send and receive Another utility I found out about is “progress” available at least on debian systems. It can monitor stuff like cp and mv without actually being used in the command

pkrumins 4y ago

I wrote a pv tutorial: https://catonmat.net/unix-utilities-pipe-viewer https://catonmat.net/unix-utilities-pipe-viewer

herpderperator 4y ago

Can I just say, there's something so nice about this webpage. It's information-dense and well organised. I wish we had more of this today.

Kukumber 4y ago

Can someone change the URL to https? https://www.ivarch.com/programs/pv.shtml https://www.ivarch.com/programs/pv.shtml