14 ms·
Linux tool to show progress for cp, mv, dd
- dale_glass 3y agoNice, but I wonder why the actual tools don't already include this. I even recall cp having been patched with a progress bar maybe a decade back in Gentoo, but for some reason that didn't stick.
- w0m 3y ago`alias cpProgress='rsync --progress -ravz'` has been in my ~/.bashrc for the majority of my career for large file transfers.
- freedomben 3y ago`rsync` interprets paths a little differently than cp (such as trailing / or no trailing / on directories). Do you have a convention you normally call this with to avoid differences in behavior?
- geraldhh 3y agogood question since shell's completion behaves differently in adding trailing slashes
- manuelabeledo 3y agoI’m going to go on a limb here, but I would say that coreutils main users aren’t human, but scripts. GUIs have had progress bars for ages as well.
- JdeBP 3y agoIt's more likely that this is the entire "upstream downstream" development model in action again. There are vast numbers of improvements to softwares that never get sent to the places where they will do the most good (or at least have a proper record of why they were rejected by their authors). A quick perusal of the bug trackers of Debian or Ubuntu will reveal tonnes of local patches that the original authors of the softwares often never even hear about. Things don't "stick" often times simply because they get lost. I was looking at something like that just the other day. Here's a bug report that describes a problem with "doas" not opening the controlling terminal to do its authentication dialogue. This is actually a problem with a package named LinuxPAM, and doesn't occur when "doas" uses OpenPAM or BSD Auth. It's LinuxPAM that's where the code in question is. Fixing LinuxPAM would improve the lives of everyone that uses LinuxPAM, because the behaviour of not allowing standard input through in a command pipeline is not confined to "doas" but affects everything that uses LinuxPAM to do login authentication. But time and again stuff like this languishes in the wrong place, for years and decades. * https://github.com/slicer69/doas/issues/17 https://github.com/slicer69/doas/issues/17
- JdeBP 3y agoI went looking and I found this: * https://github.com/jarun/advcpmv https://github.com/jarun/advcpmv This one got lost because the original author's WWW site just vanished, according to the doco.
- reddit_clone 3y agoMy thoughts as well. I think commands could have progress but only as explicit option. I wish curl and 'docker pull' didnt spew so much text by default, filling up so many jenkins server's disks.
- j33zusjuice 3y agoI know you said “by default” so I’m assuming you know some ways around this, but in case you don’t, or for anyone who might find this useful … you can be really specific with curl outputs. It’s pretty sick. curl -sS -w '%{http_code} %{http_version}' https://www.goog https://www.goog le.com -o /dev/null https://everything.curl.dev/usingcurl/verbose/writeout https://everything.curl.dev/usingcurl/verbose/writeout I’ve only recently had to use curl’s features more in-depth, so I’m still fascinated by its potential.
- xp84 3y agoThanks! This sounds like a great tool for simple scripts where something like a quick response code (etc) would be useful to know. I'm imagining something like: Creating ticket... (200 OK)
- coxley 3y ago"Make each program do one thing well" The philosophy surrounding these tools wouldn't lend well to each implementing that. Fortunately `pv` exists, is perfect for this use-case, and included in many distributions. :)
- simula67 3y agoI think it is probably because of the UNIX philosophy of the virtue of silence: 'The program should say nothing if it has nothing interesting to say'. It is an open question whether progress is 'interesting' or not. My own opinion is that it is not interesting if the operation is nearly instantaneous. If any operation can take more than 5 seconds, it should have a progress bar and an estimated time of completion. Earlier versions of Windows did this very well. Often these progress bars were a joke, but sometimes they were useful. It gave us valuable input on whether there is enough time to go get a cup of coffee, which as we all know, is the most important question for us all.
- ElectricalUnion 3y agoEarlier OSes and applications with only cooperative multitasking had to do this well; if a misbehaving application/operation was running it could just monopolize the system-wide event loop and stall the entire system.
- hulitu 3y ago> Earlier versions of Windows did this very well. Almost. For some reason the Win95 and Win98 installer needed 75% of the time to go to 99% and 25% of the time to go from 99% to 100%.
- Yizahi 3y agoFor example Linux scp has progress bar and prints success results by default. I wish more tools did this.
- Yizahi 3y agoBy the way, regarding Windows progressbars - even if a particular implementation is garbage, like for example passing to 50% instantly, then sitting there for a minute, then passing again to 75% and staying there and son on, they are still useful if your are using the same tool or operation repeatedly. My VPN progress indicator is junk, but I can roughly guess where is the problem based on where it got stuck now, simply because I've already saw multiple failure modes of it. Or if I'm installing some CPP distributives repeatedly, I can guess how long remains. If an operation is silent and hogging my prompt without any indication what is going on, is stuck, is the speed too slow etc. it is simply frustrating.
- rurban 3y agoI do maintain the mv cp progress patches, but they recently broke, when they added another feature. coreutils refused the patches as saying they are feature complete, whilst they dont have progress support, nor Unicode support. Stubborn https://github.com/rurban/coreutils/ https://github.com/rurban/coreutils/
- lathiat 3y agoI use the pv “Pipe Viewer” tool to do the same. You can either put it in the middle of a pipe, or pass it a PID using -p http://www.ivarch.com/programs/pv.shtml http://www.ivarch.com/programs/pv.shtml It works by reading /proc/PID/fdinfo/*
- mavhc 3y agoI used PV's rate limit when trying to do to a multi terabyte zfs send, slow it down and speed it back up as required
- geraldhh 3y agowhy would one want to slow it down?
- LeFantome 3y agoTo stop from saturating a slow shared connection for an extended period? This kind of transfer can interfere with what other users are trying to accomplish over the same link.
- geraldhh 3y agoah, sure thing! you want some kind of managed buffer (aqm/sqm/qos) in your bottleneck router. openwrt or any linux can do it as the needed parts (fq_codel or cake qdisc) are in upstream for some years now and wrappers like firewall distros or libre-qos make it available via gui. there is also some proprietary soho gear that allegedly works, big-iron is still on red/pie w/o fair queueing afaik
- yabones 3y agoQoS is sort of a last resort, since it works by basically breaking TCP intermittently (dropping ACK segments). Any time you can throttle further up the stack is ideal.
- shric 3y agoNewer versions of Linux dd support status=progress. iirc, dd on *BSD will show progress on ^T because it sends a SIGUSR1
- JdeBP 3y agoIt's SIGINFO, and this is a common thing on the BSDs.
- mort96 3y agoI actually believe it sends a SIGINFO, not a SIGUSR1. It's unfortunate that SIGINFO never made it to Linux, it's an incredibly useful feature.
- arp242 3y agoIndeed, but on GNU dd you can send SIGUSR1 to do the same as what SIGINFO would do on BSD. Downside of SIGUSR is that it will kill the process if there's no handler, rather than ignoring it, so "just try it" is always risky.
- garaetjjte 3y agoYou could make that a bit safer by first checking for SIGUSR1 in SigCgt mask in /proc/pid/status
- mort96 3y agoI used to have a habit of running 'killall -USR1 dd' to get the status of a dd process. Then I switched to using sddm for my display manager/login screen. It didn't have a USR1 handler. If you kill your display manager, you're forcefully and suddenly logged out of your session. I stopped using 'killall -USR1 dd' for dd status.
- jmclnx 3y agoThis is one thing I really miss on Linux when compared with the BSDs, especially with dd(1). On BSD you can press ^t to see the status of a command. All you need to do is issue this command to activate it: % stty status '^t' btw, for Linux dd(1) I know about "status=progress", but for me it is a bit hard to remember and specific to dd(1). But, nice little utility :)
- thedougd 3y agoCtrl-t works for me on DD on Linux.
- deleted 3y ago[deleted]
- yrro 3y agoNot here. I guess you've somehow got ^T to send SIGUSR1 to the foreground process?
- deleted 3y ago[deleted]
- j33zusjuice 3y agoHow often do you use dd that this really matters? Just curious. I’ve run dd maybe 20 times in the 5-6 years I’ve worked with Linux professionally.
- pstoll 3y agoCute, clever. There may be edge cases it doesn’t get right (parallel downloads/copies) but pretty useful when you’ve launched a job and didn’t think ahead of time to ask for progress. I’ve used a signal handler (SIGHUP aka control-c) before as a “show progress” mechanism that I found very useful to monitor long running processes that were compute not io related (launched in screen fwiw to stay the active process).
- loeg 3y agoThese use USR1 signal to show progress. HUP is kind of a weird choice since usually the intent of ^c is to end the program.
- pstoll 3y agoYep I know. It was easier to do in the same window with the program in the foreground vs opening another window to send the kill -HUP to this one. Double control-c still killed the process.
- rollcat 3y agoVery cute and very clever. I think this could be extended with a watcher/main loop of its own, and keep a bit of context to spot any long transfers (say longer than 5s), and display a system-wide summary of all large transfer going on.
- pstoll 3y agoRandom aside - I know formatting discussions border on the religious (and why something like gofmt is the only correct answer & yet I am also good with spaces for Python) but.. Did anyone else look at the code and ask themselves- what is the actual formatting standard being used? Looked like a mix of “open brace on same line, 4 chars indent for code” then “open brace on new line, code at same zero indent” Not a big deal obviously. Just something that tripped up my eyes scanning the code.
- antihero 3y agoYeah it just looks messy and harder to read when it’s inconsistent
- reaperducer 3y agoDid anyone else look at the code and ask themselves- what is the actual formatting standard being used? Looks like a combination of Whitesmiths and ChatGPT.
- akritid 3y agoFrom a quick look to one file, seems fairly consistent. Perhaps most unorthodox but at the same time easiest to justify: function bodies skip one level of indentation. The curly brace of function body at column zero is actually a separate, very traditional style. It's just that both styles apply to function bodies, there is no other causal relationship. Finally, indentation is 4 spaces, and tabs are expanded. Of course a very few places may be miss-styled, as happens with hand crafted code. While definitely sinful for not indenting with tabs as God intended, the style is not messy
- igtztorrero 3y agoI use pv, Pipe view since forever: pv largefile.sql > mysql -u root -psecret
- _joel 3y agoI just use pipeview (pv) in all pipes where I need visibility https://linux.die.net/man/1/pv https://linux.die.net/man/1/pv
- sva_ 3y agoDoes this take the kernel disk caches into account? (I.e. having to run sync after dd)
- loeg 3y agoNo. It's just looking at the current offset in the logical kernel file.
- jtode 3y agoI've been adding status=progress to my dd commands and getting progress reports for years now, not sure when it started working like that but any current Linux should have it.
- pjmlp 3y agoCatching up with some MS-DOS tooling of yore?
- emidoots 3y agoWow, didn't expect to see @xfennec pop up on hacker news while drinking my coffee this morning! I don't know if he'll see this, to be honest didn't know he was still doing things - but this person basically got me into programming and game development-I really can't believe it. xfennec (and some friends?) I think built a game engine called Raydium, and one of their games called Mania Drive-a Track Mania clone-got distributed with OpenSuse installation CDs back in the day. When I was just like 12 years old, my dad installed that on the family computer and it was all we had, Mania Drive was one of the coolest games on there. Me and my siblings played that for literally days and months on end, making crazy levels we couldn't beat without knowing every turn. It was a huge part of our childhood. Their game engine was in C with PHP scripting, I remember posting some levels to their forums and asking, in retrospect super dumb, questions and they were so polite and friendly. I remember us joking at the time that the French seemed like these god-like game developers, it had such a profound impact on us, I even wrote about it last year and linked a video of Mania Drive first[0]. I went on to learn Python and then lower-level languages as a result. I'm not sure I'd be coding today without them, to be honest. Sorry it's off-topic, just really blown away to see a username like that pop up in my feed. Really goes to show that kindness + some cool open source software can have profound effect on people. [0] https://devlog.hexops.com/2021/increasing-my-contribution-to-zig-to-200-a-month/ https://devlog.hexops.com/2021/increasing-my-contribution-to...
- ktm5j 3y agoI love stuff like this, thanks for sharing!!
- dilap 3y agoSuch a great story. This part made me lol: > // Don't remove this print statement. Game will crash! :-)
- xfennec 3y agoXfennec here, thank you so much for this message. ManiaDrive was a small game made with a bunch of friends, I'm so glad it had an impact on you. I'm now a dad, and it makes me very emotional to read this. Thanks again.
- asn1parse 3y agoi use rsync. even locally. its one of the best tools ever written, my goto for diffing whole directories. have a nice day
- goodpoint 3y agoGPL and C, thank goodness.
- arbitrandomuser 3y agoWhy mv ? Mv doesn't really move around bytes right ? Just moves pointers aroun right ?
- marcosdumay 3y agoThat depends on where you are moving your files from and to. If they are different filesystems, than no, it doesn't just copy inodes.
- eps 3y agoOnly if the move is within the same volume.
- zamubafoo 3y agoThat's interesting! I often found myself forgetting to turn on progress flags on many data transfer jobs and the occasional data transform batch job that I looked into something like this. I found that `iotop` is great for this kind of thing. Sure, you have to either start it before your process starts or your accumulated total is off, but usually I'm not tracking progress for files less than 1GB so being off by kilobytes is fine. My go-to's are `sudo iotop -aoP` for general monitoring, adding the `-p` flag if it's just a specific process, or `-u` if I'm monitoring something that is possibly transient.
- bornfreddy 3y agoI also often forgot to add progress flags, but lately I don't even bother... I just start, then `progress -w` or `watch progress -w`. Works nicely.
- stack_underflow 3y agoOne of my quick-and-dirty gotos for getting a rough idea of buffered-writes size + disk-write activity on random linux systems is: `watch -n1 grep -ie dirty -e writeback /proc/meminfo`. You can invoke `sync` to watch the buffered-writes queue burn down when you have lots of pending writes. see: `LESS=+/meminfo man proc` or https://github.com/torvalds/linux/blob/master/Documentation/filesystems/proc.rst#meminfo https://github.com/torvalds/linux/blob/master/Documentation/... for more info
- eps 3y ago> It simply scans /proc for interesting commands, and then looks at directories fd and fdinfo to find opened files and seek positions, and reports status for the largest file. Wasn't expecting something as simple that at all. Bloody ingenious.
- gaws 3y agoadvcpmv[1] shows progress by default for `cp` and `mv` commands. [1]: https://github.com/jarun/advcpmv https://github.com/jarun/advcpmv
- mlrhazi 3y agofor RHEL, which repo has this package?
- marcodiego 3y agoIMHO, it would be much better if linux implement SIGINFO. It has been present in BSD's since forever and there is a good linux implementation: https://lkml.org/lkml/2019/6/5/174 https://lkml.org/lkml/2019/6/5/174
- notorandit 3y agoWhy not using pv? http://www.ivarch.com/programs/pv.shtml http://www.ivarch.com/programs/pv.shtml
- colpabar 3y agowow I usually just use rsync with the progress flag but this looks way cooler. thanks!
- tylerchr 3y agopv is invaluable but my issue with it is it’s only useful if you think to include it before you run the command. Usually the thought doesn’t occur to me until I have some runaway cp or dd and I want to be reassured that it’s going to be done soon. For that case that this looks super interesting.
- rlpb 3y agoYou can attach pv to an existing process. IIRC it's the -p option.
- everybodyknows 3y ago-p is same as "--progess": https://linux.die.net/man/1/pv https://linux.die.net/man/1/pv
- rlpb 3y agoAh, it was -d I was thinking of. Anyway, the point is that you can run pv retrospectively just fine while your slow command is running.
- e40 3y agoThe "how does it work" section doesn't make sense for macOS since there is no /proc there. How does it work on macOS?! I tested it and it works like a charm!
- rollcat 3y agoLook at progress.c, start with "#ifdef __APPLE__" and keep looking from there. Something called libproc, there's some headers[1] I've found but I couldn't find any man pages unfortunately. You need some way to look at open FDs and every system will have such an API, even if it looks slightly (or very) different from the next one. [1]: https://opensource.apple.com/source/xnu/xnu-7195.81.3/libsyscall/wrappers/libproc/libproc.h.auto.html https://opensource.apple.com/source/xnu/xnu-7195.81.3/libsys...
- awill 3y agoI'm sure it's nearly impossible, but this needs to be integrated with cp, mv and dd. Otherwise, usage will be a fraction of a percent. I SSH into many many different machines (desktops, routers, servers, IoT devices etc..), and unless something is in the default install of RHEL, Debian, Arch etc..,I tend to not rely on it, as my muscle memory will cause me problems.
- crznp 3y agoI have the same issue, but with the opposite conclusion: if this were integrated with cp/mv/dd, I wouldn't be able to rely on getting the new versions. But since it is separate, I can install it where and when I need it, perhaps with the transfer already in progress. That might be hard for IoT/containers, but do I really need to see file copy progress in those kinds of places?
- awill 3y agoGood point on updates. But if Debian pulled in latest every 2 years (Debian release cycle), wouldn't that be enough. I can't imaging improvements being so urgent. Also, agreed on the need for this on IoT stuff. But it's about muscle memory. I want to type the same way on. all devices. Not "oh, I'm on a server, use this command. Oh, I'm on an IoT device, use a different command"
- mmh0000 3y agoOh this is fun! It reminds me of a Bash script i wrote years ago that does something very similiar[1]. [1] https://xn0.co/rp https://xn0.co/rp
- jadia 3y agoHi mmh0000, How did you set up your website xn0 to deliver the /rp file?
- hk1337 3y agoI'm curious to see how well it works when using -v I have all my basic commands, cp, mv, rm aliased to add the -v argument.
- sampa 3y agoI always forget about pv, though use status=progess of dd. When I need it in the middle of process, I just `ls /proc/_pid_/fd` and then take that _fd#_ to `cat /proc/_pid_/fdinfo/_fd#_`
- natmaka 3y agoSee also: https://github.com/ebikt/progress.pl https://github.com/ebikt/progress.pl
- agilob 3y agoOn FreeBSD you can do CTRL+T to show progress, if I remember, that's a special shourtcut similar to CTRL+Z/C supported by BSD systems. There were some discussions a few years ago that Linux should introduce the same thing, but they never did.
- kazinator 3y agoIf you have a program that can spew lots of output about the progress it is making, you can redirect it to Pipe Watch: https://www.kylheku.com/cgit/pw/about/ https://www.kylheku.com/cgit/pw/about/ Pipe Watch continues to read from the pipe even when backgrounded. Pipe Watch shows you snapshots of the text that is passing through it. You can set triggers and filter and such. The triggers work even when it's in the background, not refreshing the display..
- abotsis 3y agoThe method this uses is cute, hacky, and useful. Makes me want to write an osx background thing that uses the same scheme and pops up progress windows whenever I run a coreutils thing.
- dang 3y agoRelated: Show HN: Linux tool to show progress for cp, rm, dd, etc. - https://news.ycombinator.com/item?id=8023713 https://news.ycombinator.com/item?id=8023713 - July 2014 (53 comments) (as it doesn't seem to have been posted by the creator, Show HN was a mislabel there)
- francois_h 3y agoThis is cool i guess. I usually just use rsync to get the same/better effect.
- anonymousiam 3y agoGreat to see this here! I've never used it until now because instead I used a 20 year old shell script (later converted to Python by a friend) to do the same thing. I gave it a try on a ddrescue job (unfortunately not recognized by default) and the estimated remaining time varies quite a bit between what ddrescue says and what progress says. I think ddrescue uses a larger moving average window and it seems to give more accurate estimates, although they are still far from perfect.
- vram22 3y agohttps://catonmat.net/unix-utilities-pipe-viewer https://catonmat.net/unix-utilities-pipe-viewer is a related tool.
- vram22 3y agoThere is also watch, a Linux command. Had blogged about it here, along with a Python program inspired by it, that I wrote: https://jugad2.blogspot.com/2018/05/a-python-version-of-linux-watch-command.html?m=0 https://jugad2.blogspot.com/2018/05/a-python-version-of-linu...
- vram22 3y agoAnd that drawing of the watcher at the top of the post is by yours truly :)
- nhoughto 3y agovery neat! definitely will come in handy.
- stevefan1999 3y agoGreat! This looks like a good candidate to Rewrite-it-in-Rust!
- xk3 3y agoIt also works with ffmpeg but only if you specify it progress -wc ffmpeg
- juliangmp 3y agoI still think its weirdly unreasonable that the gnu coreutils dont give you the option to print out progress in that command.