28 ms·
Moreutils – Unix tools that nobody thought to write (2012)
- forgotmypw17 6y agoIt's funny that `vidir` spawns `nano` by default on my system.
- endgame 6y agoCheck your $EDITOR environment variable.
- jdub 6y agoThat'd be the Debian alternatives system in action. $ update-alternatives --list editor
- JdeBP 6y agoOr sensible-editor. * http://jdebp.uk./FGA/unix-editors-and-pagers.html http://jdebp.uk./FGA/unix-editors-and-pagers.html
- jfreax 6y agoIt probably honers your $EDITOR env variable which often defaults to `nano`. You can change it to anything you want including `vi[m]` :)
- cel1ne 6y ago> sponge Why not use > file? > mispipe In bash there's PIPESTATUS for that.
- charliesome 6y agoRe sponge: the shell will open the output file for writing before invoking the command so in the example joey provides, /etc/passwd will be an empty file by the time sed opens it.
- samatman 6y agoheh, yeah that's a mistake everyone makes once, and hopefully not more than once.
- prakashdanish 6y agoThat makes sense but in the example provided, why can't we use sed with the in-line flag?
- gpanders 6y agoBecause the -i flag works differently on GNU sed vs non-GNU sed and there is no way to use it portably.
- gravypod 6y ago> Why not use > file? I'm assuming sponge buffers everything in memory so you don't get concurrent modification bugs.
- masklinn 6y agoYes, if you try to sed a file back to itself for instance you'll get a truncated file (or maybe a no-op?). In fact, that's exactly the example sponge's manpage uses.
- lloeki 6y agoThat redirection will make the shell truncate the file long before sed has a chance to open it.
- moonchild 6y agoIt's a truncated file: $ cat test moo $ cat < test > test $ cat test $
- Izkata 6y agoI was wondering about tee for that: $ cat test moo $ cat < test | tee test > /dev/null $ cat test moo ...seems to work. Am I just getting lucky with a race condition?
- notimetorelax 6y agoPossible, since you don’t pipe into the file but instead ask tee to write into it.
- Erwin 6y agoYes, check this, with f.txt containing 99999 numbers: $ seq 1 99999 > f.txt $ cat < f.txt | tee f.txt > /dev/null $ wc -l f.txt 23696 f.tx
- moonchild 6y ago
- wodenokoto 6y agoI think this is a good question as moreutils doesn’t do a very clear job of explaining it. Sponge lets you do a series of piped transformations on 1 or more files and overwrite the result back to the original files. Using > things will just break. I’m guessing this also means you can’t pipe something bigger than what your memory can hold.
- semicolonandson 6y agoGreat question. There's a good Stack Overflow answer here: https://unix.stackexchange.com/questions/207919/sponge-from-moreutils-whats-the-difference-to-shell-redirect-useful-examples https://unix.stackexchange.com/questions/207919/sponge-from-...
- GuB-42 6y ago> Why not use > file? This, I learned the hard way, and I am not the only one. So in order for you not to make that mistake: the first thing that happens is that "file" is truncated, that happens before any command is run, so sed s/foo/bar/ file > file will always result in an empty "file", because it will be truncated before "sed" is run. I lost a couple of hours of work like that, once, and then I learned. From the manpage of sponge sponge reads standard input and writes it out to the specified file. Unlike a shell redirect, sponge soaks up all its input before opening the output file. This allows constricting pipelines that read from and write to the same file. So, the command sed s/foo/bar/ file | sponge file will do what you expect
- atsaloli 6y agoTry sed s/foo/bar/ -i file to edit file.
- GuB-42 6y agoThat what I do now, one thing to note though is that the "-i" option is not always available. Usually, now, it is, but we still have a couple of Solaris servers at work that don't support it. This, btw, is why I hate shell scripts. There are so many variants of bourne shells and UNIX tools that writing a portable script is a minefield, as if properly dealing with spaces wasn't tricky enough...
- atsaloli 6y agoRight you are. The -i option is not standard (https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sed.html https://pubs.opengroup.org/onlinepubs/9699919799/utilities/s...)
- dang 6y agoIf curious see also 2016: https://news.ycombinator.com/item?id=12023277 https://news.ycombinator.com/item?id=12023277 2015: https://news.ycombinator.com/item?id=9013570 https://news.ycombinator.com/item?id=9013570 2015 (1 comment): https://news.ycombinator.com/item?id=9004302 https://news.ycombinator.com/item?id=9004302
- aserafini 6y agoI wrote a util a bit like this when I realised there was no command for emptying a directory: https://github.com/adamserafini/emptydir https://github.com/adamserafini/emptydir
- jfreax 6y agoWhat the difference in just deleting the whole directory and recreating it with `mkdir`. Or just do an `rm -rf dir/*`?
- ggrrhh_ta 6y agoHidden directories, links within dir... (usually, you want a "nofollow" default for destructive operations)
- slavik81 6y agoWhile your code is very nicely factored, well-documented and quite readable, bash is not an easy language for this sort of utility. Any names containing spaces or starting with dashes require very careful handling. If you do stick with bash, I would recommend giving shellcheck a try. It won't catch everything, but I've learned a lot from running it on my code.
- jfreax 6y agoNice collection of quite useful tools. Some of these can be easily replicated by using a more modern shell (bash, zsh) like mispipe, others are just shortcuts (e.g. ifne, chronic). But what immediately stood out to me is `vidir`. I really like the idea of editing file names with an editor. Using loops and regex in a shell for mass renaming can be a mess. It should be way easier with `vim`. This tool made me install moreutils.
- pmoriarty 6y agoCheck out renameutils[1], which has the advantage over vidir of doing sanity checks before renaming. Also, this can be done in emacs using wdired. [1] - http://www.nongnu.org/renameutils/ http://www.nongnu.org/renameutils/
- brobdingnagians 6y agoThe ranger file manager also has this built in.
- RMPR 6y agoWas about to point this out, but it's good to know if you ssh in a box which doesn't have Python installed. Edit: Or use a more portable terminal FM, nnn pops to my mind
- nieve 6y agoMMV is reasonably powerful, but straightforward, can produce & consume rename listings, and is included in a lot of distros.: https://git.deb.at/w/pkg/mmv.git https://git.deb.at/w/pkg/mmv.git
- markrages 6y agoSee writable dired-mode in emacs. Here's a short demo: https://youtu.be/8l4YVttibiI https://youtu.be/8l4YVttibiI
- ainar-g 6y agoOn most POSIX systems you can use fc(1) to edit a command in your $EDITOR. In vi(1) and friends you can then use "%!ls" to replace the contents of the buffer with directory listing and edit the commands you want. https://pubs.opengroup.org/onlinepubs/9699919799/utilities/fc.html https://pubs.opengroup.org/onlinepubs/9699919799/utilities/f... https://pubs.opengroup.org/onlinepubs/9699919799/utilities/vi.html https://pubs.opengroup.org/onlinepubs/9699919799/utilities/v...
- siraben 6y agoIs it possible to make some of these moreutil commands aliases of some combination of coreutils commands?
- skrause 6y agoI use `chronic` in almost all of my cron jobs so that cron sends me an email with the output only if the command has a return code of != 0.
- rurban 6y agoWith the annoying half-way implementation of GNU parallel. Does not support the basic features, is not better nor faster, but still tramples over this namespace. I really like many of the moreutils tools, but dealing with constant parallel breakage (kind of a Hadoop for dummies) is annoying.
- rozab 6y agoThere's a few programs I wish were standard on all Unix systems by now: * tree - print directory structure * bat - cat, but actually designed for reading files. Syntax highlighting, line numbers, automatic paging. * rg - better grep * direnv - local environment variables
- beojan 6y ago> rg Never going to happen because it's written in rust. ag (https://github.com/ggreer/the_silver_searcher https://github.com/ggreer/the_silver_searcher) on the other hand is written in C.
- feanaro 6y agoWhat's never going to happen?
- saagarjha 6y agoA Rust program being standard on all UNIX systems.
- genuinebyte 6y agoI'll bite: Why?
- saagarjha 6y agoRust is designed for a fairly small number of relatively modern systems; there are many close-to-POSIX systems that can’t (and I would guess likely will never) be able to run Rust code in any formal capacity. To the hopeful: even C++ has been unable to win here, despite having many more years of a head start and much more accommodation for stranger platforms. It’s just not going to happen.
- MaxBarraclough 6y ago
- parentheses 6y agoI'm still stuck on the fact that one of the utilities is called `pee`.. In fact the docs even say: > make sponge buffer to stdout if no file is given, and use it to buffer the data from pee. Translation, pee into a sponge!
- rmetzler 6y agoI wish there was a short and simple command for “list all files of a directory and no subdirectories“. ls doesn’t seem to have a switch for that kind of functionality. It’s discussed on stack-overflow. [0] Someone even wrote a nodejs tool for this functionality [1], but I would rather have something written in a compiled language. [0]: https://stackoverflow.com/questions/10574794/how-to-list-only-files-and-not-directories-of-a-directory-bash https://stackoverflow.com/questions/10574794/how-to-list-onl... [1]: https://github.com/mklement0/fls https://github.com/mklement0/fls
- MaxBarraclough 6y agoBut the top answer has it: > find . -maxdepth 1 -type f A little clunky, but there's certainly no need to mess about in JavaScript land. If you use it often, create a shell macro.
- rmetzler 6y agofind doesn't return the same output as ls when run on another directory. Maybe there is a switch to do this? I'm not sure. $ mkdir -p tmp && touch tmp/a tmp/b tmp/c $ find ./tmp -maxdepth 1 -type f ./tmp/a ./tmp/c ./tmp/b $ ls -1 ./tmp a b c
- timonoko 6y agoWhere is Msdos utility "ncd"? It was like "cd" but guessed from few character hint where you wanted to go. If the choice was not immediately obvious if offered a menu or a tree. I shortened it to "n". -- There was some linux utility but setting it up was annoyingly tedious, with lots of useless and cryptic options.
- andrewshadura 6y agocd cannot be an external command in Unix-like systems, it has to be a shell built-in.
- boring_twenties 6y agoIt could have been a script for adding to shell startup files.
- oweiler 6y agoBut cd can be invoked from a shell function (which doesn't create a subshell).
- Lio 6y agoI think that's the default behaviour in Zsh or Fish if you press tab.
- timonoko 6y agoProbably immediate subdir. What I am talking about is universal jump-to-anywhere. Like I have subdir "/media/tnoko/sdc1/capture/Roinaa/". "ncd Roi" should be totally sufficient and unique command to go there.
- RMPR 6y agoFor this I use autojump[0], what I really like is the integration with ranger FM[1]. I don't know exactly how ncd works, but with autojump you need to visit the directory at least one time to be able to fuzzy-jump. 0: https://github.com/wting/autojump https://github.com/wting/autojump 1: https://github.com/fdw/ranger-autojump https://github.com/fdw/ranger-autojump
- blauditore 6y agoWhat is the difference between `sponge` and redirecting with `>` (or `>>`)? Is it about better compatibility with pipes or something?
- deleted 6y ago[deleted]
- JoelMcCracken 6y agoWell one thing is that if you’ve tried it you’ll run into the problem he illustrates there that you are writing to the Sam file you are reading from, which won’t work right. My guess is sponge buffers all the input and then sends it to output once stdin is closed.
- deleted 6y ago[deleted]
- circular_logic 6y agoI wondered this too so looked at the man page [1] > Unlike a shell redirect, sponge soaks up all its input before opening the output file. This allows constricting pipelines that read from and write to the same file. [1] https://linux.die.net/man/1/sponge https://linux.die.net/man/1/sponge
- jleahy 6y agoImo sponge is the most useful of all of them. For example if you want to filter a file in place you can do ‘grep str file | sponge file’. If you do that with a redirect you’ll end up with an empty file. Some commands (eg. sed) have an in-place option, but many don’t.
- ORioN63 6y agoI use vipe frequently, to let me pipe from/into vim. This enables fun shortcuts as: xclip -o | vipe | xclip This one in particular, let's you edit your clipboard with $EDITOR.
- donatj 6y agoA lot of these seem easily achieved in a POSIX shell manner, but most annoyingly to me > sponge: soak up standard input and write to a file I do that all the time... cat > file.txt Update: reading the comments on here, apparently sponge sucks up all content before opening the destination file which allows editing an input file in place. Minor advantage there.
- jpab 6y agoThe purpose of sponge is that you can do this: grep foo file.txt | sponge file.txt If you do this with redirections then file.txt will be truncated before it's been processed, leaving you with an empty file instead of what you wanted. Sponge collects its input first and then writes everything out at the end, so you can output to a file that was used as an input. (Parent updated while I was writing. Oh well)
- tarruda 6y agoThat doesn't seem like an efficient way to do it. Commands that process file in place (sed -i) write to a temporary file in the same filesystem and then rename to the target file, which works if you want to process files that don't fit into memory.
- mslm 6y agoWithout inspecting the implementation of `sponge` that's how I assumed it already internally worked.
- jpab 6y agoI apologise, I was speaking loosely. I don't know if sponge collects input in memory or in a file. (Frankly I've never needed to care since the files I've ever needed this tool for have all been small)
- deleted 6y ago[deleted]
- nextaccountic 6y ago
- stevekemp 6y agoInspired by this project I put together sysadmin-tools: https://github.com/skx/sysadmin-util https://github.com/skx/sysadmin-util Later they were replaced by a busy-box style collection of utilities written in golang (mostly to ease installation): https://github.com/skx/sysbox https://github.com/skx/sysbox
- iforgotpassword 6y agoSo this where errno comes from. I discovered it by accident on my system a few years ago and have been using it ever since. Really handy when working in C. Still no idea why I have that package installed though. On another note > pee: tee standard input to pipes Goddammit guys!
- tarruda 6y agoThe sponge example: sed "s/root/toor/" /etc/passwd | grep -v joey | sponge /etc/passwd I think it can be rewritten as: sed -n '/joey/! s/root/toor/p' -i /etc/passwd
- ISO-morphism 6y agoYes, but -i on sed is specific to GNU, I don't think it exists on BSD/OSX/busybox
- lhoursquentin 6y ago-i is definitely not specificed by POSIX, but it supported on all those platforms with some small differences, for instance on OSX the backup extension (-i.bak) is not optional.
- gpanders 6y agoAmazingly, there is no portable way to use -i that works on both GNU and BSD sed implementations. Which means if you’re writing a portable script, you can’t use -i at all. (Would love to be proved wrong on this)
- bewuethr 6y agoAccording to Stack Overflow [1], this works: sed -i.bak 's/foo/bar/' filename [1]: https://stackoverflow.com/a/22084103/3266847 https://stackoverflow.com/a/22084103/3266847
- gpanders 6y agoThat works when specifying a backup extension, but not if you don’t want to create a backup file. sed -i ‘’ ... works on BSD sed but not GNU. Meanwhile: sed -i’’ ... sed -i ... both work on GNU but not BSD.
- 6y ago
- pkphilip 6y agoCan I use pee into a sponge?
- thomashabets2 6y agots looks like it's a subset of https://github.com/ThomasHabets/ind https://github.com/ThomasHabets/ind sponge can be replaced with dd Other than that, yeah nice ideas. And very much in the unix spirit of one tool to do one thing.
- pepve 6y agosponge buffers the whole input before writing the output. Its utility would be in reading from a file, working on it with other tools, and writing the result back to the original file, all in one line.
- kyran_adept 6y agoI know I wanted to do this, but I am very happy such a utility doesn't exist. Unix utilities, usually work in a pipeline fashion, read line -> process -> write to output. This allows them to allocate a limited amount of memory. If you read all input in memory, you are asking for trouble: 'cat /dev/zero | sponge a'.
- FeepingCreature 6y agoThat is the point.
- zeroimpl 6y agoI routinely operate on machines with gigabytes of memory, but rarely write pipelines which output gigabytes of data, so this has never been a concern for me. But even then, there’s still swap space which is effectively like using a temp file but lazier. By the way, I think you’d be in even more trouble if you wrote: cat /dev/zero > a
- thomashabets2 6y agoOh, neat! Thanks.
- thomashabets2 6y agoSeems like the example command should be easily doable with just `sed -i`, though. sed -i 's/root/toor/;/joey/d' /etc/passwd But I get your point.
- krick 6y agoI seriously hate this package and the manner of combining different utils with different names in a same package in general. The reason being, "sponge" is an actually useful tool and for me it's pretty much the only useful tool in the package. So I need to install whole moreutils package on ubuntu to have "sponge" and I have to clog my bin namespace with all this trash. It would be mildly annoying from the perfectionist point of view, but ok. But there is also "parallel" command in this package, which is trash as well. Meanwhile GNU "parallel" is not trash, but an actually useful program, maybe more useful than "sponge", and it comes in a separate package. So I either have to do some tinkering, or I have to choose which one I need more: "sponge" or "parallel" and install only one package. It's year 2020. This is stupid. This is by the way the reason why GraphicsMagick is better than ImageMagick (I still use the latter, though, because it doesn't cause the same problems for me as moreutils does, and it's just more popular than GM).
- nickthemagicman 6y agoIs there a way to namespace on install? Or does that have to be configured in the package?
- krick 6y agoI am not aware of such a way with apt on ubuntu. It can be different with some other package managers, obviously.
- andrewshadura 6y agodpkg-divert
- waynesonfire 6y agoDocker
- kps 6y agoYou put them in different directories, e.g. Solaris-style /opt/package/…, and arrange your PATH accordingly.
- thekaleb 6y agots seems to fit the Unix philosophy of do one thing and do one thing well quite nicely. It works pretty well for writing to logs.
- deleted 6y ago[deleted]
- vagab0nd 6y agoIn case you are like me and didn't get the sponge example, the difference from a shell redirect is it doesn't truncate the input so allows in-place modification of the input/output file.
- aasasd 6y agoWhile we're on the topic and on the wave of modern riffs on classic tools, personally I'm pining for a remake of xargs—because I never can whack it into submission with anything more complex than `xargs rm`. Specifically, passing multiple arguments from the input to the called command apparently just can't be properly done, at least not on OSX.
- nick0garvey 6y agoparallel may be what you are looking for: https://www.gnu.org/software/bash/manual/html_node/GNU-Parallel.html https://www.gnu.org/software/bash/manual/html_node/GNU-Paral... The defaults are a lot saner, and it's easier to pass arguments how you want.
- aasasd 6y agoHmm, I guess I considered Unix utils rather narrowly specialized, so `parallel` meant ‘my CPU got too many free cores’ for me. I'll take a closer look at it, thanks.
- zzo38computer 6y agoSome of these I do not find so useful and stuff, although I do use ts, and sometimes sponge.
- JdeBP 6y agoYes, they weren't written "when Unix was young", but people had thought to write some of them over a decade before this toolset. lckdo and ts came long after setlock and cyclog (also the precursor to multilog) from the 1990s. * https://cr.yp.to/daemontools/setlock.html https://cr.yp.to/daemontools/setlock.html * http://cr.yp.to/daemontools/multilog.html http://cr.yp.to/daemontools/multilog.html * http://cr.yp.to/daemontools/upgrade.html http://cr.yp.to/daemontools/upgrade.html * http://jdebp.uk./Softwares/nosh/guide/commands/cyclog.xml#COMPATIBILITY http://jdebp.uk./Softwares/nosh/guide/commands/cyclog.xml#CO...