6 ms·
Named Pipes in Bash
- zokier 13y agoVery useful, hopefully I remember this when I need it next time (that being the bane of all bash cleverness). The >(foo) would benefit from some extra explanation though, currently it is not very clear how it differs from regular |foo.
- bartbes 13y ago>(foo) turns into a file name, so it can be passed to things that absolutely want a file name to write to. And multiple occurrences of this should work as you'd expect.
- e12e 13y agoInteresting, indeed it does: # define a function that echoes it's first argument: $ arg1() { echo $1 ; } $ arg1 test test $ arg1 <(echo hello) /dev/fd/63
- Hello71 13y agoecho <(:)
- e12e 13y agoThanks, I've never investigated ":" beyond noting it's presence in fork bombs. : Null command. No effect; the command does nothing. Exit Status: Always succeeds.
- taejo 13y agoBasically a builtin for `true`. Its use in forkbombs is unrelated except for the fact that it's a valid character in a bash function name. Defining `:(){:|:&)` obviously shadows the builtin.
- pstadler 13y agoYou can create named pipes explicitly: $ mkfifo ~/pipe $ ./background.sh > ~/pipe & $ tail -f ~/pipe http://www.linuxjournal.com/article/2156 http://www.linuxjournal.com/article/2156
- runn1ng 13y agoYes, but I have always found that cumbersome and I was never sure which program has which pipe, and what will exactly happen if one of the reading/writing programs dies. (I am still not sure.) So I am glad that what OP wrote exists.
- mtdewcmu 13y agoTrue. I've known about that feature of bash, but I have found precious few good opportunities to use it. As someone mentioned below, if a program expects a file and you give it a pipe, it may try to seek and cause the whole thing to fail. So trying to use it is always a gamble. Moreover, a named pipe is an object in the filesystem that bash has to create, only to remove it a moment later. So it is not really any cleaner than using a temp file. The only case where you'd clearly benefit is when the amount of data is too big to justify temporarily writing it out to a file.
- thwarted 13y agobash doesn't create named pipe filesystem objects when you use process substitution, it passes file descriptors via the dynamically created kernel managed /dev/fd paths. There is nothing to clean up.
- mtdewcmu 13y agoOh, I think I see what you're saying. So calling them named pipes is inaccurate, then. If I'm not mistaken, using features of the /dev filesystem would be dependent on Linux, then. Either way, it isn't really a very robust design. Not like a pipe returned by pipe(2) and passed to a program that is expecting it. Simple unnamed pipes are a fundamental part of unix. Here's what the man page says: Process substitution is supported on systems that support named pipes (FIFOs) or the /dev/fd method of naming open files. It takes the form of <(list) or >(list). The process list is run with its input or output connected to a FIFO or some file in /dev/fd. The name of this file is passed as an argument to the current command as the result of the expansion. If the >(list) form is used, writing to the file will provide input for list. If the <(list) form is used, the file passed as an argument should be read to obtain the output of list. So it's done with /dev/fd if that's available, which basically means Linux with that filesystem mounted. Otherwise it falls back on named pipes.
- saljam 13y agoThere's a caveat with process substitution; the command can't seek() on the input file. If if a command, for example, needs to do 2 passes, it'll will either have to be 'smart' enough to stick the output in a buffer or fail.
- eatitraw 13y agoYes, this is a problem in bash. In zsh you can use temporary files for process substitution, like this: vimdiff =(ls /bin) =(ls /usr/bin) You can use it even from bash: zsh -c 'vimdiff =(ls /bin) =(ls /usr/bin)'
- deleted 13y ago[deleted]
- e12e 13y agoActually vimdiff seems to have no problem with: vimdiff <(ls /bin) <(ls /usr/bin)
- dllthomas 13y agoThis and so much more is in http://www.tldp.org/LDP/abs/html http://www.tldp.org/LDP/abs/html - well worth perusing if you're needing to do a lot of bash.
- recuter 13y agoFuture readers, #bash on freenode says of this link: < greybot> The infamous "Advanced" Bash Scripting Guide should be avoided unless you know how to filter out the junk. It will teach you to write bugs, not scripts. In that light, the BashGuide was written: http://mywiki.wooledge.org/BashGuide http://mywiki.wooledge.org/BashGuide They felt strongly enough to learn their bot of it. Keep this in mind, I haven't read much of either resource yet.
- dllthomas 13y agoInteresting. I'd not heard anything negative about it, and found it useful both in my day-to-day BASH-ing and in a position where I maintained a huge pile of BASH scripts. It may very well be that I knew how to filter out the junk. There is certainly a lot that doesn't belong in scripts but is quite useful interactively.
- perlgeek 13y agoAre these really named pipes? And if yes, what are the names? Kinda looks like anonymous pipes to my untrained eyes.
- gnaritas 13y agoThey're anonymous named pipes. Do this in a shell to see the name: echo <(true)
- Pitarou 13y agoanonymous named? We really need to refactor our terminology. :)
- hamburglar 13y agoAnonymous in that their name doesn't matter and you need not care about it. They are implemented using pipes with actual names, but the names aren't important from the shell user's point of view.
- gnaritas 13y agoWhy, that's exactly what it is? It's a named pipe, but not one used explicitly by you but implicitly by the process substitution feature to pass a file, which of course has a name, to another process. It's only anonymous to you because you didn't name it, to the system it's an ordinary named pipe.
- taway2012 13y agoPretty cool. Related: anybody has ideas to achieve the following? display the stdout of a program on the console and also write a gzipped version of stdout to a file. I currently jump through some hoops (involving two term windows) to achieve. 'tee -a' to a named pipe in one window and in another window, 'gzip <' from that named pipe. Would prefer a single-window, single-command solution.
- inevitable 13y agoTry this: echo testing | tee -a >(gzip >test.gz)
- _paulc 13y agols -al | gzip -c | tee out.gz | gzip -dc
- runejuhl 13y agoSomething like this? ls -al | tee >(cat) | gzip -c
- runejuhl 13y ago...or you could run something like cat /proc/`pidof program`/fd/2 That, of course, won't eliminate the need for a second terminal. (change to your needs, stdin==1, stdout==2, stderr=3)
- ColinWright 13y agoWhat's wrong with: cmd | tee >( gzip - > cmd.out.gz )
- taway2012 13y agoYeah, I think this is probably going to work. Thx! PS: 'inevitable', you made the same suggestion. But you seem to be hellbanned.
- panzi 13y agoIf I would have seen this one or two hours ago it could have spared me hate on irc://irc.freenode.net/bash
- vsbuffalo 13y agoI wrote a post about this a while back, which may be of some help to some. http://vincebuffalo.com/2013/08/08/the-mighty-named-pipe.html http://vincebuffalo.com/2013/08/08/the-mighty-named-pipe.htm...
- nkurz 13y agoGreat article, thanks! You should submit it separately if you haven't. Noticed a typo in the title that you might fix first: "Using Names Pipes"
- kps 13y agoThis 'bash' feature, among others, was in ksh before bash existed.