9 ms·

Are there any particularly clever uses for named pipes in shell scripts? I don't think I've ever seen any.

sikhnerd 13y ago
Only time I've used it is as a poor man's screen -x mkfifo file; script -f file Other user simply does cat file
ars 13y ago
Example 1: In one shell: mplayer -input file=named_pipe In a different shell: echo 'pause' > named_pipe or: echo 'quit' > named_pipe So you can control your music player easily. (I bind the echo script to hotkeys.) Example 2: mysql LOAD DATA INFILE wants a filename. So I made a named pipe and sent my data there (putting that command in the background since it will hang till the data is read). Then I use that named pipe as the filename for load data. Example 3: You want to send your data over ssh, but you also want to send commands over ssh (you can't do both at the same time, except in the command line which might not be enough). You could send the data over a second ssh session - but then you can't do your preliminary commands first. So you make a named pipe on the server and send the data to it in one ssh session. Then you open a second ssh session, do your preliminary commands, then your data command using the named pipe as a filename. (There are other ways to do this with renaming file descriptors in a shell, but a named pipe is easier, and much more flexible.) Example 4: You want to start a long running process that waits for data. You let it read from the named pipe, and then you can write to the named pipe from any other program as needed.
- Hello71 13y ago
  > You want to send your data over ssh, but you also want to send commands over ssh (you can't do both at the same time, except in the command line which might not be enough). You could send the data over a second ssh session - but then you can't do your preliminary commands first. SSH already has a Control facility for this.
  - ars 13y ago
    That won't help. How would you feed the data to stdin of a command run in a different process? The Control facility is for multiplexing the data streams, but that solves a different problem.
  - nine_k 13y ago
    The Control facility lets you reuse the connection, bit not the session. So you still need two sessions, say, one ssh and one scp.
- gwern 13y ago
  I actually did exactly this exactly a few years ago (I thought I was so clever). I had a video with hardcoded subtitles, which I wanted to transcribe; I couldn't just let mplayer run because I cannot transcribe in realtime, and slowing it down would still be a problem because it would take much more time and would necessitate backtracking. So, I set up a FIFO like described, and then I bound the key '1' in Emacs to echo 'p' to it. Voila. I could seamlessly pause and unpause the video while typing away furiously. The strategy was quite a hack, but it worked very well for my purpose.
smartaleckkill 13y ago
I use a FIFO to pipe alpine's new mail alert to notifyd; not sure how clever that is, but it works for me.
- ars 13y ago
  Can you post the script? I'd love to use it.
  - smartaleckkill 13y ago
    Used to use my own wee C prog that just passed the alert unparsed to libnotify; currently using a modified version of this far superior shell script (which I can't take any credit for): https://github.com/Bruce-Connor/alpine-osd-notify/blob/master/alpine-notify.sh https://github.com/Bruce-Connor/alpine-osd-notify/blob/maste... (You have to set newmail-fifo-path in your pinerc.)
cynwoody 13y ago
In this discussion† from yesterday, the task of finding the longest path below a directory came up. You can see both a clumsy solution, using a named pipe and iterative Python, and an elegant one-liner (in a reply), using an anonymous pipe and functional Python. This is not to say named pipes do not have their uses. Just that you should think before reaching for them first. †https://news.ycombinator.com/item?id=6510640 https://news.ycombinator.com/item?id=6510640
gizmo686 13y ago
I don't think named pipes work particuarly well in shell scripts. A program writing to a pipe will not terminate until the output is consumed. However, consuming the output will not happen until the program finishes and the script arrives that the program that reads from the pipe. Similarly, a program reading from the pipe will not terminate until the program writing to it does. If you start reading before you start writing, then you simply wait. Anonymous pipes solves this problem by launching all of the programs at the same time. You could work around this by telling the shell to run the command in the background (append & in bash). Having said that, I have had occasion to use named pipes in a shell script. Basically, I had two programs and I wanted to determine which lines were in one, but not both, of their outputs. Using named pipes, I was able to do something like: mkfifo foo ./prog1 > foo & ./prog2 > foo & cat foo | sort | uniq -u Effectively, this provides a way to combine the output of multiple commands. I took it as a matter of faith that doing this won't interleave lines. I assume that as long as both programs flush only at line breaks then there is not any problem.
- derleth 13y ago
  Does: sort < foo | uniq -u not work with named pipes?
  - dllthomas 13y ago
    It works fine. The advantage to cat foo | sort | uniq -u is that it is (marginally) easier to drop in another filter in front of sort.
    - knz42 13y ago
      This argument is less strong in the face of: <foo sort | uniq -u (redirections can be at the beginning too.) Also: sort -u instead of sort|uniq.
      - dllthomas 13y ago
        It's still a reason to disprefer the form that was given. I don't see any reason to prefer the cat version over yours, except habits, though. That said, there's only weak reason to avoid the cat.
smlacy 13y ago
Sort a file in "parallel" (note quotes!) in a multicore machine (note I'm showing 2 cores but this could be generalized to more): mkfifo split_aa mkfifo split_ab mkfifo sorted_aa mkfifo sorted_ab split -n2 input_file split_ & sort split_aa >> sorted_aa & sort split_ab >> sorted_ab & sort -m sorted_aa sorted_bb > final_sorted It's left as an exercise for the reader to determine if this actually saves any time, and if this technique could be used to implement a truly parallel sort.
- timClicks 13y ago
  What about? sort --parallel=2 input_file
  - smlacy 13y ago
    Well, yeah, that's the easy way. :) This is mostly to illustrate a technique of "parallel processing using fifos and split input" It's almost like a miniature mapreduce running locally. See also "bashreduce" (https://github.com/erikfrey/bashreduce https://github.com/erikfrey/bashreduce) which I didn't write.
christianmann 13y ago
Poor man's (unauthenticated!) remote shell: mkfifo /tmp/buf; cat /tmp/buf | /bin/sh -i 2>&1 | nc -lp 31337 > /tmp/buf And then, in another terminal: nc 127.0.0.1 31337
unhammer 13y ago
If you have a slow-starting program, you can use named pipes as its stdin/stdout. Then write a little script that writes to the inpipe and reads from the outpipe (interleavedly, so as not to fill up the pipe buffer, I think that's about 4k or so). Now that little script starts up "instantly" compared to the original slow-starting program. http://mywiki.wooledge.org/NamedPipes http://mywiki.wooledge.org/NamedPipes shows an outline of it. http://mywiki.wooledge.org/ProcessManagement#I_want_to_process_a_bunch_of_files_in_parallel.2C_and_when_one_finishes.2C_I_want_to_start_the_next._And_I_want_to_make_sure_there_are_exactly_5_jobs_running_at_a_time http://mywiki.wooledge.org/ProcessManagement#I_want_to_proce.... is another neat trick: when a background process ends, make it echo to some pipe, some other "controller" process is continually reading from that pipe. Since the read is blocking, it'll wait patiently for the echo until it does whatever it should do (maybe even based on whatever it read from the pipe). So much less hacky than sleep-polling :-)

Only time I've used it is as a poor man's screen -x mkfifo file; script -f file Other user simply does cat file

ars 13y ago

Example 1: In one shell: mplayer -input file=named_pipe In a different shell: echo 'pause' > named_pipe or: echo 'quit' > named_pipe So you can control your music player easily. (I bind the echo script to hotkeys.) Example 2: mysql LOAD DATA INFILE wants a filename. So I made a named pipe and sent my data there (putting that command in the background since it will hang till the data is read). Then I use that named pipe as the filename for load data. Example 3: You want to send your data over ssh, but you also want to send commands over ssh (you can't do both at the same time, except in the command line which might not be enough). You could send the data over a second ssh session - but then you can't do your preliminary commands first. So you make a named pipe on the server and send the data to it in one ssh session. Then you open a second ssh session, do your preliminary commands, then your data command using the named pipe as a filename. (There are other ways to do this with renaming file descriptors in a shell, but a named pipe is easier, and much more flexible.) Example 4: You want to start a long running process that waits for data. You let it read from the named pipe, and then you can write to the named pipe from any other program as needed.

Hello71 13y ago
> You want to send your data over ssh, but you also want to send commands over ssh (you can't do both at the same time, except in the command line which might not be enough). You could send the data over a second ssh session - but then you can't do your preliminary commands first. SSH already has a Control facility for this.
- ars 13y ago
  That won't help. How would you feed the data to stdin of a command run in a different process? The Control facility is for multiplexing the data streams, but that solves a different problem.
- nine_k 13y ago
  The Control facility lets you reuse the connection, bit not the session. So you still need two sessions, say, one ssh and one scp.
gwern 13y ago
I actually did exactly this exactly a few years ago (I thought I was so clever). I had a video with hardcoded subtitles, which I wanted to transcribe; I couldn't just let mplayer run because I cannot transcribe in realtime, and slowing it down would still be a problem because it would take much more time and would necessitate backtracking. So, I set up a FIFO like described, and then I bound the key '1' in Emacs to echo 'p' to it. Voila. I could seamlessly pause and unpause the video while typing away furiously. The strategy was quite a hack, but it worked very well for my purpose.

smartaleckkill 13y ago

I use a FIFO to pipe alpine's new mail alert to notifyd; not sure how clever that is, but it works for me.

ars 13y ago
Can you post the script? I'd love to use it.
- smartaleckkill 13y ago
  Used to use my own wee C prog that just passed the alert unparsed to libnotify; currently using a modified version of this far superior shell script (which I can't take any credit for): https://github.com/Bruce-Connor/alpine-osd-notify/blob/master/alpine-notify.sh https://github.com/Bruce-Connor/alpine-osd-notify/blob/maste... (You have to set newmail-fifo-path in your pinerc.)

cynwoody 13y ago

In this discussion† from yesterday, the task of finding the longest path below a directory came up. You can see both a clumsy solution, using a named pipe and iterative Python, and an elegant one-liner (in a reply), using an anonymous pipe and functional Python. This is not to say named pipes do not have their uses. Just that you should think before reaching for them first. †https://news.ycombinator.com/item?id=6510640 https://news.ycombinator.com/item?id=6510640

gizmo686 13y ago

I don't think named pipes work particuarly well in shell scripts. A program writing to a pipe will not terminate until the output is consumed. However, consuming the output will not happen until the program finishes and the script arrives that the program that reads from the pipe. Similarly, a program reading from the pipe will not terminate until the program writing to it does. If you start reading before you start writing, then you simply wait. Anonymous pipes solves this problem by launching all of the programs at the same time. You could work around this by telling the shell to run the command in the background (append & in bash). Having said that, I have had occasion to use named pipes in a shell script. Basically, I had two programs and I wanted to determine which lines were in one, but not both, of their outputs. Using named pipes, I was able to do something like: mkfifo foo ./prog1 > foo & ./prog2 > foo & cat foo | sort | uniq -u Effectively, this provides a way to combine the output of multiple commands. I took it as a matter of faith that doing this won't interleave lines. I assume that as long as both programs flush only at line breaks then there is not any problem.

derleth 13y ago
Does: sort < foo | uniq -u not work with named pipes?
- dllthomas 13y ago
  It works fine. The advantage to cat foo | sort | uniq -u is that it is (marginally) easier to drop in another filter in front of sort.
  - knz42 13y ago
    This argument is less strong in the face of: <foo sort | uniq -u (redirections can be at the beginning too.) Also: sort -u instead of sort|uniq.
    - dllthomas 13y ago
      It's still a reason to disprefer the form that was given. I don't see any reason to prefer the cat version over yours, except habits, though. That said, there's only weak reason to avoid the cat.

smlacy 13y ago

Sort a file in "parallel" (note quotes!) in a multicore machine (note I'm showing 2 cores but this could be generalized to more): mkfifo split_aa mkfifo split_ab mkfifo sorted_aa mkfifo sorted_ab split -n2 input_file split_ & sort split_aa >> sorted_aa & sort split_ab >> sorted_ab & sort -m sorted_aa sorted_bb > final_sorted It's left as an exercise for the reader to determine if this actually saves any time, and if this technique could be used to implement a truly parallel sort.

timClicks 13y ago
What about? sort --parallel=2 input_file
- smlacy 13y ago
  Well, yeah, that's the easy way. :) This is mostly to illustrate a technique of "parallel processing using fifos and split input" It's almost like a miniature mapreduce running locally. See also "bashreduce" (https://github.com/erikfrey/bashreduce https://github.com/erikfrey/bashreduce) which I didn't write.

christianmann 13y ago

Poor man's (unauthenticated!) remote shell: mkfifo /tmp/buf; cat /tmp/buf | /bin/sh -i 2>&1 | nc -lp 31337 > /tmp/buf And then, in another terminal: nc 127.0.0.1 31337

unhammer 13y ago

If you have a slow-starting program, you can use named pipes as its stdin/stdout. Then write a little script that writes to the inpipe and reads from the outpipe (interleavedly, so as not to fill up the pipe buffer, I think that's about 4k or so). Now that little script starts up "instantly" compared to the original slow-starting program. http://mywiki.wooledge.org/NamedPipes http://mywiki.wooledge.org/NamedPipes shows an outline of it. http://mywiki.wooledge.org/ProcessManagement#I_want_to_process_a_bunch_of_files_in_parallel.2C_and_when_one_finishes.2C_I_want_to_start_the_next._And_I_want_to_make_sure_there_are_exactly_5_jobs_running_at_a_time http://mywiki.wooledge.org/ProcessManagement#I_want_to_proce.... is another neat trick: when a background process ends, make it echo to some pipe, some other "controller" process is continually reading from that pipe. Since the read is blocking, it'll wait patiently for the echo until it does whatever it should do (maybe even based on whatever it read from the pipe). So much less hacky than sleep-polling :-)