9 ms·

Why “process substitution” is a late feature in Unix shells

168 points by r4um 5y ago

veltas 5y ago
Didn't even know about process substitution, had been using fifo's to achieve this!
- ogogmad 5y ago
  Fish tries to use FIFOs to emulate process substitution, and it leads to deadlock. Not sure why. By default, Fish actually runs the processes in a strict sequence. But this is to avoid the above deadlock situation. And it therefore isn't process substitution.
  - ridiculous_fish 5y ago
    To be sure, fish runs external processes in parallel. It's only internal functions which are serialized against each other. I'm hoping to lift this restriction eventually. This SO answers a bit about the issues with FIFOs and fish shell. Basically it comes down to evaluating redirections before fork instead of after, which is because fish uses threads. https://stackoverflow.com/questions/61946995/ls-fifo-blocks-fish-shell/61947293#61947293 https://stackoverflow.com/questions/61946995/ls-fifo-blocks-...
    - ogogmad 5y ago
      Thanks. Love the Fish shell! Does this mean that every time you direct output to a file, like so: the_prog > the_file The output of the_prog passes into Fish first and then into the_file? Because then if the write to the_file blocks then Fish would itself get blocked.
zokier 5y ago
The usability of unixy shells generally falls down a cliff when you need to deal with more than just one input one output. The awkwardness in trying to shoehorn process substitution is just one of the examples of that.
- rscnt 5y ago
  do you know alternatives to that? I assume PowerShell but don't know if there's anything beyond that.
  - zokier 5y ago
    I remember seeing some academic work on extending shell sematics to more complicated pipe networks, but nothing particularly promising. In industry, I think that is generally the point where people pick up "real" programming language instead of trying to work in shell; on top of my head I imagine golang with its channels and goroutines to be particularly well suited for these sort of problems. I can't say if there is something in golang that shells could adapt somehow.
    - sillysaurusx 5y ago
      But of course you can do the same thing in 30 seconds that Go would take 30 minutes for. Especially if you’re trying to process-substitute a shell pipeline, not just one command.
  - lnanek2 5y ago
    You can always setup some named pipes. E.g.: mkfifo named_pipe echo "Hi" > named_pipe & cat named_pipe I used to do this in bash scripts to keep them cleaner and the lines simpler.
    - Too 5y ago
      How does error handling together with this work? Can pipefail catch this or does one explicitly need to ‘wait’ for the background processes and check them there?
      - cerved 5y ago
        Guessing that pipefail cares 0% of which filedescriptors are used and only the exit codes of processes
TorKlingberg 5y ago
Great read, but worth noting from the end of the article that "late feature" here means it was added in the early '90s. The late addition to Unix that surprises me is ssh. It was only invented in the late '90s. Before that everyone used unencrypted remote shells like telnet. Encryption in general was in a pretty bad state in the '90s: original http was unencrypted and early mobile phone standards that are still in use have very weak encryption.
- teddyh 5y ago
  Telnet has encryption options, it just wasn't widely implemented.
- zokier 5y ago
  Everything was unencrypted until late 90s (and in many cases until late 00s). Email (both smtp and pop3/imap), irc, web, gopher, telnet, ftp, local disks, removable storage, network storage (smb/nfs etc), everything. Computing and the internet was much nicer place, there wasn't such an adversial attitude where everything would be broken just because its out there like today.
  - mturmon 5y ago
    Yeah, on the time sharing Unix systems I would use in the 80s and 90s, everyone’s home directory (and most everything under it) was world readable by default. You could change the permissions, but most people didn’t. I feel like those old folks who tell of a time when people didn’t bother to lock their doors at night.
    - throwawayboise 5y ago
      I think on Ubuntu/Debian this is still the default; the UMASK in /etc/login.defs is 022.
      - mongol 5y ago
        I think OpenSuse too. I recently converted mine to user private group handling https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/4/html/reference_guide/s1-users-groups-private-groups https://access.redhat.com/documentation/en-us/red_hat_enterp...
vimax 5y ago
A fun trick is using the `paste` command with process substitution to combine lines of output from different commands. ./cmd1 outputs: a b c ./cmd2 outputs: 1 2 3 paste <(./cmd1) <(./cmd2) outputs: a 1 b 2 c 3 You can then use xargs to use each line as arguments for another command: paste <(./cmd1) <(./cmd2) | xargs -L1 ./cmd3 calls: ./cmd3 a 1 ./cmd3 b 2 ./cmd3 c 3
- sillysaurusx 5y ago
  Should’ve been named zip. Is there an unzip? But of course it can’t be named zip because that’s the compressor, so I’m really asking whether there’s an unpaste. I guess the closest is awk {print $1;}, but it makes me wince every time I write it. (I already know that would give syntax errors as written here.)
  - vimax 5y ago
    Depending on the data, the right options for xargs would work, and there's the pr command I haven't used much.
    - sillysaurusx 5y ago
      Hmm. Does xargs give you a way to skip the first arg? xargs -n 2 I {} almost works, but you’d need a way to extract a specific arg out of the {}. Which I just realized I don’t know how to do. “(at 0 {})” would be a nice construction. But then of course now you’re writing xargs -n 2 -I {} echo (at 1 {}) instead of just awk {print $2}, so it’s mission failed successfully.
      - quicklime 5y ago
        If you're just looking for a single column, then as zamfi mentioned, cut does the job: $ paste <(printf "a\nb\nc\n") <(printf "1\n2\n3\n") | cut -f2 1 2 3 It makes sense that "cut" and "paste" would be inverse operations, but too bad that GUIs repurposed these words for clipboard operations. But awk can do both, and does so in a single pass: $ paste <(printf "a\nb\nc\n") <(printf "1\n2\n3\n") | awk '{ print $1 > "letters"; print $2 > "numbers" }' $ cat letters a b c $ cat numbers 1 2 3
sillysaurusx 5y ago
I love zsh’s take on process substitution: cat =(echo foo) It’s almost identical to <(echo foo), but crucially it drains the output before performing the substitution. It’s like <(... | sponge), but somehow much more reliable. I’ve used it in a few situations where process substation failed, though being old and feeble I can’t remember why it failed. But for example you can copy the file safely, knowing that it will contain the full output of the shell pipeline, unlike process substitution. I don’t even know the name of =(...). Hmm. Sponge substitution?
- sillysaurusx 5y ago
  Thought of an example. vs cp =( ( echo foo; sleep 1; echo bar) | sponge) ) foobar The first will complete instantly and result in a file named foobar but whose contents is either zero length or foo, depending on how unlucky you are. The second takes one second to complete and results in a file named foobar containing foobar. It’s strange that vanilla process substitution can’t seem to solve this at all.
  - LukeShu 5y ago
    I'm not sure what you're trying to say here. With GNU cp on Linux, the commands cp <( ( echo foo; sleep 1; echo bar) | sponge ) foobar cp =( ( echo foo; sleep 1; echo bar) | sponge ) foobar both have the same result: the file "foobar" containing the string "foo\nbar\n".
    - sillysaurusx 5y ago
      I didn't test it. You're right, I'm simply mistaken. Here's a better (but awkward) illustration of the problem: :~$ python3 -c 'import sys; import shutil; shutil.copy(sys.argv[1], "foobar")' < (echo foo ; sleep 1 ; echo bar) Traceback (most recent call last): File "<string>", line 1, in <module> File "/opt/homebrew/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/shutil.py", line 427, in copy copyfile(src, dst, follow_symlinks=follow_symlinks) File "/opt/homebrew/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/shutil.py", line 257, in copyfile raise SpecialFileError("`%s` is a named pipe" % fn) shutil.SpecialFileError: `/dev/fd/11` is a named pipe :~$ python3 -c 'import sys; import shutil; shutil.copy(sys.argv[1], "foobar")' =(echo foo ; sleep 1 ; echo bar) Notice the difference: :~$ echo =(echo foo ; sleep 1; echo bar) /tmp/zshbusU5D :~$ echo <(echo foo ; sleep 1; echo bar) /dev/fd/11 /dev/fd/11 is a named pipe, whereas /tmp/zshbEF1D9 is a plain file. There are situations where named pipes fail, where you actually do need a plain file, and I can't think of a single one offhand. :) "I've run into it in the past" is all I can say. And the above snippet at least gives one specific example of a hypothetical case.
renewiltord 5y ago
In practice I end up caching the output often. I have used process substitution but the iteration process feels more useful to me if I've slowly built up data and I can inspect the internal pieces each time and reuse them in different ways. But I can see if it's relatively fast. I like it. I just don't end up using it often.
unixhero 5y ago
Which is the best she'll, fish or zsh?//
- yjftsjthsd-h 5y ago
  There is no possible way to usefully answer that question as given; every shell has its own advantages and disadvantages. Plain bourne is universal but bare-bones, bash is mostly ubiquitous on Linux, zsh is powerful but just different enough to occasionally bite you, fish is very user friendly but doesn't even try to be compatible with anything else, ksh is a nice option and is built-in on BSDs, dash sucks for interactive work but is great for running scripts...
- nmz 5y ago
  tcl
  - errcorrectcode 5y ago
    Nawh, it's all about tk. ;)
- errcorrectcode 5y ago
  When everything is both nails and screws, a hammer isn't the best tool for everything (unless it has a screwdriver on the end and the hammer part is big enough to be actually useful as a hammer).
shuffel 5y ago
For whatever reason I can never remember the syntax of <(command) and end up "rediscovering" it every year. It's seldom used but when it's needed it's rather elegant. Another somewhat related useful bash feature is using this form: wc <<< 'a b c' instead of: echo 'a b c' | wc
- JNRowe 5y ago
  With the caveat that the here string causes a tempfile to be written¹, so they're not quite equivalent. How much that matters for your use cases though is a different question, but it is worth thinking about if you're doing lots of repeated calls. ¹ With Bash v5 it may use a pipe if the data is small enough, but you can't guarantee people will have that because of GPLv3 phobia. I believe it is always a tempfile with zsh.
iamevn 5y ago
I quite like how fish does process substitution. foo (bar | psub) Instead of syntax it's just a plain function. docs: https://fishshell.com/docs/current/cmds/psub.html https://fishshell.com/docs/current/cmds/psub.html source of psub.fish: https://github.com/fish-shell/fish-shell/blob/master/share/functions/psub.fish https://github.com/fish-shell/fish-shell/blob/master/share/f...
- limoce 5y ago
  But fish's psub only supports file or named pipe, not the one mentioned in the post, which is described as the "the best way".
  - zanchey 5y ago
    It doesn't even support named pipes - the function's been broken for years. Interestingly it doesn't actually seem to matter a great deal, at least now that temporary files are usually on a fast disk.
- ogogmad 5y ago
  While the syntax might be nice, strictly speaking, process substitution isn't currently possible in Fish: https://news.ycombinator.com/item?id=29845845 https://news.ycombinator.com/item?id=29845845
jakub_g 5y ago
An unfortunate thing is that process substitution does not work in git bash on Windows. (at least it was the case last time I tested; googling around I found a random comment in random github repo saying it's been fixed in 2.25 but I don't have a laptop handy to test it now).
- cerved 5y ago
  it does
chubot 5y ago
Fun fact, in bash you lose the exit codes of process subs. It doesn't even wait() on those processes in many cases. So there is no way to abort a bash script if something like <(sort nonexistent) fails. OSH lets you opt into stricter behavior: $ osh -c set -e shopt -s oil:basic diff <(sort left) <(sort nonexistent) echo "should not get here"' ' sort: cannot read: nonexistent: No such file or directory diff <(sort left) <(sort nonexistent) ^~ [ -c flag ]:1: fatal: Exiting with status 2 (command in PID 29359) In contrast, bash will just keep going and ignore failure. You can also get all the exit codes with @_process_sub_status, which is analogous to PIPESTATUS. (I should probably write a blog post about this: http://www.oilshell.org/blog/ http://www.oilshell.org/blog/)
- andreyv 5y ago
  > So there is no way to abort a bash script if something like <(sort nonexistent) fails. The process ID of the last executed background command in Bash is available as $!. cat <(sort nonexistent) wait $! || echo fail gives sort: cannot read: nonexistent: No such file or directory fail
  - mkdirp 5y ago
    > The process ID of the last executed background command in Bash is available as $!. After a quick search it doesn't seem like this is an option for anything but the last executed background process. So if you have the typical `diff <(left) <(right)`, there is no easy way to get the pid of `<(left)`. I suppose you could output the pid to a file, and use it from there, though I haven't tested if that would work.
errcorrectcode 5y ago
0. Process substitution is a potential DoS vector as it could take up all of RAM and/or disk space. 1. Also, not all commands are compatible with it, especially if they need rewinding or reopening. diff has issues with using it for both arguments often. It's likely the use of memory mapped files, but I could be wrong. 2. Shells ought to implement a flag for process substitution to allow temporary files to reside on disk for the lifetime of the command line. This way, it can operate on extremely large files.
FedericoRazzoli 5y ago
This is a great explanation! I wonder many times why we had to play with obscure xargs instead of being able to pipe a command output to an argument.
kazinator 5y ago
The claim is false; process substitution can be cobbed together with named fifos,* and those are "ancient". Only problem that those are temporary objects that have to be created in the file system, and cleaned up. However, temporary objects (files, no fifos) are also used in here doc implementations. Process substitution is a late feature simply because the creativity juice in Unix (tm) dried up some time before the middle 1990's, leaving the FOSS reimplementations of Unix to carry the development torch. Those projects had to balance among other goals like quality/robustness and compatibility. (If we look at the quality of the FOSS tools compared to the Unix originals, we could also remark that "quality and robustness was late in coming to Unix". But we equivocate on Unix, because GNU stands for GNU is Not Unix!) Features appearing in FOSS utilities like GNU Bash take time to make into Unix (tm). Process substitution is not yet in the standard, therefore it is not in in fact in Unix (tm). Shell scripting is a conservative activity. The language isn't very good and so improving it is like kicking a dead horse in some ways; the most important matter in any new shell release is that old scripts keep working. (Like configuration scripts for the build systems of nicer languages). --- * See GNU Bash manual: https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Process-Substitution https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash....: " Process substitution is supported on systems that support named pipes (FIFOs) or the /dev/fd method of naming open files. "
- xeromal 5y ago
  Yeah, but Saget could still have had a damaged heart from it regardless of Clapton's experience.

168 points by r4um 5y ago

veltas 5y ago

Didn't even know about process substitution, had been using fifo's to achieve this!

ogogmad 5y ago
Fish tries to use FIFOs to emulate process substitution, and it leads to deadlock. Not sure why. By default, Fish actually runs the processes in a strict sequence. But this is to avoid the above deadlock situation. And it therefore isn't process substitution.
- ridiculous_fish 5y ago
  To be sure, fish runs external processes in parallel. It's only internal functions which are serialized against each other. I'm hoping to lift this restriction eventually. This SO answers a bit about the issues with FIFOs and fish shell. Basically it comes down to evaluating redirections before fork instead of after, which is because fish uses threads. https://stackoverflow.com/questions/61946995/ls-fifo-blocks-fish-shell/61947293#61947293 https://stackoverflow.com/questions/61946995/ls-fifo-blocks-...
  - ogogmad 5y ago
    Thanks. Love the Fish shell! Does this mean that every time you direct output to a file, like so: the_prog > the_file The output of the_prog passes into Fish first and then into the_file? Because then if the write to the_file blocks then Fish would itself get blocked.

zokier 5y ago

The usability of unixy shells generally falls down a cliff when you need to deal with more than just one input one output. The awkwardness in trying to shoehorn process substitution is just one of the examples of that.

rscnt 5y ago
do you know alternatives to that? I assume PowerShell but don't know if there's anything beyond that.
- zokier 5y ago
  I remember seeing some academic work on extending shell sematics to more complicated pipe networks, but nothing particularly promising. In industry, I think that is generally the point where people pick up "real" programming language instead of trying to work in shell; on top of my head I imagine golang with its channels and goroutines to be particularly well suited for these sort of problems. I can't say if there is something in golang that shells could adapt somehow.
  - sillysaurusx 5y ago
    But of course you can do the same thing in 30 seconds that Go would take 30 minutes for. Especially if you’re trying to process-substitute a shell pipeline, not just one command.
- lnanek2 5y ago
  You can always setup some named pipes. E.g.: mkfifo named_pipe echo "Hi" > named_pipe & cat named_pipe I used to do this in bash scripts to keep them cleaner and the lines simpler.
  - Too 5y ago
    How does error handling together with this work? Can pipefail catch this or does one explicitly need to ‘wait’ for the background processes and check them there?
    - cerved 5y ago
      Guessing that pipefail cares 0% of which filedescriptors are used and only the exit codes of processes

TorKlingberg 5y ago

Great read, but worth noting from the end of the article that "late feature" here means it was added in the early '90s. The late addition to Unix that surprises me is ssh. It was only invented in the late '90s. Before that everyone used unencrypted remote shells like telnet. Encryption in general was in a pretty bad state in the '90s: original http was unencrypted and early mobile phone standards that are still in use have very weak encryption.

teddyh 5y ago
Telnet has encryption options, it just wasn't widely implemented.
zokier 5y ago
Everything was unencrypted until late 90s (and in many cases until late 00s). Email (both smtp and pop3/imap), irc, web, gopher, telnet, ftp, local disks, removable storage, network storage (smb/nfs etc), everything. Computing and the internet was much nicer place, there wasn't such an adversial attitude where everything would be broken just because its out there like today.
- mturmon 5y ago
  Yeah, on the time sharing Unix systems I would use in the 80s and 90s, everyone’s home directory (and most everything under it) was world readable by default. You could change the permissions, but most people didn’t. I feel like those old folks who tell of a time when people didn’t bother to lock their doors at night.
  - throwawayboise 5y ago
    I think on Ubuntu/Debian this is still the default; the UMASK in /etc/login.defs is 022.
    - mongol 5y ago
      I think OpenSuse too. I recently converted mine to user private group handling https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/4/html/reference_guide/s1-users-groups-private-groups https://access.redhat.com/documentation/en-us/red_hat_enterp...

vimax 5y ago

A fun trick is using the `paste` command with process substitution to combine lines of output from different commands. ./cmd1 outputs: a b c ./cmd2 outputs: 1 2 3 paste <(./cmd1) <(./cmd2) outputs: a 1 b 2 c 3 You can then use xargs to use each line as arguments for another command: paste <(./cmd1) <(./cmd2) | xargs -L1 ./cmd3 calls: ./cmd3 a 1 ./cmd3 b 2 ./cmd3 c 3

sillysaurusx 5y ago
Should’ve been named zip. Is there an unzip? But of course it can’t be named zip because that’s the compressor, so I’m really asking whether there’s an unpaste. I guess the closest is awk {print $1;}, but it makes me wince every time I write it. (I already know that would give syntax errors as written here.)
- vimax 5y ago
  Depending on the data, the right options for xargs would work, and there's the pr command I haven't used much.
  - sillysaurusx 5y ago
    Hmm. Does xargs give you a way to skip the first arg? xargs -n 2 I {} almost works, but you’d need a way to extract a specific arg out of the {}. Which I just realized I don’t know how to do. “(at 0 {})” would be a nice construction. But then of course now you’re writing xargs -n 2 -I {} echo (at 1 {}) instead of just awk {print $2}, so it’s mission failed successfully.
    - quicklime 5y ago
      If you're just looking for a single column, then as zamfi mentioned, cut does the job: $ paste <(printf "a\nb\nc\n") <(printf "1\n2\n3\n") | cut -f2 1 2 3 It makes sense that "cut" and "paste" would be inverse operations, but too bad that GUIs repurposed these words for clipboard operations. But awk can do both, and does so in a single pass: $ paste <(printf "a\nb\nc\n") <(printf "1\n2\n3\n") | awk '{ print $1 > "letters"; print $2 > "numbers" }' $ cat letters a b c $ cat numbers 1 2 3

sillysaurusx 5y ago

I love zsh’s take on process substitution: cat =(echo foo) It’s almost identical to <(echo foo), but crucially it drains the output before performing the substitution. It’s like <(... | sponge), but somehow much more reliable. I’ve used it in a few situations where process substation failed, though being old and feeble I can’t remember why it failed. But for example you can copy the file safely, knowing that it will contain the full output of the shell pipeline, unlike process substitution. I don’t even know the name of =(...). Hmm. Sponge substitution?

sillysaurusx 5y ago
Thought of an example. vs cp =( ( echo foo; sleep 1; echo bar) | sponge) ) foobar The first will complete instantly and result in a file named foobar but whose contents is either zero length or foo, depending on how unlucky you are. The second takes one second to complete and results in a file named foobar containing foobar. It’s strange that vanilla process substitution can’t seem to solve this at all.
- LukeShu 5y ago
  I'm not sure what you're trying to say here. With GNU cp on Linux, the commands cp <( ( echo foo; sleep 1; echo bar) | sponge ) foobar cp =( ( echo foo; sleep 1; echo bar) | sponge ) foobar both have the same result: the file "foobar" containing the string "foo\nbar\n".
  - sillysaurusx 5y ago
    I didn't test it. You're right, I'm simply mistaken. Here's a better (but awkward) illustration of the problem: :~$ python3 -c 'import sys; import shutil; shutil.copy(sys.argv[1], "foobar")' < (echo foo ; sleep 1 ; echo bar) Traceback (most recent call last): File "<string>", line 1, in <module> File "/opt/homebrew/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/shutil.py", line 427, in copy copyfile(src, dst, follow_symlinks=follow_symlinks) File "/opt/homebrew/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/shutil.py", line 257, in copyfile raise SpecialFileError("`%s` is a named pipe" % fn) shutil.SpecialFileError: `/dev/fd/11` is a named pipe :~$ python3 -c 'import sys; import shutil; shutil.copy(sys.argv[1], "foobar")' =(echo foo ; sleep 1 ; echo bar) Notice the difference: :~$ echo =(echo foo ; sleep 1; echo bar) /tmp/zshbusU5D :~$ echo <(echo foo ; sleep 1; echo bar) /dev/fd/11 /dev/fd/11 is a named pipe, whereas /tmp/zshbEF1D9 is a plain file. There are situations where named pipes fail, where you actually do need a plain file, and I can't think of a single one offhand. :) "I've run into it in the past" is all I can say. And the above snippet at least gives one specific example of a hypothetical case.

renewiltord 5y ago

In practice I end up caching the output often. I have used process substitution but the iteration process feels more useful to me if I've slowly built up data and I can inspect the internal pieces each time and reuse them in different ways. But I can see if it's relatively fast. I like it. I just don't end up using it often.

unixhero 5y ago

Which is the best she'll, fish or zsh?//

yjftsjthsd-h 5y ago
There is no possible way to usefully answer that question as given; every shell has its own advantages and disadvantages. Plain bourne is universal but bare-bones, bash is mostly ubiquitous on Linux, zsh is powerful but just different enough to occasionally bite you, fish is very user friendly but doesn't even try to be compatible with anything else, ksh is a nice option and is built-in on BSDs, dash sucks for interactive work but is great for running scripts...
nmz 5y ago
tcl
- errcorrectcode 5y ago
  Nawh, it's all about tk. ;)
errcorrectcode 5y ago
When everything is both nails and screws, a hammer isn't the best tool for everything (unless it has a screwdriver on the end and the hammer part is big enough to be actually useful as a hammer).

shuffel 5y ago

For whatever reason I can never remember the syntax of <(command) and end up "rediscovering" it every year. It's seldom used but when it's needed it's rather elegant. Another somewhat related useful bash feature is using this form: wc <<< 'a b c' instead of: echo 'a b c' | wc

JNRowe 5y ago
With the caveat that the here string causes a tempfile to be written¹, so they're not quite equivalent. How much that matters for your use cases though is a different question, but it is worth thinking about if you're doing lots of repeated calls. ¹ With Bash v5 it may use a pipe if the data is small enough, but you can't guarantee people will have that because of GPLv3 phobia. I believe it is always a tempfile with zsh.

iamevn 5y ago

I quite like how fish does process substitution. foo (bar | psub) Instead of syntax it's just a plain function. docs: https://fishshell.com/docs/current/cmds/psub.html https://fishshell.com/docs/current/cmds/psub.html source of psub.fish: https://github.com/fish-shell/fish-shell/blob/master/share/functions/psub.fish https://github.com/fish-shell/fish-shell/blob/master/share/f...

limoce 5y ago
But fish's psub only supports file or named pipe, not the one mentioned in the post, which is described as the "the best way".
- zanchey 5y ago
  It doesn't even support named pipes - the function's been broken for years. Interestingly it doesn't actually seem to matter a great deal, at least now that temporary files are usually on a fast disk.
ogogmad 5y ago
While the syntax might be nice, strictly speaking, process substitution isn't currently possible in Fish: https://news.ycombinator.com/item?id=29845845 https://news.ycombinator.com/item?id=29845845

jakub_g 5y ago

An unfortunate thing is that process substitution does not work in git bash on Windows. (at least it was the case last time I tested; googling around I found a random comment in random github repo saying it's been fixed in 2.25 but I don't have a laptop handy to test it now).

cerved 5y ago
it does

chubot 5y ago

Fun fact, in bash you lose the exit codes of process subs. It doesn't even wait() on those processes in many cases. So there is no way to abort a bash script if something like <(sort nonexistent) fails. OSH lets you opt into stricter behavior: $ osh -c set -e shopt -s oil:basic diff <(sort left) <(sort nonexistent) echo "should not get here"' ' sort: cannot read: nonexistent: No such file or directory diff <(sort left) <(sort nonexistent) ^~ [ -c flag ]:1: fatal: Exiting with status 2 (command in PID 29359) In contrast, bash will just keep going and ignore failure. You can also get all the exit codes with @_process_sub_status, which is analogous to PIPESTATUS. (I should probably write a blog post about this: http://www.oilshell.org/blog/ http://www.oilshell.org/blog/)

andreyv 5y ago
> So there is no way to abort a bash script if something like <(sort nonexistent) fails. The process ID of the last executed background command in Bash is available as $!. cat <(sort nonexistent) wait $! || echo fail gives sort: cannot read: nonexistent: No such file or directory fail
- mkdirp 5y ago
  > The process ID of the last executed background command in Bash is available as $!. After a quick search it doesn't seem like this is an option for anything but the last executed background process. So if you have the typical `diff <(left) <(right)`, there is no easy way to get the pid of `<(left)`. I suppose you could output the pid to a file, and use it from there, though I haven't tested if that would work.

errcorrectcode 5y ago

0. Process substitution is a potential DoS vector as it could take up all of RAM and/or disk space. 1. Also, not all commands are compatible with it, especially if they need rewinding or reopening. diff has issues with using it for both arguments often. It's likely the use of memory mapped files, but I could be wrong. 2. Shells ought to implement a flag for process substitution to allow temporary files to reside on disk for the lifetime of the command line. This way, it can operate on extremely large files.

FedericoRazzoli 5y ago

This is a great explanation! I wonder many times why we had to play with obscure xargs instead of being able to pipe a command output to an argument.

kazinator 5y ago

The claim is false; process substitution can be cobbed together with named fifos,* and those are "ancient". Only problem that those are temporary objects that have to be created in the file system, and cleaned up. However, temporary objects (files, no fifos) are also used in here doc implementations. Process substitution is a late feature simply because the creativity juice in Unix (tm) dried up some time before the middle 1990's, leaving the FOSS reimplementations of Unix to carry the development torch. Those projects had to balance among other goals like quality/robustness and compatibility. (If we look at the quality of the FOSS tools compared to the Unix originals, we could also remark that "quality and robustness was late in coming to Unix". But we equivocate on Unix, because GNU stands for GNU is Not Unix!) Features appearing in FOSS utilities like GNU Bash take time to make into Unix (tm). Process substitution is not yet in the standard, therefore it is not in in fact in Unix (tm). Shell scripting is a conservative activity. The language isn't very good and so improving it is like kicking a dead horse in some ways; the most important matter in any new shell release is that old scripts keep working. (Like configuration scripts for the build systems of nicer languages). --- * See GNU Bash manual: https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Process-Substitution https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash....: " Process substitution is supported on systems that support named pipes (FIFOs) or the /dev/fd method of naming open files. "

xeromal 5y ago
Yeah, but Saget could still have had a damaged heart from it regardless of Clapton's experience.