4 ms·

I’ve been doing Linux stuff for ten years and I am apparently just learning that the pipeline commands run in parallel not serially. If I had put any thought in

I’ve been doing Linux stuff for ten years and I am apparently just learning that the pipeline commands run in parallel not serially. If I had put any thought into it I would have realized it because otherwise tailing logs to grep wouldn’t work...

muststopmyths 8y ago
Pipes are commonly used when the right side is consuming the output of the left side. That introduces serialization at least insofar as the right side is forced to wait for input. You could replace the | with an &; and get the same behavior because nothing in the 2nd stage depends on the first stage of the pipe. In my old man grumbling at clouds voice: This is not arcane.
- yakshaving_jgt 8y ago
  Perhaps not arcane, but at least somewhat counterintuitive if one's mind is too invested in the "pipe" analogy.
  - sjy 8y ago
    But physical pipes run in parallel too! If you have a pipe full of water, and you push some more water in one end, there is an instant effect at the other end. If it's empty, then sequentiality is introduced because it takes time for the water to flow through the pipe. That's how a Unix pipe works – (the processes at) both ends of the pipe come into existence at the same time, but if the right side is waiting for input (the pipe buffer is empty), it won't do anything until the left side has produced output.
    - Y_Y 8y ago
      That's parallel in time, but serial in "space". That might be the source of the confusion here.
    - oblio 8y ago
      > But physical pipes run in parallel too! For normal people, pipes are as serial as can be. For the average person, parallel pipes = 2 pipes.
rconti 8y ago
Twenty five here, and I still get confused about stdout/stderr redirection.
- mjevans 8y ago
  As I was just commenting in a different reply... The shell ties the prior program's (or user input to the first) output to the input of the next program. That's (old) stdout and (new) stdin taken care of. stderr is sent to the terminal as output by default. For reference stdin, stdout, and stderr are normally numbered 0 through 2 respectively. When you're directing input and output on the shell (usually) the default is to wire up things as they'd make sense, but (without any IFS characters) a number on either side of the redirection operator codes tells the shell to grab a different input or output. These will yield different results due to left to right parsing: echo test >/tmp/test 2>&1 echo test 2>&1 >/tmp/test In the second if there were any errors they'd be sent to the file, while in the first (assuming you can make temporary files, which normally you can) there won't be any errors printed but they would be sent as standard output on the shell. Here's a more interesting example echo test >/dev/null/test 2>/dev/null echo test 2>/dev/null >/dev/null/test The second line will silently fail (though it'll still return an exit value of 1) because the standard error output was already sent to /dev/null, while the first attempts to open a file that can't exist and prints the failure message before the error output is redirected.
- kangnkodos 8y ago
  Here's a cheat sheet which should cover the most common situations. Send stdout to a file. ls myFileWhichExists > myStdLog - or - ls myFileWhichExists 1> myStdLog Send stderr to a file. ls myFileWhichDoesNotExist 2> myErrLog Send stdout to one file and stderr to a different file. ls myFileWhichExists myFileWhichDoesNotExist 1> myStdLog 2> myErrLog Send stdout and stderr to the same file ls myFileWhichExists myFileWhichDoesNotExist 1> myBothLog 2>&1 I read that last part "2>&1" as "Send stderr (2) to the same place as stdout (1) is already going to". Notice that if you send stdout and stderr to the same file, because of caching and other issues, the output from stdout and stderr will overlap in unpredictable ways.
  - 8y ago

muststopmyths 8y ago

Pipes are commonly used when the right side is consuming the output of the left side. That introduces serialization at least insofar as the right side is forced to wait for input. You could replace the | with an &; and get the same behavior because nothing in the 2nd stage depends on the first stage of the pipe. In my old man grumbling at clouds voice: This is not arcane.

yakshaving_jgt 8y ago
Perhaps not arcane, but at least somewhat counterintuitive if one's mind is too invested in the "pipe" analogy.
- sjy 8y ago
  But physical pipes run in parallel too! If you have a pipe full of water, and you push some more water in one end, there is an instant effect at the other end. If it's empty, then sequentiality is introduced because it takes time for the water to flow through the pipe. That's how a Unix pipe works – (the processes at) both ends of the pipe come into existence at the same time, but if the right side is waiting for input (the pipe buffer is empty), it won't do anything until the left side has produced output.
  - Y_Y 8y ago
    That's parallel in time, but serial in "space". That might be the source of the confusion here.
  - oblio 8y ago
    > But physical pipes run in parallel too! For normal people, pipes are as serial as can be. For the average person, parallel pipes = 2 pipes.

rconti 8y ago

Twenty five here, and I still get confused about stdout/stderr redirection.

mjevans 8y ago
As I was just commenting in a different reply... The shell ties the prior program's (or user input to the first) output to the input of the next program. That's (old) stdout and (new) stdin taken care of. stderr is sent to the terminal as output by default. For reference stdin, stdout, and stderr are normally numbered 0 through 2 respectively. When you're directing input and output on the shell (usually) the default is to wire up things as they'd make sense, but (without any IFS characters) a number on either side of the redirection operator codes tells the shell to grab a different input or output. These will yield different results due to left to right parsing: echo test >/tmp/test 2>&1 echo test 2>&1 >/tmp/test In the second if there were any errors they'd be sent to the file, while in the first (assuming you can make temporary files, which normally you can) there won't be any errors printed but they would be sent as standard output on the shell. Here's a more interesting example echo test >/dev/null/test 2>/dev/null echo test 2>/dev/null >/dev/null/test The second line will silently fail (though it'll still return an exit value of 1) because the standard error output was already sent to /dev/null, while the first attempts to open a file that can't exist and prints the failure message before the error output is redirected.
kangnkodos 8y ago
Here's a cheat sheet which should cover the most common situations. Send stdout to a file. ls myFileWhichExists > myStdLog - or - ls myFileWhichExists 1> myStdLog Send stderr to a file. ls myFileWhichDoesNotExist 2> myErrLog Send stdout to one file and stderr to a different file. ls myFileWhichExists myFileWhichDoesNotExist 1> myStdLog 2> myErrLog Send stdout and stderr to the same file ls myFileWhichExists myFileWhichDoesNotExist 1> myBothLog 2>&1 I read that last part "2>&1" as "Send stderr (2) to the same place as stdout (1) is already going to". Notice that if you send stdout and stderr to the same file, because of caching and other issues, the output from stdout and stderr will overlap in unpredictable ways.
- 8y ago