7 ms·

How do Unix pipes work?

269 points by v3gas 7y ago

nneonneo 7y ago
This article promotes bad practices for dealing with SIGPIPE. 1. Closing stderr in Python is not a good idea because that’ll swallow any other errors that occur at exit. Redirecting stdout to devnull is really just a way to prevent the flushed output from going to the now-closed stdout and triggering another SIGPIPE. That’s more preferable than closing stderr and losing error output at exit. 2. Ignoring SIGPIPE is a terrible idea for a process that should do stream processing. Try making a yes clone and ignoring SIGPIPE - your process will likely run forever trying to shove “y” into a closed pipe. There’s a reason SIGPIPE was invented! Very few programs bother to check the return value from write/printf/etc.
- v3gas 7y ago
  Thanks! So the preferred way is to redirect to dev null?
  - hoytech 7y ago
    It's better to just do nothing and allow SIGPIPE to kill your program. That is the reason this signal exists. Python is not a good unix citizen in this case. Compare it to Perl for example where nothing special is required to do the right thing: $ perl -E 'say "y" while 1' | head -1 y $
    - v3gas 7y ago
      I see
    - hk__2 7y ago
      > Python is not a good unix citizen in this case. Compare it to Perl for example where nothing special is required to do the right thing How is this better? At least with the exception you can cleanly terminate your program if you need to.
      - hoytech 7y ago
        In the unix pipeline model, components read input, perform some transformation, and print output. There is no need to do any clean-up. As nneonneo explained, because they rely on SIGPIPE, most unix programs don't even need to check for errors from write(2). Naturally, in the rare case you want to do some clean-up you can: $ perl -E '$SIG{PIPE} = sub { die "cleanup" }; say "y" while 1' | head -1 y cleanup at -e line 1.
    - seneca 7y ago
      I disagree. Saying python is "not a good Unix citizen" here is akin to saying cars are a bad highway citizen because people can crash them. Python provides mechanisms to handle signals. The point of a signal is to indicate something outside of your process has happened and you may want to respond to it. It's up to the program to respond to relevant signals and Python in no way stops a developer from doing that.
      - hoytech 7y ago
        > Saying python is "not a good Unix citizen" here is akin to saying cars are a bad highway citizen because people can crash them. I don't follow your analogy. The default signal disposition for SIGPIPE is to silently terminate the process. Normally processes compose nicely on the command-line and this requires no extra work from the programmer. By disregarding this convention, most Python programs pollute your terminal when used in pipelines, which is why I claim that in this case Python is a bad unix citizen.
        seneca 7y ago
        Yep, I follow your logic. As you say, in other languages it "requires no extra work", it's more a matter of convenience. Python in no way stops you from a having a typical response to SIGPIPE, it's as simple as handling the exception and doing a sys.exit(), but it doesn't "just happen". This makes the typical path a little more contrived, but I don't think having explicit handling makes it a bad citizen. To torture my analogy, in my mind a "bad citizen" of the highway may be a car incapable of doing the minimum speed limit, whereas Python just has a manual transmission vs Perl's automatic (in this case). I'm splitting hairs though, your point is fair. I just think your objection is a little strong for what comes down to convenience.
        derefr 7y ago
        It's not a matter of "convenience" any more, when you know for a fact that nobody dashing off quick five-line scripts in your language is going to bother doing the extra thing, and so the default behavior will be the only behavior for those scripts. Defaults matter. It's like the difference between opt-in and opt-out for organ donation, or the difference between encouraging and requiring cars to have airbags.
    - nonofr34 7y ago
      Not an expert is no way, so I could be wrong, but in python could you use signal.SIG_DFL to have that expected behavior: # cat.py import sys from signal import signal, SIGPIPE, SIG_DFL signal(SIGPIPE,SIG_DFL) for line in sys.stdin: print(line)
      - hoytech 7y ago
        True, but most Python programs don't do this, so they pollute your terminal with error messages when you attempt to use them as interior processes in a unix pipeline.
      - gnomewascool 7y ago
        That seems to be discouraged by the official python docs.[0] [0] https://docs.python.org/3/library/signal.html#note-on-sigpipe https://docs.python.org/3/library/signal.html#note-on-sigpip...
        jolmg 7y ago
        > sys.exit(1) # Python exits with error code 1 on EPIPE Should be 141 instead of 1. Convention is that when a program dies because of a signal, the exit code should be 128 + the signal number, in this case 13. So, 128 + 13 = 141. Here are some examples: $ ( yes; >&2 echo $? ) | head -0 141 $ ( ping localhost; >&2 echo $? ) | head -0 141 $ ( perl -E 'say "y" while 1'; >&2 echo $? ) | head -0 141 In python, when using signal(SIGPIPE, SIG_DFL), you get the correct behaviour (at least regarding the exit status): $ ( python -c ' import sys from signal import signal, SIGPIPE, SIG_DFL signal(SIGPIPE,SIG_DFL) while True: print("y"); ' ; >&2 echo $? ) | head -0 141 > Do not set SIGPIPE’s disposition to SIG_DFL in order to avoid BrokenPipeError. Doing that would cause your program to exit unexpectedly also whenever any socket connection is interrupted while your program is still writing to it. Regarding that socket behavior, if that's the standard in other programming environments, why is it an issue from python's perspective? Doesn't seem worth eschewing standard unix conventions. I mean, it says "unexpectedly", but isn't it actually the expected behavior? Unexpected is for python to say that what's default (SIG_DFL) is unexpected.
    - dooglius 7y ago
      This is bad advice, if you are doing anything other than output to stdout/stderr (which you almost assuredly are, unless you are doing something very simple like "yes"), you want to switch to /dev/null. For instance, running rm -vfr folder/ | head involves SIGPIPE causing problems because it may or may not kill rm before it finishes deleting the directory, based on its internal output buffer size.
      - hoytech 7y ago
        The problem here is connecting a program you want to run to completion to the head command, not how rm handles SIGPIPE.
        dooglius 7y ago
        head is just an example, if you pipe rm -vfr (or any program that does more than just output) into _anything_ the reasonable and default behavior should be to run rm to completion. (And from a programmer's POV, the reasonably thing is for syscalls to handle errors by returning an error, the way every other error is handled.)
        kees99 7y ago
        Wouldn't it be better to be explicit? If intention is to "display first 10 lines, at most; discard the rest", you can write: rm -vrf directory/ | (head; cat >/dev/null)
        dooglius 7y ago
        I guess it depends on your semantics/mental model of what a pipeline should do. My mental model says that each part should run fully, and the pipeline should emulate sequential behavior, with "yes" being a bit of a hack in this regard. Cases where the behavior of a pipeline depends on the internal buffer size and operation ordering (e.g. what if rm decided to print everything it did only after removing everything?) seem like bugs to me. Maybe your mental model is different? In any case, I think that in the vast majority of pipelines, the default SIGPIPE behavior is not what is desired.
- seneca 7y ago
  Agreed. I like articles like this because they show the learning process, which I think is super valuable. However, they really need a big disclaimer stating the author is experimenting and doesn't know the correct answer. Otherwise people stumble upon it and take it as authoritative.
  - v3gas 7y ago
    Great point, I should probably add that disclaimer!
- developer2 7y ago
  The biggest takeaway here IMO is that Python breaks the standard contract regarding signals–at least for SIGPIPE. Python should not be catching and throwing an exception for SIGPIPE; it should simply exit immediately, which is literally the default POSIX behaviour... unless a script/program/process specifically installs a signal handler to perform cleanup before exit. Python has some pretty awful behaviours built into it, and this is one of them. Half of this article is not "How do Unix pipes work", but "how to fix broken SIGPIPE handling in Python".
  - loeg 7y ago
    Python doesn't catch SIGPIPE, it ignores it. The exception is raised from a -1/EPIPE return from libc write(). I fully agree that Python is often a bad citizen in terms of signal handling — it wants to only process signals on 'the main thread', but also wants end-users to fully control signal-handling. The two ideas are sort of at-odds and in general I find handling signals in Python frustrating.
- loeg 7y ago
  > 1. Closing stderr in Python is not a good idea because that’ll swallow any other errors that occur at exit. Redirecting stdout to devnull is really just a way to prevent the flushed output from going to the now-closed stdout and triggering another SIGPIPE. That’s more preferable than closing stderr and losing error output at exit. I don't follow. The standard for EPIPE/SIGPIPE handling is to silently exit with an error status. It's fine to close stderr to prevent spurious warning messages about flushing stdout. > 2. Ignoring SIGPIPE is a terrible idea for a process that should do stream processing. Try making a yes clone and ignoring SIGPIPE - your process will likely run forever trying to shove “y” into a closed pipe. There’s a reason SIGPIPE was invented! Very few programs bother to check the return value from write/printf/etc. Programs can correctly handle lost pipes masking SIGPIPE entirely, with error checking alone. Python's BrokenPipeError is raised on the basis of EPIPE, not SIGPIPE. Re: programs not checking error returns of write() and close(): that is not really true in a language like Python with exceptions raised on IO errors. It always does the check, and the unwinder aborts the program if nothing handles the error. Sigpipe is completely unnecessary for Python programs. (It's also not necessary for C programs, but I guess AT&T didn't want to fix their programs to check for errors.)
  - nneonneo 7y ago
    The suppression of SIGPIPE was done in the Go code, not in the Python code. Does the POSIX standard mandate that programs receiving EPIPE/SIGPIPE die silently? I don’t know of such a rule, and there’s plenty of programs that violate this. Python is a bit too verbose with the errors (with a full trace back and two copies of the error) so suppressing those errors somehow seems like a good idea for a general-purpose command line tool.
    - userbinator 7y ago
      Does the POSIX standard mandate that programs receiving EPIPE/SIGPIPE die silently? https://pubs.opengroup.org/onlinepubs/7908799/xsh/signal.h.html https://pubs.opengroup.org/onlinepubs/7908799/xsh/signal.h.h... Yes. The default action of SIGPIPE is to terminate the process.
- pixelbeat__ 7y ago
  Agreed. Correct handling of SIGPIPE is quite subtle and often done incorrectly. I've some notes on SIGPIPE considerations at: http://www.pixelbeat.org/programming/sigpipe_handling.html http://www.pixelbeat.org/programming/sigpipe_handling.html
- asveikau 7y ago
  The biggest reason I can think of not to close stderr (or any other of the standard handles) is that the next open(2) call is likely to get that same descriptor recycled. So now joe random open file is going to receive all the error messages from random libraries, or perhaps even from unrelated programs you may have forked with that file as fd 2.
- microtherion 7y ago
  Another bad practice, IMHO, is not to quit the program once the exception is thrown. Instead, the loop continues and the rest of the input gets fed to /dev/null
ur-whale 7y ago
This article only shows basic usage of pipes (this is what they mean by "how pipes works"), but doesn't explain at all "how pipe works" (as in: how are they implemented).
- chaps 7y ago
  Same, I was hoping for some mention of /proc/PID/fd*, but nothin'.
  - emmelaich 7y ago
    /proc/ is not a fundamental to understanding Unix or Unix pipes and is not present on many Unixes.
    - chaps 7y ago
      You're right, but for what it's worth, the folks that I've taught the nuance of pipes seem to really only "get it" after pointing them to /proc/PID/fd* and having them cat in/out the files there. It directly leads into much deeper understandings of what "everything is a file" means.
- userbinator 7y ago
  It's implemented as a buffer and some associated state. A process that writes to the buffer can do so until it is full, at which point the thread is suspended (blocked on the write() call) until it is not full. The read() side is similar --- reads return successive data in the buffer unless it is empty, at which point the read() call will block.
  - emmelaich 7y ago
    ur-whale probably knows that
RoutinePlayer 7y ago
My favorite sentence from Brian Kernighan's latest book "UNIX A History and a Memoir": Pipes are the quintessential Unix invention, an elegant and efficient way to use temporary connections of programs .. so I'll read this article :-)
pierremenard 7y ago
See Section 1.2 this & 1.3 of the MIT Unix teaching OS for a great intro to FDs and pipes: https://pdos.csail.mit.edu/6.828/2019/xv6/book-riscv-rev0.pdf https://pdos.csail.mit.edu/6.828/2019/xv6/book-riscv-rev0.pd...
cperciva 7y ago
If we cat this file, it will be printed to the terminal. > cat brothers_karamazov.txt ... many lines of text! ***FINIS*** It takes a noticeable amount of time to finish. The amount of time it takes for cat(1) to read and output the file is almost certainly insignificant. The time the author is noticing is probably related to how long it takes for his console to process the text.
- kccqzy 7y ago
  Agreed. This can be easily verified by putting `time` in front of the cat to measure the time taken. Even for huge text files, the wall clock time might be significant but the "user" time is likely still zero.
  - wolf550e 7y ago
    or redirect cat to /dev/null and see how fast that is
- happytoexplain 7y ago
  This is the first thing I noticed too. >how does cat know to stop when head is finished I'm no expert on Unix, so correct me if I'm wrong, but surely this line of reasoning is misleading because pipes create a unidirectional data flow, so `cat` can not know anything about `head`. It does not "stop" - it passes the whole text along just as it did without the pipe. As you said, the delay comes in printing to the console, not in the `cat` command.
  - cperciva 7y ago
    Pipes aren't completely unidirectional. You get one bit of information flowing back: Whether the read end of the pipe is still open.
  - kyuudou 7y ago
    This is a great example of Useless Use of cat and why it is bad - the full text is indeed sent through the pipe simply for head to chop n initial lines. I've actually had "developers" go "but, readability". Yea ok.
no_gravity 7y ago
I had a nice surprise and learning experience, when I discovered that the output of (echo red; echo green 1>&2) | echo blue is indeterministic: http://www.gibney.de/the_output_of_linux_pipes_can_be_indeter http://www.gibney.de/the_output_of_linux_pipes_can_be_indete... As it turns out, this short line and its behavior nicely demonstrate a bunch of aspects that happen under the hood when you use a pipe.
kccqzy 7y ago
My own rule of thumb of whether or not to ignore SIGPIPE is simple: * If you only deal with file descriptors provided to you (stdin, stdout, stderr) as well as some files that you open (including special files like FIFOs), do not ignore SIGPIPE. * If you deal with sophisticated file descriptors (socket(2) and pipe(2) count as sophisticated), you'd better ignore SIGPIPE, but also make sure to check for EPIPE in every single write. In my view, SIGPIPE is a kludge so that programs that are too lazy to check for errors from write(2) (and fwrite(3) and related friends) will not waste resources. But if you are dealing with sophisticated file descriptors, there is a lot more happening than just open/read/write and a lot more error cases you must handle, and at that point the incremental cost of handling EPIPE isn't a significant addition.
ryanmccullagh 7y ago
Here's something that you should remember about using pipes and fork(2) in Python 3: By default, O_CLOEXEC is passed to the pipe(2) system from the CPython runtime. This means, that reading the read end of the side in the parent process after you forked will not work. Thefore you should explicitly change fctl flags and remove os.O_CLOEXEC: fcntl.fcntl(readfd, fcntl.F_SETFL, fcntl.fcntl(readfd, fcntl.F_GETFL) & ~os.O_CLOEXEC)
ilammy 7y ago
Another point where you have to ignore SIGPIPE is concurrent code that handles multiple fds (say, like a web server). In this case you have to ignore the signal and process EPIPE correctly, because the signal is not associated with a particular fd so you cannot tell which one of them failed.

269 points by v3gas 7y ago

This article promotes bad practices for dealing with SIGPIPE. 1. Closing stderr in Python is not a good idea because that’ll swallow any other errors that occur at exit. Redirecting stdout to devnull is really just a way to prevent the flushed output from going to the now-closed stdout and triggering another SIGPIPE. That’s more preferable than closing stderr and losing error output at exit. 2. Ignoring SIGPIPE is a terrible idea for a process that should do stream processing. Try making a yes clone and ignoring SIGPIPE - your process will likely run forever trying to shove “y” into a closed pipe. There’s a reason SIGPIPE was invented! Very few programs bother to check the return value from write/printf/etc.

v3gas 7y ago
Thanks! So the preferred way is to redirect to dev null?
- hoytech 7y ago
  It's better to just do nothing and allow SIGPIPE to kill your program. That is the reason this signal exists. Python is not a good unix citizen in this case. Compare it to Perl for example where nothing special is required to do the right thing: $ perl -E 'say "y" while 1' | head -1 y $
  - v3gas 7y ago
    I see
  - hk__2 7y ago
    > Python is not a good unix citizen in this case. Compare it to Perl for example where nothing special is required to do the right thing How is this better? At least with the exception you can cleanly terminate your program if you need to.
    - hoytech 7y ago
      In the unix pipeline model, components read input, perform some transformation, and print output. There is no need to do any clean-up. As nneonneo explained, because they rely on SIGPIPE, most unix programs don't even need to check for errors from write(2). Naturally, in the rare case you want to do some clean-up you can: $ perl -E '$SIG{PIPE} = sub { die "cleanup" }; say "y" while 1' | head -1 y cleanup at -e line 1.
  - seneca 7y ago
    I disagree. Saying python is "not a good Unix citizen" here is akin to saying cars are a bad highway citizen because people can crash them. Python provides mechanisms to handle signals. The point of a signal is to indicate something outside of your process has happened and you may want to respond to it. It's up to the program to respond to relevant signals and Python in no way stops a developer from doing that.
    - hoytech 7y ago
      > Saying python is "not a good Unix citizen" here is akin to saying cars are a bad highway citizen because people can crash them. I don't follow your analogy. The default signal disposition for SIGPIPE is to silently terminate the process. Normally processes compose nicely on the command-line and this requires no extra work from the programmer. By disregarding this convention, most Python programs pollute your terminal when used in pipelines, which is why I claim that in this case Python is a bad unix citizen.
      - seneca 7y ago
        Yep, I follow your logic. As you say, in other languages it "requires no extra work", it's more a matter of convenience. Python in no way stops you from a having a typical response to SIGPIPE, it's as simple as handling the exception and doing a sys.exit(), but it doesn't "just happen". This makes the typical path a little more contrived, but I don't think having explicit handling makes it a bad citizen. To torture my analogy, in my mind a "bad citizen" of the highway may be a car incapable of doing the minimum speed limit, whereas Python just has a manual transmission vs Perl's automatic (in this case). I'm splitting hairs though, your point is fair. I just think your objection is a little strong for what comes down to convenience.
        derefr 7y ago
        It's not a matter of "convenience" any more, when you know for a fact that nobody dashing off quick five-line scripts in your language is going to bother doing the extra thing, and so the default behavior will be the only behavior for those scripts. Defaults matter. It's like the difference between opt-in and opt-out for organ donation, or the difference between encouraging and requiring cars to have airbags.
  - nonofr34 7y ago
    Not an expert is no way, so I could be wrong, but in python could you use signal.SIG_DFL to have that expected behavior: # cat.py import sys from signal import signal, SIGPIPE, SIG_DFL signal(SIGPIPE,SIG_DFL) for line in sys.stdin: print(line)
    - hoytech 7y ago
      True, but most Python programs don't do this, so they pollute your terminal with error messages when you attempt to use them as interior processes in a unix pipeline.
    - gnomewascool 7y ago
      That seems to be discouraged by the official python docs.[0] [0] https://docs.python.org/3/library/signal.html#note-on-sigpipe https://docs.python.org/3/library/signal.html#note-on-sigpip...
      - jolmg 7y ago
        > sys.exit(1) # Python exits with error code 1 on EPIPE Should be 141 instead of 1. Convention is that when a program dies because of a signal, the exit code should be 128 + the signal number, in this case 13. So, 128 + 13 = 141. Here are some examples: $ ( yes; >&2 echo $? ) | head -0 141 $ ( ping localhost; >&2 echo $? ) | head -0 141 $ ( perl -E 'say "y" while 1'; >&2 echo $? ) | head -0 141 In python, when using signal(SIGPIPE, SIG_DFL), you get the correct behaviour (at least regarding the exit status): $ ( python -c ' import sys from signal import signal, SIGPIPE, SIG_DFL signal(SIGPIPE,SIG_DFL) while True: print("y"); ' ; >&2 echo $? ) | head -0 141 > Do not set SIGPIPE’s disposition to SIG_DFL in order to avoid BrokenPipeError. Doing that would cause your program to exit unexpectedly also whenever any socket connection is interrupted while your program is still writing to it. Regarding that socket behavior, if that's the standard in other programming environments, why is it an issue from python's perspective? Doesn't seem worth eschewing standard unix conventions. I mean, it says "unexpectedly", but isn't it actually the expected behavior? Unexpected is for python to say that what's default (SIG_DFL) is unexpected.
  - dooglius 7y ago
    This is bad advice, if you are doing anything other than output to stdout/stderr (which you almost assuredly are, unless you are doing something very simple like "yes"), you want to switch to /dev/null. For instance, running rm -vfr folder/ | head involves SIGPIPE causing problems because it may or may not kill rm before it finishes deleting the directory, based on its internal output buffer size.
    - hoytech 7y ago
      The problem here is connecting a program you want to run to completion to the head command, not how rm handles SIGPIPE.
      - dooglius 7y ago
        head is just an example, if you pipe rm -vfr (or any program that does more than just output) into _anything_ the reasonable and default behavior should be to run rm to completion. (And from a programmer's POV, the reasonably thing is for syscalls to handle errors by returning an error, the way every other error is handled.)
        kees99 7y ago
        Wouldn't it be better to be explicit? If intention is to "display first 10 lines, at most; discard the rest", you can write: rm -vrf directory/ | (head; cat >/dev/null)
        dooglius 7y ago
        I guess it depends on your semantics/mental model of what a pipeline should do. My mental model says that each part should run fully, and the pipeline should emulate sequential behavior, with "yes" being a bit of a hack in this regard. Cases where the behavior of a pipeline depends on the internal buffer size and operation ordering (e.g. what if rm decided to print everything it did only after removing everything?) seem like bugs to me. Maybe your mental model is different? In any case, I think that in the vast majority of pipelines, the default SIGPIPE behavior is not what is desired.
seneca 7y ago
Agreed. I like articles like this because they show the learning process, which I think is super valuable. However, they really need a big disclaimer stating the author is experimenting and doesn't know the correct answer. Otherwise people stumble upon it and take it as authoritative.
- v3gas 7y ago
  Great point, I should probably add that disclaimer!
developer2 7y ago
The biggest takeaway here IMO is that Python breaks the standard contract regarding signals–at least for SIGPIPE. Python should not be catching and throwing an exception for SIGPIPE; it should simply exit immediately, which is literally the default POSIX behaviour... unless a script/program/process specifically installs a signal handler to perform cleanup before exit. Python has some pretty awful behaviours built into it, and this is one of them. Half of this article is not "How do Unix pipes work", but "how to fix broken SIGPIPE handling in Python".
- loeg 7y ago
  Python doesn't catch SIGPIPE, it ignores it. The exception is raised from a -1/EPIPE return from libc write(). I fully agree that Python is often a bad citizen in terms of signal handling — it wants to only process signals on 'the main thread', but also wants end-users to fully control signal-handling. The two ideas are sort of at-odds and in general I find handling signals in Python frustrating.
loeg 7y ago
> 1. Closing stderr in Python is not a good idea because that’ll swallow any other errors that occur at exit. Redirecting stdout to devnull is really just a way to prevent the flushed output from going to the now-closed stdout and triggering another SIGPIPE. That’s more preferable than closing stderr and losing error output at exit. I don't follow. The standard for EPIPE/SIGPIPE handling is to silently exit with an error status. It's fine to close stderr to prevent spurious warning messages about flushing stdout. > 2. Ignoring SIGPIPE is a terrible idea for a process that should do stream processing. Try making a yes clone and ignoring SIGPIPE - your process will likely run forever trying to shove “y” into a closed pipe. There’s a reason SIGPIPE was invented! Very few programs bother to check the return value from write/printf/etc. Programs can correctly handle lost pipes masking SIGPIPE entirely, with error checking alone. Python's BrokenPipeError is raised on the basis of EPIPE, not SIGPIPE. Re: programs not checking error returns of write() and close(): that is not really true in a language like Python with exceptions raised on IO errors. It always does the check, and the unwinder aborts the program if nothing handles the error. Sigpipe is completely unnecessary for Python programs. (It's also not necessary for C programs, but I guess AT&T didn't want to fix their programs to check for errors.)
- nneonneo 7y ago
  The suppression of SIGPIPE was done in the Go code, not in the Python code. Does the POSIX standard mandate that programs receiving EPIPE/SIGPIPE die silently? I don’t know of such a rule, and there’s plenty of programs that violate this. Python is a bit too verbose with the errors (with a full trace back and two copies of the error) so suppressing those errors somehow seems like a good idea for a general-purpose command line tool.
  - userbinator 7y ago
    Does the POSIX standard mandate that programs receiving EPIPE/SIGPIPE die silently? https://pubs.opengroup.org/onlinepubs/7908799/xsh/signal.h.html https://pubs.opengroup.org/onlinepubs/7908799/xsh/signal.h.h... Yes. The default action of SIGPIPE is to terminate the process.
pixelbeat__ 7y ago
Agreed. Correct handling of SIGPIPE is quite subtle and often done incorrectly. I've some notes on SIGPIPE considerations at: http://www.pixelbeat.org/programming/sigpipe_handling.html http://www.pixelbeat.org/programming/sigpipe_handling.html
asveikau 7y ago
The biggest reason I can think of not to close stderr (or any other of the standard handles) is that the next open(2) call is likely to get that same descriptor recycled. So now joe random open file is going to receive all the error messages from random libraries, or perhaps even from unrelated programs you may have forked with that file as fd 2.
microtherion 7y ago
Another bad practice, IMHO, is not to quit the program once the exception is thrown. Instead, the loop continues and the rest of the input gets fed to /dev/null

ur-whale 7y ago

This article only shows basic usage of pipes (this is what they mean by "how pipes works"), but doesn't explain at all "how pipe works" (as in: how are they implemented).

chaps 7y ago
Same, I was hoping for some mention of /proc/PID/fd*, but nothin'.
- emmelaich 7y ago
  /proc/ is not a fundamental to understanding Unix or Unix pipes and is not present on many Unixes.
  - chaps 7y ago
    You're right, but for what it's worth, the folks that I've taught the nuance of pipes seem to really only "get it" after pointing them to /proc/PID/fd* and having them cat in/out the files there. It directly leads into much deeper understandings of what "everything is a file" means.
userbinator 7y ago
It's implemented as a buffer and some associated state. A process that writes to the buffer can do so until it is full, at which point the thread is suspended (blocked on the write() call) until it is not full. The read() side is similar --- reads return successive data in the buffer unless it is empty, at which point the read() call will block.
- emmelaich 7y ago
  ur-whale probably knows that

RoutinePlayer 7y ago

My favorite sentence from Brian Kernighan's latest book "UNIX A History and a Memoir": Pipes are the quintessential Unix invention, an elegant and efficient way to use temporary connections of programs .. so I'll read this article :-)

pierremenard 7y ago

See Section 1.2 this & 1.3 of the MIT Unix teaching OS for a great intro to FDs and pipes: https://pdos.csail.mit.edu/6.828/2019/xv6/book-riscv-rev0.pdf https://pdos.csail.mit.edu/6.828/2019/xv6/book-riscv-rev0.pd...

cperciva 7y ago

If we cat this file, it will be printed to the terminal. > cat brothers_karamazov.txt ... many lines of text! ***FINIS*** It takes a noticeable amount of time to finish. The amount of time it takes for cat(1) to read and output the file is almost certainly insignificant. The time the author is noticing is probably related to how long it takes for his console to process the text.

kccqzy 7y ago
Agreed. This can be easily verified by putting `time` in front of the cat to measure the time taken. Even for huge text files, the wall clock time might be significant but the "user" time is likely still zero.
- wolf550e 7y ago
  or redirect cat to /dev/null and see how fast that is
happytoexplain 7y ago
This is the first thing I noticed too. >how does cat know to stop when head is finished I'm no expert on Unix, so correct me if I'm wrong, but surely this line of reasoning is misleading because pipes create a unidirectional data flow, so `cat` can not know anything about `head`. It does not "stop" - it passes the whole text along just as it did without the pipe. As you said, the delay comes in printing to the console, not in the `cat` command.
- cperciva 7y ago
  Pipes aren't completely unidirectional. You get one bit of information flowing back: Whether the read end of the pipe is still open.
- kyuudou 7y ago
  This is a great example of Useless Use of cat and why it is bad - the full text is indeed sent through the pipe simply for head to chop n initial lines. I've actually had "developers" go "but, readability". Yea ok.

no_gravity 7y ago

I had a nice surprise and learning experience, when I discovered that the output of (echo red; echo green 1>&2) | echo blue is indeterministic: http://www.gibney.de/the_output_of_linux_pipes_can_be_indeter http://www.gibney.de/the_output_of_linux_pipes_can_be_indete... As it turns out, this short line and its behavior nicely demonstrate a bunch of aspects that happen under the hood when you use a pipe.

kccqzy 7y ago

My own rule of thumb of whether or not to ignore SIGPIPE is simple: * If you only deal with file descriptors provided to you (stdin, stdout, stderr) as well as some files that you open (including special files like FIFOs), do not ignore SIGPIPE. * If you deal with sophisticated file descriptors (socket(2) and pipe(2) count as sophisticated), you'd better ignore SIGPIPE, but also make sure to check for EPIPE in every single write. In my view, SIGPIPE is a kludge so that programs that are too lazy to check for errors from write(2) (and fwrite(3) and related friends) will not waste resources. But if you are dealing with sophisticated file descriptors, there is a lot more happening than just open/read/write and a lot more error cases you must handle, and at that point the incremental cost of handling EPIPE isn't a significant addition.

ryanmccullagh 7y ago

Here's something that you should remember about using pipes and fork(2) in Python 3: By default, O_CLOEXEC is passed to the pipe(2) system from the CPython runtime. This means, that reading the read end of the side in the parent process after you forked will not work. Thefore you should explicitly change fctl flags and remove os.O_CLOEXEC: fcntl.fcntl(readfd, fcntl.F_SETFL, fcntl.fcntl(readfd, fcntl.F_GETFL) & ~os.O_CLOEXEC)

ilammy 7y ago

Another point where you have to ignore SIGPIPE is concurrent code that handles multiple fds (say, like a web server). In this case you have to ignore the signal and process EPIPE correctly, because the signal is not associated with a particular fd so you cannot tell which one of them failed.