6 ms·

This article promotes bad practices for dealing with SIGPIPE. 1. Closing stderr in Python is not a good idea because that’ll swallow any other errors that occu

This article promotes bad practices for dealing with SIGPIPE. 1. Closing stderr in Python is not a good idea because that’ll swallow any other errors that occur at exit. Redirecting stdout to devnull is really just a way to prevent the flushed output from going to the now-closed stdout and triggering another SIGPIPE. That’s more preferable than closing stderr and losing error output at exit. 2. Ignoring SIGPIPE is a terrible idea for a process that should do stream processing. Try making a yes clone and ignoring SIGPIPE - your process will likely run forever trying to shove “y” into a closed pipe. There’s a reason SIGPIPE was invented! Very few programs bother to check the return value from write/printf/etc.

v3gas 7y ago
Thanks! So the preferred way is to redirect to dev null?
- hoytech 7y ago
  It's better to just do nothing and allow SIGPIPE to kill your program. That is the reason this signal exists. Python is not a good unix citizen in this case. Compare it to Perl for example where nothing special is required to do the right thing: $ perl -E 'say "y" while 1' | head -1 y $
  - v3gas 7y ago
    I see
  - hk__2 7y ago
    > Python is not a good unix citizen in this case. Compare it to Perl for example where nothing special is required to do the right thing How is this better? At least with the exception you can cleanly terminate your program if you need to.
    - hoytech 7y ago
      In the unix pipeline model, components read input, perform some transformation, and print output. There is no need to do any clean-up. As nneonneo explained, because they rely on SIGPIPE, most unix programs don't even need to check for errors from write(2). Naturally, in the rare case you want to do some clean-up you can: $ perl -E '$SIG{PIPE} = sub { die "cleanup" }; say "y" while 1' | head -1 y cleanup at -e line 1.
  - seneca 7y ago
    I disagree. Saying python is "not a good Unix citizen" here is akin to saying cars are a bad highway citizen because people can crash them. Python provides mechanisms to handle signals. The point of a signal is to indicate something outside of your process has happened and you may want to respond to it. It's up to the program to respond to relevant signals and Python in no way stops a developer from doing that.
    - hoytech 7y ago
      > Saying python is "not a good Unix citizen" here is akin to saying cars are a bad highway citizen because people can crash them. I don't follow your analogy. The default signal disposition for SIGPIPE is to silently terminate the process. Normally processes compose nicely on the command-line and this requires no extra work from the programmer. By disregarding this convention, most Python programs pollute your terminal when used in pipelines, which is why I claim that in this case Python is a bad unix citizen.
      - seneca 7y ago
        Yep, I follow your logic. As you say, in other languages it "requires no extra work", it's more a matter of convenience. Python in no way stops you from a having a typical response to SIGPIPE, it's as simple as handling the exception and doing a sys.exit(), but it doesn't "just happen". This makes the typical path a little more contrived, but I don't think having explicit handling makes it a bad citizen. To torture my analogy, in my mind a "bad citizen" of the highway may be a car incapable of doing the minimum speed limit, whereas Python just has a manual transmission vs Perl's automatic (in this case). I'm splitting hairs though, your point is fair. I just think your objection is a little strong for what comes down to convenience.
        derefr 7y ago
        It's not a matter of "convenience" any more, when you know for a fact that nobody dashing off quick five-line scripts in your language is going to bother doing the extra thing, and so the default behavior will be the only behavior for those scripts. Defaults matter. It's like the difference between opt-in and opt-out for organ donation, or the difference between encouraging and requiring cars to have airbags.
  - nonofr34 7y ago
    Not an expert is no way, so I could be wrong, but in python could you use signal.SIG_DFL to have that expected behavior: # cat.py import sys from signal import signal, SIGPIPE, SIG_DFL signal(SIGPIPE,SIG_DFL) for line in sys.stdin: print(line)
    - hoytech 7y ago
      True, but most Python programs don't do this, so they pollute your terminal with error messages when you attempt to use them as interior processes in a unix pipeline.
    - gnomewascool 7y ago
      That seems to be discouraged by the official python docs.[0] [0] https://docs.python.org/3/library/signal.html#note-on-sigpipe https://docs.python.org/3/library/signal.html#note-on-sigpip...
      - jolmg 7y ago
        > sys.exit(1) # Python exits with error code 1 on EPIPE Should be 141 instead of 1. Convention is that when a program dies because of a signal, the exit code should be 128 + the signal number, in this case 13. So, 128 + 13 = 141. Here are some examples: $ ( yes; >&2 echo $? ) | head -0 141 $ ( ping localhost; >&2 echo $? ) | head -0 141 $ ( perl -E 'say "y" while 1'; >&2 echo $? ) | head -0 141 In python, when using signal(SIGPIPE, SIG_DFL), you get the correct behaviour (at least regarding the exit status): $ ( python -c ' import sys from signal import signal, SIGPIPE, SIG_DFL signal(SIGPIPE,SIG_DFL) while True: print("y"); ' ; >&2 echo $? ) | head -0 141 > Do not set SIGPIPE’s disposition to SIG_DFL in order to avoid BrokenPipeError. Doing that would cause your program to exit unexpectedly also whenever any socket connection is interrupted while your program is still writing to it. Regarding that socket behavior, if that's the standard in other programming environments, why is it an issue from python's perspective? Doesn't seem worth eschewing standard unix conventions. I mean, it says "unexpectedly", but isn't it actually the expected behavior? Unexpected is for python to say that what's default (SIG_DFL) is unexpected.
  - dooglius 7y ago
    This is bad advice, if you are doing anything other than output to stdout/stderr (which you almost assuredly are, unless you are doing something very simple like "yes"), you want to switch to /dev/null. For instance, running rm -vfr folder/ | head involves SIGPIPE causing problems because it may or may not kill rm before it finishes deleting the directory, based on its internal output buffer size.
    - hoytech 7y ago
      The problem here is connecting a program you want to run to completion to the head command, not how rm handles SIGPIPE.
      - dooglius 7y ago
        head is just an example, if you pipe rm -vfr (or any program that does more than just output) into _anything_ the reasonable and default behavior should be to run rm to completion. (And from a programmer's POV, the reasonably thing is for syscalls to handle errors by returning an error, the way every other error is handled.)
        kees99 7y ago
        Wouldn't it be better to be explicit? If intention is to "display first 10 lines, at most; discard the rest", you can write: rm -vrf directory/ | (head; cat >/dev/null)
        dooglius 7y ago
        I guess it depends on your semantics/mental model of what a pipeline should do. My mental model says that each part should run fully, and the pipeline should emulate sequential behavior, with "yes" being a bit of a hack in this regard. Cases where the behavior of a pipeline depends on the internal buffer size and operation ordering (e.g. what if rm decided to print everything it did only after removing everything?) seem like bugs to me. Maybe your mental model is different? In any case, I think that in the vast majority of pipelines, the default SIGPIPE behavior is not what is desired.
seneca 7y ago
Agreed. I like articles like this because they show the learning process, which I think is super valuable. However, they really need a big disclaimer stating the author is experimenting and doesn't know the correct answer. Otherwise people stumble upon it and take it as authoritative.
- v3gas 7y ago
  Great point, I should probably add that disclaimer!
developer2 7y ago
The biggest takeaway here IMO is that Python breaks the standard contract regarding signals–at least for SIGPIPE. Python should not be catching and throwing an exception for SIGPIPE; it should simply exit immediately, which is literally the default POSIX behaviour... unless a script/program/process specifically installs a signal handler to perform cleanup before exit. Python has some pretty awful behaviours built into it, and this is one of them. Half of this article is not "How do Unix pipes work", but "how to fix broken SIGPIPE handling in Python".
- loeg 7y ago
  Python doesn't catch SIGPIPE, it ignores it. The exception is raised from a -1/EPIPE return from libc write(). I fully agree that Python is often a bad citizen in terms of signal handling — it wants to only process signals on 'the main thread', but also wants end-users to fully control signal-handling. The two ideas are sort of at-odds and in general I find handling signals in Python frustrating.
loeg 7y ago
> 1. Closing stderr in Python is not a good idea because that’ll swallow any other errors that occur at exit. Redirecting stdout to devnull is really just a way to prevent the flushed output from going to the now-closed stdout and triggering another SIGPIPE. That’s more preferable than closing stderr and losing error output at exit. I don't follow. The standard for EPIPE/SIGPIPE handling is to silently exit with an error status. It's fine to close stderr to prevent spurious warning messages about flushing stdout. > 2. Ignoring SIGPIPE is a terrible idea for a process that should do stream processing. Try making a yes clone and ignoring SIGPIPE - your process will likely run forever trying to shove “y” into a closed pipe. There’s a reason SIGPIPE was invented! Very few programs bother to check the return value from write/printf/etc. Programs can correctly handle lost pipes masking SIGPIPE entirely, with error checking alone. Python's BrokenPipeError is raised on the basis of EPIPE, not SIGPIPE. Re: programs not checking error returns of write() and close(): that is not really true in a language like Python with exceptions raised on IO errors. It always does the check, and the unwinder aborts the program if nothing handles the error. Sigpipe is completely unnecessary for Python programs. (It's also not necessary for C programs, but I guess AT&T didn't want to fix their programs to check for errors.)
- nneonneo 7y ago
  The suppression of SIGPIPE was done in the Go code, not in the Python code. Does the POSIX standard mandate that programs receiving EPIPE/SIGPIPE die silently? I don’t know of such a rule, and there’s plenty of programs that violate this. Python is a bit too verbose with the errors (with a full trace back and two copies of the error) so suppressing those errors somehow seems like a good idea for a general-purpose command line tool.
  - userbinator 7y ago
    Does the POSIX standard mandate that programs receiving EPIPE/SIGPIPE die silently? https://pubs.opengroup.org/onlinepubs/7908799/xsh/signal.h.html https://pubs.opengroup.org/onlinepubs/7908799/xsh/signal.h.h... Yes. The default action of SIGPIPE is to terminate the process.
pixelbeat__ 7y ago
Agreed. Correct handling of SIGPIPE is quite subtle and often done incorrectly. I've some notes on SIGPIPE considerations at: http://www.pixelbeat.org/programming/sigpipe_handling.html http://www.pixelbeat.org/programming/sigpipe_handling.html
asveikau 7y ago
The biggest reason I can think of not to close stderr (or any other of the standard handles) is that the next open(2) call is likely to get that same descriptor recycled. So now joe random open file is going to receive all the error messages from random libraries, or perhaps even from unrelated programs you may have forked with that file as fd 2.
microtherion 7y ago
Another bad practice, IMHO, is not to quit the program once the exception is thrown. Instead, the loop continues and the rest of the input gets fed to /dev/null

v3gas 7y ago

Thanks! So the preferred way is to redirect to dev null?

hoytech 7y ago
It's better to just do nothing and allow SIGPIPE to kill your program. That is the reason this signal exists. Python is not a good unix citizen in this case. Compare it to Perl for example where nothing special is required to do the right thing: $ perl -E 'say "y" while 1' | head -1 y $
- v3gas 7y ago
  I see
- hk__2 7y ago
  > Python is not a good unix citizen in this case. Compare it to Perl for example where nothing special is required to do the right thing How is this better? At least with the exception you can cleanly terminate your program if you need to.
  - hoytech 7y ago
    In the unix pipeline model, components read input, perform some transformation, and print output. There is no need to do any clean-up. As nneonneo explained, because they rely on SIGPIPE, most unix programs don't even need to check for errors from write(2). Naturally, in the rare case you want to do some clean-up you can: $ perl -E '$SIG{PIPE} = sub { die "cleanup" }; say "y" while 1' | head -1 y cleanup at -e line 1.
- seneca 7y ago
  I disagree. Saying python is "not a good Unix citizen" here is akin to saying cars are a bad highway citizen because people can crash them. Python provides mechanisms to handle signals. The point of a signal is to indicate something outside of your process has happened and you may want to respond to it. It's up to the program to respond to relevant signals and Python in no way stops a developer from doing that.
  - hoytech 7y ago
    > Saying python is "not a good Unix citizen" here is akin to saying cars are a bad highway citizen because people can crash them. I don't follow your analogy. The default signal disposition for SIGPIPE is to silently terminate the process. Normally processes compose nicely on the command-line and this requires no extra work from the programmer. By disregarding this convention, most Python programs pollute your terminal when used in pipelines, which is why I claim that in this case Python is a bad unix citizen.
    - seneca 7y ago
      Yep, I follow your logic. As you say, in other languages it "requires no extra work", it's more a matter of convenience. Python in no way stops you from a having a typical response to SIGPIPE, it's as simple as handling the exception and doing a sys.exit(), but it doesn't "just happen". This makes the typical path a little more contrived, but I don't think having explicit handling makes it a bad citizen. To torture my analogy, in my mind a "bad citizen" of the highway may be a car incapable of doing the minimum speed limit, whereas Python just has a manual transmission vs Perl's automatic (in this case). I'm splitting hairs though, your point is fair. I just think your objection is a little strong for what comes down to convenience.
      - derefr 7y ago
        It's not a matter of "convenience" any more, when you know for a fact that nobody dashing off quick five-line scripts in your language is going to bother doing the extra thing, and so the default behavior will be the only behavior for those scripts. Defaults matter. It's like the difference between opt-in and opt-out for organ donation, or the difference between encouraging and requiring cars to have airbags.
- nonofr34 7y ago
  Not an expert is no way, so I could be wrong, but in python could you use signal.SIG_DFL to have that expected behavior: # cat.py import sys from signal import signal, SIGPIPE, SIG_DFL signal(SIGPIPE,SIG_DFL) for line in sys.stdin: print(line)
  - hoytech 7y ago
    True, but most Python programs don't do this, so they pollute your terminal with error messages when you attempt to use them as interior processes in a unix pipeline.
  - gnomewascool 7y ago
    That seems to be discouraged by the official python docs.[0] [0] https://docs.python.org/3/library/signal.html#note-on-sigpipe https://docs.python.org/3/library/signal.html#note-on-sigpip...
    - jolmg 7y ago
      > sys.exit(1) # Python exits with error code 1 on EPIPE Should be 141 instead of 1. Convention is that when a program dies because of a signal, the exit code should be 128 + the signal number, in this case 13. So, 128 + 13 = 141. Here are some examples: $ ( yes; >&2 echo $? ) | head -0 141 $ ( ping localhost; >&2 echo $? ) | head -0 141 $ ( perl -E 'say "y" while 1'; >&2 echo $? ) | head -0 141 In python, when using signal(SIGPIPE, SIG_DFL), you get the correct behaviour (at least regarding the exit status): $ ( python -c ' import sys from signal import signal, SIGPIPE, SIG_DFL signal(SIGPIPE,SIG_DFL) while True: print("y"); ' ; >&2 echo $? ) | head -0 141 > Do not set SIGPIPE’s disposition to SIG_DFL in order to avoid BrokenPipeError. Doing that would cause your program to exit unexpectedly also whenever any socket connection is interrupted while your program is still writing to it. Regarding that socket behavior, if that's the standard in other programming environments, why is it an issue from python's perspective? Doesn't seem worth eschewing standard unix conventions. I mean, it says "unexpectedly", but isn't it actually the expected behavior? Unexpected is for python to say that what's default (SIG_DFL) is unexpected.
- dooglius 7y ago
  This is bad advice, if you are doing anything other than output to stdout/stderr (which you almost assuredly are, unless you are doing something very simple like "yes"), you want to switch to /dev/null. For instance, running rm -vfr folder/ | head involves SIGPIPE causing problems because it may or may not kill rm before it finishes deleting the directory, based on its internal output buffer size.
  - hoytech 7y ago
    The problem here is connecting a program you want to run to completion to the head command, not how rm handles SIGPIPE.
    - dooglius 7y ago
      head is just an example, if you pipe rm -vfr (or any program that does more than just output) into _anything_ the reasonable and default behavior should be to run rm to completion. (And from a programmer's POV, the reasonably thing is for syscalls to handle errors by returning an error, the way every other error is handled.)
      - kees99 7y ago
        Wouldn't it be better to be explicit? If intention is to "display first 10 lines, at most; discard the rest", you can write: rm -vrf directory/ | (head; cat >/dev/null)
        dooglius 7y ago
        I guess it depends on your semantics/mental model of what a pipeline should do. My mental model says that each part should run fully, and the pipeline should emulate sequential behavior, with "yes" being a bit of a hack in this regard. Cases where the behavior of a pipeline depends on the internal buffer size and operation ordering (e.g. what if rm decided to print everything it did only after removing everything?) seem like bugs to me. Maybe your mental model is different? In any case, I think that in the vast majority of pipelines, the default SIGPIPE behavior is not what is desired.

seneca 7y ago

Agreed. I like articles like this because they show the learning process, which I think is super valuable. However, they really need a big disclaimer stating the author is experimenting and doesn't know the correct answer. Otherwise people stumble upon it and take it as authoritative.

v3gas 7y ago
Great point, I should probably add that disclaimer!

developer2 7y ago

The biggest takeaway here IMO is that Python breaks the standard contract regarding signals–at least for SIGPIPE. Python should not be catching and throwing an exception for SIGPIPE; it should simply exit immediately, which is literally the default POSIX behaviour... unless a script/program/process specifically installs a signal handler to perform cleanup before exit. Python has some pretty awful behaviours built into it, and this is one of them. Half of this article is not "How do Unix pipes work", but "how to fix broken SIGPIPE handling in Python".

loeg 7y ago
Python doesn't catch SIGPIPE, it ignores it. The exception is raised from a -1/EPIPE return from libc write(). I fully agree that Python is often a bad citizen in terms of signal handling — it wants to only process signals on 'the main thread', but also wants end-users to fully control signal-handling. The two ideas are sort of at-odds and in general I find handling signals in Python frustrating.

loeg 7y ago

> 1. Closing stderr in Python is not a good idea because that’ll swallow any other errors that occur at exit. Redirecting stdout to devnull is really just a way to prevent the flushed output from going to the now-closed stdout and triggering another SIGPIPE. That’s more preferable than closing stderr and losing error output at exit. I don't follow. The standard for EPIPE/SIGPIPE handling is to silently exit with an error status. It's fine to close stderr to prevent spurious warning messages about flushing stdout. > 2. Ignoring SIGPIPE is a terrible idea for a process that should do stream processing. Try making a yes clone and ignoring SIGPIPE - your process will likely run forever trying to shove “y” into a closed pipe. There’s a reason SIGPIPE was invented! Very few programs bother to check the return value from write/printf/etc. Programs can correctly handle lost pipes masking SIGPIPE entirely, with error checking alone. Python's BrokenPipeError is raised on the basis of EPIPE, not SIGPIPE. Re: programs not checking error returns of write() and close(): that is not really true in a language like Python with exceptions raised on IO errors. It always does the check, and the unwinder aborts the program if nothing handles the error. Sigpipe is completely unnecessary for Python programs. (It's also not necessary for C programs, but I guess AT&T didn't want to fix their programs to check for errors.)

nneonneo 7y ago
The suppression of SIGPIPE was done in the Go code, not in the Python code. Does the POSIX standard mandate that programs receiving EPIPE/SIGPIPE die silently? I don’t know of such a rule, and there’s plenty of programs that violate this. Python is a bit too verbose with the errors (with a full trace back and two copies of the error) so suppressing those errors somehow seems like a good idea for a general-purpose command line tool.
- userbinator 7y ago
  Does the POSIX standard mandate that programs receiving EPIPE/SIGPIPE die silently? https://pubs.opengroup.org/onlinepubs/7908799/xsh/signal.h.html https://pubs.opengroup.org/onlinepubs/7908799/xsh/signal.h.h... Yes. The default action of SIGPIPE is to terminate the process.

pixelbeat__ 7y ago

Agreed. Correct handling of SIGPIPE is quite subtle and often done incorrectly. I've some notes on SIGPIPE considerations at: http://www.pixelbeat.org/programming/sigpipe_handling.html http://www.pixelbeat.org/programming/sigpipe_handling.html

asveikau 7y ago

The biggest reason I can think of not to close stderr (or any other of the standard handles) is that the next open(2) call is likely to get that same descriptor recycled. So now joe random open file is going to receive all the error messages from random libraries, or perhaps even from unrelated programs you may have forked with that file as fd 2.

microtherion 7y ago

Another bad practice, IMHO, is not to quit the program once the exception is thrown. Instead, the loop continues and the rest of the input gets fed to /dev/null