12 ms·

No. As I say in the blog post, I need to kill all the commands to the right of the failing command, and set -o pipefail and $PIPESTATUS can't do that.

kasabali 4y ago
I'd simply run the pipe in a subshell, go over $PIPESTATUS one by one and kill the subshell if I found a failed one. But if you insist on killing to the right I'm afraid there's no analogue of $PIPESTATUS array providing list of PIDs.
- thomashabets2 4y ago
  > if you insist on killing to the right Well, as described in the blog post it's the entire point, the very problem I'm trying to solve; to notify downstream programs that upstream failed, not finished, before it's had time to act on the fact that its input returned EOF. But I'm open to other solutions to the problem. Until I hear anything else though I claiming that bash pipes are unfit for purpose.
  - kasabali 4y ago
    > Well, as described in the blog post it's the entire point then I'm sorry but I can't understand your point. I don't disagree bash's process management facilities are severely lacking in many respects, but your example ain't one of them. If middle command has failed it must have exited, so did the first and last command. Because it's a pipe. It is what commands in a pipe do. When one process exits so do the others. All you need to know is whether one of the commands has failed (which is provided with pipefail, and PIPESTATUS if you want even more granularity) and you can do cleanup if there was a failure. If for some bizarre reason the rightmost command still continues to run (despite process to its left has exited) or you want to be extra sure, you may run the pipe in a subshell and end it with 'kill -- -$BASHPID'. None of these require meticulously setting up named pipes for each command. I fail to see in what kind of a scenario a command in a pipe would be (or you'd want it to be) still running on the left hand side despite command(s) to its right has exited. But again, if you insist on doing so, whatever floats your boat.
    - thomashabets2 4y ago
      > I don't disagree bash's process management facilities are severely lacking in many respects, but your example ain't one of them. It is. I really did upload bad data because of bash's poor pipe handling. Maybe a better example: generate_data | commit_to_database commit_to_database reads from stdin and writes to a database. When it reaches EOF it'll COMMIT. I don't want to commit something half-generated. Maybe commit_to_database replaces the current set of data with the updated version, serving bad data to users. It needs to know that it's complete. Another example is the classic "curl https:// https://... | bash". This command is of course a problem. But not for the obvious reason. The obvious reason is that you don't know what it does, yet you just run it. But really most people are not going to audit that shellscript anyway. So lets say you actually trust that source. There's still a problem. If the file has a command like "rm -fr /tmp/foo", and the TCP connection is cut right after "rm -fr /" then even though there's no end-of-line that "rm -fr /" is still run. Bash thinks the file just ended, and runs the command. It's "committing" data that was partial, because it thought it was complete. (this is why bash scripts often actually create functions to do this, and then have the very last line of the file run the function. But they don't all do that) > If for some bizarre reason the rightmost command still continues to run (despite process to its left has exited) or you want to be extra sure, you may run the pipe in a subshell and end it with 'kill -- -$BASHPID'. That's still a race condition. And that kills the parent process, not the downstream process, so does not prevent the downstream process from committing. > I fail to see in what kind of a scenario a command in a pipe would be (or you'd want it to be) still running on the left hand side despite command(s) to its right has exited. I would not. I'm trying to communicate an "failure, abort, do not commit" to downstream commands.
      - kasabali 4y ago
        > commit_to_database reads from stdin and writes to a database. When it reaches EOF it'll COMMIT. > I don't want to commit something half-generated. Maybe commit_to_database replaces the current set of data with the updated version, serving bad data to users. Thanks, it's clear now. I (and I assume others recommending pipefail) was puzzled because idiomatic way to do your sorting example in a regular filesystem is: set -o pipefail whatever | false | tee output.tmp if [ $? = 0 ]; then echo Commit mv output.tmp output.sorted else echo Abort rm output.tmp exit 1 fi ...which gsutil command can't because it does the commit step implicitly, with no way to do it explicitly, I gather? But at that point I still don't understand how can you avoid the race condition (regardless of the language) and kill gsutil before it does the implicit commit step. > that kills the parent process, not the downstream process It kills all child processes, including commands started in a pipe.

kasabali 4y ago

I'd simply run the pipe in a subshell, go over $PIPESTATUS one by one and kill the subshell if I found a failed one. But if you insist on killing to the right I'm afraid there's no analogue of $PIPESTATUS array providing list of PIDs.

thomashabets2 4y ago
> if you insist on killing to the right Well, as described in the blog post it's the entire point, the very problem I'm trying to solve; to notify downstream programs that upstream failed, not finished, before it's had time to act on the fact that its input returned EOF. But I'm open to other solutions to the problem. Until I hear anything else though I claiming that bash pipes are unfit for purpose.
- kasabali 4y ago
  > Well, as described in the blog post it's the entire point then I'm sorry but I can't understand your point. I don't disagree bash's process management facilities are severely lacking in many respects, but your example ain't one of them. If middle command has failed it must have exited, so did the first and last command. Because it's a pipe. It is what commands in a pipe do. When one process exits so do the others. All you need to know is whether one of the commands has failed (which is provided with pipefail, and PIPESTATUS if you want even more granularity) and you can do cleanup if there was a failure. If for some bizarre reason the rightmost command still continues to run (despite process to its left has exited) or you want to be extra sure, you may run the pipe in a subshell and end it with 'kill -- -$BASHPID'. None of these require meticulously setting up named pipes for each command. I fail to see in what kind of a scenario a command in a pipe would be (or you'd want it to be) still running on the left hand side despite command(s) to its right has exited. But again, if you insist on doing so, whatever floats your boat.
  - thomashabets2 4y ago
    > I don't disagree bash's process management facilities are severely lacking in many respects, but your example ain't one of them. It is. I really did upload bad data because of bash's poor pipe handling. Maybe a better example: generate_data | commit_to_database commit_to_database reads from stdin and writes to a database. When it reaches EOF it'll COMMIT. I don't want to commit something half-generated. Maybe commit_to_database replaces the current set of data with the updated version, serving bad data to users. It needs to know that it's complete. Another example is the classic "curl https:// https://... | bash". This command is of course a problem. But not for the obvious reason. The obvious reason is that you don't know what it does, yet you just run it. But really most people are not going to audit that shellscript anyway. So lets say you actually trust that source. There's still a problem. If the file has a command like "rm -fr /tmp/foo", and the TCP connection is cut right after "rm -fr /" then even though there's no end-of-line that "rm -fr /" is still run. Bash thinks the file just ended, and runs the command. It's "committing" data that was partial, because it thought it was complete. (this is why bash scripts often actually create functions to do this, and then have the very last line of the file run the function. But they don't all do that) > If for some bizarre reason the rightmost command still continues to run (despite process to its left has exited) or you want to be extra sure, you may run the pipe in a subshell and end it with 'kill -- -$BASHPID'. That's still a race condition. And that kills the parent process, not the downstream process, so does not prevent the downstream process from committing. > I fail to see in what kind of a scenario a command in a pipe would be (or you'd want it to be) still running on the left hand side despite command(s) to its right has exited. I would not. I'm trying to communicate an "failure, abort, do not commit" to downstream commands.
    - kasabali 4y ago
      > commit_to_database reads from stdin and writes to a database. When it reaches EOF it'll COMMIT. > I don't want to commit something half-generated. Maybe commit_to_database replaces the current set of data with the updated version, serving bad data to users. Thanks, it's clear now. I (and I assume others recommending pipefail) was puzzled because idiomatic way to do your sorting example in a regular filesystem is: set -o pipefail whatever | false | tee output.tmp if [ $? = 0 ]; then echo Commit mv output.tmp output.sorted else echo Abort rm output.tmp exit 1 fi ...which gsutil command can't because it does the commit step implicitly, with no way to do it explicitly, I gather? But at that point I still don't understand how can you avoid the race condition (regardless of the language) and kill gsutil before it does the implicit commit step. > that kills the parent process, not the downstream process It kills all child processes, including commands started in a pipe.