8 ms·

This is tangental, but I would say never trust bash pipes. I wrote a tool to get around the problem: https://github.com/ThomasHabets/goodpipe https://github.c

This is tangental, but I would say never trust bash pipes. I wrote a tool to get around the problem: https://github.com/ThomasHabets/goodpipe https://github.com/ThomasHabets/goodpipe I have more info about the problem in the README and this blog post: https://blog.habets.se/2021/06/The-uselessness-of-bash.html https://blog.habets.se/2021/06/The-uselessness-of-bash.html But tl;dr: If gpg fails in your example, then zfs receive will just think "oh that was all the data, apparently". Hopefully zfs receive is fine with that, and will notice it. But most tools would not.

fluidcruft 4y ago
goodpipe looks interesting! My first thought is that it seems to lack support for bash file descriptor things that I often rely on.
- thomashabets2 4y ago
  Can't you just refer to these as /proc/self/fd/N ? Or am I missing what kind of support you need?
  - fluidcruft 4y ago
    I'm not sure and I'm not great with the semantics of these features. For example there's the process-feeds a file type: command <(other command) Where "command" sees the output of "other command" as a file. But other times you do things like rename or use tee to duplicate data to different descriptors and rename them back and forth to jump over one sub command and get used in a different subshell to produce different output from the same input from a different command. I think part of this is called "moving file descriptors". It's one of the things that c-shells totally suck at. Usually in bash it looks cleanest to break these up into bash functions and then assemble one huge pipe based on that. Like run command rename stdout to like file descriptor six to jump over the next command and then swap them around. This mostly comes up when I'm writing things that trawl through very slow archives or filesystems to avoid multiple passes. It's hard to think of a simple example, but being clever often cuts processing time by an order of magnitude. Anyway often "pipe" is probably a bad description. Pipes often look more like chains but what I really want more often is more of a directed graph where processes are nodes and data flows in the top and out to different file descriptors that connect to different nodes. Conceptually it isn't complex--each node has stream inputs and outputs that just need to be connected. It's the semantics of it that suck. So I sort of liked the json description approach in goodpipe, and it would be cool if there were ways to be more flexible about this. goodpipe as is seems limited to the c-shell style of simplistic pipes. But... maybe this is possible with named file descriptors in goodpipe. I'll have to think about it.
    - thomashabets2 4y ago
      Ah, those features. Gotcha. I only implemented a straight pipeline, no forking, merging, or loops. Mainly because I know enough about pipeline technologies to know that this means creating a language that won't be simple, because the problem isn't simple. I agree, ideally any pipeline should be expressable. Without turning into something better solved by Apache Beam or other existing streaming or batch pipelines. The main problem goodpipe tries to solve is the ordered termination of pipeline parts, so that the output bit doesn't finalize its output when the input closes. In the general directed graph case it would be necessary to define the order of termination. So, in short, the problems here are to define the scope of problems/digraphs, design a language that allows expressing that (yet be simple enough to get right), and then the implementation is the easy part. If you want to help improve goodpipe to handle more use cases then I'd be happy to work with you. Or if you want to make something new then I can cheer from the sidelines too. :-)
nolist_policy 4y ago
set -o pipefail is your friend
- thomashabets2 4y ago
  I mention this in the blog post. It doesn't fix the big problem. If fixes a smaller problem only.
Joker_vD 4y ago
"Hey, I want to run N copies of this one program in parallel, and I also would like to be able to press Ctrl+C to stop them all at once, surely it's easy?" Yes, but only if you know the right spell: (trap 'kill 0' SIGINT; for i in {1..$N} ; do this_one_program $i & done ; wait) I've seen quote enough of coworkers who struggled trying to Ctrl-C the &-ed copies of programs from a single shell, only to give up and run them from several xterm windows/Terminator tabs instead, can't really blame them.
- Filligree 4y ago
  GNU Parallel handles this nicely, and is quite a powerful GNU toolbox of GNU as well. I use it quite a lot. GNU. Don't mix it up with the (IIRC) Suckless tool of the same name, which is significantly less powerful. GNU. Just be prepared to silence the GNU.
  - metadat 4y ago
    I don't use gnu parallel because it is not installed by default. Xargs on the other hand, is.
kasabali 4y ago
Would $PIPESTATUS be a solution for the 'sort fails' issue in your blog post?
- thomashabets2 4y ago
  No. As I say in the blog post, I need to kill all the commands to the right of the failing command, and set -o pipefail and $PIPESTATUS can't do that.
  - kasabali 4y ago
    I'd simply run the pipe in a subshell, go over $PIPESTATUS one by one and kill the subshell if I found a failed one. But if you insist on killing to the right I'm afraid there's no analogue of $PIPESTATUS array providing list of PIDs.
    - thomashabets2 4y ago
      > if you insist on killing to the right Well, as described in the blog post it's the entire point, the very problem I'm trying to solve; to notify downstream programs that upstream failed, not finished, before it's had time to act on the fact that its input returned EOF. But I'm open to other solutions to the problem. Until I hear anything else though I claiming that bash pipes are unfit for purpose.
      - kasabali 4y ago
        > Well, as described in the blog post it's the entire point then I'm sorry but I can't understand your point. I don't disagree bash's process management facilities are severely lacking in many respects, but your example ain't one of them. If middle command has failed it must have exited, so did the first and last command. Because it's a pipe. It is what commands in a pipe do. When one process exits so do the others. All you need to know is whether one of the commands has failed (which is provided with pipefail, and PIPESTATUS if you want even more granularity) and you can do cleanup if there was a failure. If for some bizarre reason the rightmost command still continues to run (despite process to its left has exited) or you want to be extra sure, you may run the pipe in a subshell and end it with 'kill -- -$BASHPID'. None of these require meticulously setting up named pipes for each command. I fail to see in what kind of a scenario a command in a pipe would be (or you'd want it to be) still running on the left hand side despite command(s) to its right has exited. But again, if you insist on doing so, whatever floats your boat.

fluidcruft 4y ago

goodpipe looks interesting! My first thought is that it seems to lack support for bash file descriptor things that I often rely on.

thomashabets2 4y ago
Can't you just refer to these as /proc/self/fd/N ? Or am I missing what kind of support you need?
- fluidcruft 4y ago
  I'm not sure and I'm not great with the semantics of these features. For example there's the process-feeds a file type: command <(other command) Where "command" sees the output of "other command" as a file. But other times you do things like rename or use tee to duplicate data to different descriptors and rename them back and forth to jump over one sub command and get used in a different subshell to produce different output from the same input from a different command. I think part of this is called "moving file descriptors". It's one of the things that c-shells totally suck at. Usually in bash it looks cleanest to break these up into bash functions and then assemble one huge pipe based on that. Like run command rename stdout to like file descriptor six to jump over the next command and then swap them around. This mostly comes up when I'm writing things that trawl through very slow archives or filesystems to avoid multiple passes. It's hard to think of a simple example, but being clever often cuts processing time by an order of magnitude. Anyway often "pipe" is probably a bad description. Pipes often look more like chains but what I really want more often is more of a directed graph where processes are nodes and data flows in the top and out to different file descriptors that connect to different nodes. Conceptually it isn't complex--each node has stream inputs and outputs that just need to be connected. It's the semantics of it that suck. So I sort of liked the json description approach in goodpipe, and it would be cool if there were ways to be more flexible about this. goodpipe as is seems limited to the c-shell style of simplistic pipes. But... maybe this is possible with named file descriptors in goodpipe. I'll have to think about it.
  - thomashabets2 4y ago
    Ah, those features. Gotcha. I only implemented a straight pipeline, no forking, merging, or loops. Mainly because I know enough about pipeline technologies to know that this means creating a language that won't be simple, because the problem isn't simple. I agree, ideally any pipeline should be expressable. Without turning into something better solved by Apache Beam or other existing streaming or batch pipelines. The main problem goodpipe tries to solve is the ordered termination of pipeline parts, so that the output bit doesn't finalize its output when the input closes. In the general directed graph case it would be necessary to define the order of termination. So, in short, the problems here are to define the scope of problems/digraphs, design a language that allows expressing that (yet be simple enough to get right), and then the implementation is the easy part. If you want to help improve goodpipe to handle more use cases then I'd be happy to work with you. Or if you want to make something new then I can cheer from the sidelines too. :-)

nolist_policy 4y ago

set -o pipefail is your friend

thomashabets2 4y ago
I mention this in the blog post. It doesn't fix the big problem. If fixes a smaller problem only.

Joker_vD 4y ago

"Hey, I want to run N copies of this one program in parallel, and I also would like to be able to press Ctrl+C to stop them all at once, surely it's easy?" Yes, but only if you know the right spell: (trap 'kill 0' SIGINT; for i in {1..$N} ; do this_one_program $i & done ; wait) I've seen quote enough of coworkers who struggled trying to Ctrl-C the &-ed copies of programs from a single shell, only to give up and run them from several xterm windows/Terminator tabs instead, can't really blame them.

Filligree 4y ago
GNU Parallel handles this nicely, and is quite a powerful GNU toolbox of GNU as well. I use it quite a lot. GNU. Don't mix it up with the (IIRC) Suckless tool of the same name, which is significantly less powerful. GNU. Just be prepared to silence the GNU.
- metadat 4y ago
  I don't use gnu parallel because it is not installed by default. Xargs on the other hand, is.

kasabali 4y ago

Would $PIPESTATUS be a solution for the 'sort fails' issue in your blog post?

thomashabets2 4y ago
No. As I say in the blog post, I need to kill all the commands to the right of the failing command, and set -o pipefail and $PIPESTATUS can't do that.
- kasabali 4y ago
  I'd simply run the pipe in a subshell, go over $PIPESTATUS one by one and kill the subshell if I found a failed one. But if you insist on killing to the right I'm afraid there's no analogue of $PIPESTATUS array providing list of PIDs.
  - thomashabets2 4y ago
    > if you insist on killing to the right Well, as described in the blog post it's the entire point, the very problem I'm trying to solve; to notify downstream programs that upstream failed, not finished, before it's had time to act on the fact that its input returned EOF. But I'm open to other solutions to the problem. Until I hear anything else though I claiming that bash pipes are unfit for purpose.
    - kasabali 4y ago
      > Well, as described in the blog post it's the entire point then I'm sorry but I can't understand your point. I don't disagree bash's process management facilities are severely lacking in many respects, but your example ain't one of them. If middle command has failed it must have exited, so did the first and last command. Because it's a pipe. It is what commands in a pipe do. When one process exits so do the others. All you need to know is whether one of the commands has failed (which is provided with pipefail, and PIPESTATUS if you want even more granularity) and you can do cleanup if there was a failure. If for some bizarre reason the rightmost command still continues to run (despite process to its left has exited) or you want to be extra sure, you may run the pipe in a subshell and end it with 'kill -- -$BASHPID'. None of these require meticulously setting up named pipes for each command. I fail to see in what kind of a scenario a command in a pipe would be (or you'd want it to be) still running on the left hand side despite command(s) to its right has exited. But again, if you insist on doing so, whatever floats your boat.