4 ms·

Can't you just refer to these as /proc/self/fd/N ? Or am I missing what kind of support you need?

fluidcruft 4y ago
I'm not sure and I'm not great with the semantics of these features. For example there's the process-feeds a file type: command <(other command) Where "command" sees the output of "other command" as a file. But other times you do things like rename or use tee to duplicate data to different descriptors and rename them back and forth to jump over one sub command and get used in a different subshell to produce different output from the same input from a different command. I think part of this is called "moving file descriptors". It's one of the things that c-shells totally suck at. Usually in bash it looks cleanest to break these up into bash functions and then assemble one huge pipe based on that. Like run command rename stdout to like file descriptor six to jump over the next command and then swap them around. This mostly comes up when I'm writing things that trawl through very slow archives or filesystems to avoid multiple passes. It's hard to think of a simple example, but being clever often cuts processing time by an order of magnitude. Anyway often "pipe" is probably a bad description. Pipes often look more like chains but what I really want more often is more of a directed graph where processes are nodes and data flows in the top and out to different file descriptors that connect to different nodes. Conceptually it isn't complex--each node has stream inputs and outputs that just need to be connected. It's the semantics of it that suck. So I sort of liked the json description approach in goodpipe, and it would be cool if there were ways to be more flexible about this. goodpipe as is seems limited to the c-shell style of simplistic pipes. But... maybe this is possible with named file descriptors in goodpipe. I'll have to think about it.
- thomashabets2 4y ago
  Ah, those features. Gotcha. I only implemented a straight pipeline, no forking, merging, or loops. Mainly because I know enough about pipeline technologies to know that this means creating a language that won't be simple, because the problem isn't simple. I agree, ideally any pipeline should be expressable. Without turning into something better solved by Apache Beam or other existing streaming or batch pipelines. The main problem goodpipe tries to solve is the ordered termination of pipeline parts, so that the output bit doesn't finalize its output when the input closes. In the general directed graph case it would be necessary to define the order of termination. So, in short, the problems here are to define the scope of problems/digraphs, design a language that allows expressing that (yet be simple enough to get right), and then the implementation is the easy part. If you want to help improve goodpipe to handle more use cases then I'd be happy to work with you. Or if you want to make something new then I can cheer from the sidelines too. :-)

I'm not sure and I'm not great with the semantics of these features. For example there's the process-feeds a file type: command <(other command) Where "command" sees the output of "other command" as a file. But other times you do things like rename or use tee to duplicate data to different descriptors and rename them back and forth to jump over one sub command and get used in a different subshell to produce different output from the same input from a different command. I think part of this is called "moving file descriptors". It's one of the things that c-shells totally suck at. Usually in bash it looks cleanest to break these up into bash functions and then assemble one huge pipe based on that. Like run command rename stdout to like file descriptor six to jump over the next command and then swap them around. This mostly comes up when I'm writing things that trawl through very slow archives or filesystems to avoid multiple passes. It's hard to think of a simple example, but being clever often cuts processing time by an order of magnitude. Anyway often "pipe" is probably a bad description. Pipes often look more like chains but what I really want more often is more of a directed graph where processes are nodes and data flows in the top and out to different file descriptors that connect to different nodes. Conceptually it isn't complex--each node has stream inputs and outputs that just need to be connected. It's the semantics of it that suck. So I sort of liked the json description approach in goodpipe, and it would be cool if there were ways to be more flexible about this. goodpipe as is seems limited to the c-shell style of simplistic pipes. But... maybe this is possible with named file descriptors in goodpipe. I'll have to think about it.

thomashabets2 4y ago
Ah, those features. Gotcha. I only implemented a straight pipeline, no forking, merging, or loops. Mainly because I know enough about pipeline technologies to know that this means creating a language that won't be simple, because the problem isn't simple. I agree, ideally any pipeline should be expressable. Without turning into something better solved by Apache Beam or other existing streaming or batch pipelines. The main problem goodpipe tries to solve is the ordered termination of pipeline parts, so that the output bit doesn't finalize its output when the input closes. In the general directed graph case it would be necessary to define the order of termination. So, in short, the problems here are to define the scope of problems/digraphs, design a language that allows expressing that (yet be simple enough to get right), and then the implementation is the easy part. If you want to help improve goodpipe to handle more use cases then I'd be happy to work with you. Or if you want to make something new then I can cheer from the sidelines too. :-)