4 ms·
Ah, those features. Gotcha. I only implemented a straight pipeline, no forking, merging, or loops. Mainly because I know enough about pipeline technologies to
by thomashabets2 4y ago
Ah, those features. Gotcha.
I only implemented a straight pipeline, no forking, merging, or loops. Mainly because I know enough about pipeline technologies to know that this means creating a language that won't be simple, because the problem isn't simple.
I agree, ideally any pipeline should be expressable. Without turning into something better solved by Apache Beam or other existing streaming or batch pipelines.
The main problem goodpipe tries to solve is the ordered termination of pipeline parts, so that the output bit doesn't finalize its output when the input closes. In the general directed graph case it would be necessary to define the order of termination.
So, in short, the problems here are to define the scope of problems/digraphs, design a language that allows expressing that (yet be simple enough to get right), and then the implementation is the easy part.
If you want to help improve goodpipe to handle more use cases then I'd be happy to work with you. Or if you want to make something new then I can cheer from the sidelines too. :-)