3 ms·
This is a great technique; thanks for sharing it. While I knew about all the individual parts that you used, I don't think I ever thought of combining all the p
by vram22 8y ago
This is a great technique; thanks for sharing it. While I knew about all the individual parts that you used, I don't think I ever thought of combining all the parts with the "view -" at end, although I knew that "-" as a command-line arg means standard input (to many commands that support it).
Very useful stuff, and as you say, the possibilities are endless.
I had done something sort of analogous, an experimental Python tool called pipe_controller, which is not about piping commands to each other in the traditional Unix sense; rather it is about "piping" the output of one function to a second one, and the output of the second to a third one, and so on, as many as one needs, under the control of a for loop. The net effect is like normal composition of function calls, like f(g(h(x))), but some other interesting effects can be achieved by doing it with a for loop, and changing some things at run time:
After first creating pipe_controller (which is simple, really), I played around with using it in a few different ways, and found that it can be used for at least a couple of interesting things:
- running a "pipe" (of those functions) incrementally (something like the technique you showed), and saving / viewing the output of each intermediate stage;
- swapping components of the "pipe" at run time, under program control, which again can lead to some interesting use cases.
I blogged a small series of posts about pipe_controller and such uses of it. Here are the two last or so posts, and the previous posts can be reached by following links in those posts:
Swapping pipe components at runtime with pipe_controller:
https://jugad2.blogspot.com/2012/10/swapping-pipe-components-at-runtime.html https://jugad2.blogspot.com/2012/10/swapping-pipe-components...
Using PipeController to run a pipe incrementally:
https://jugad2.blogspot.com/2012/09/using-pipecontroller-to-run-pipe.html https://jugad2.blogspot.com/2012/09/using-pipecontroller-to-...
The pipe_controller code is here:
https://bitbucket.org/vasudevram/pipe_controller https://bitbucket.org/vasudevram/pipe_controller
- vram22 8y agoAnother related couple of posts are about creating Unix pipelines under the control of a Python program: Generate PDF from a Python-controlled Unix pipeline: https://jugad2.blogspot.com/2016/01/generate-pdf-from-python-controlled.html https://jugad2.blogspot.com/2016/01/generate-pdf-from-python... Code for recent post about PDF from a Python pipeline: https://jugad2.blogspot.com/2016/01/code-for-recent-post-about-pdf-from.html https://jugad2.blogspot.com/2016/01/code-for-recent-post-abo...
- fizixer 8y agoInteresting. Looks like you're trying to do shell piping in python but without the annoying nesting. I have thought of this myself too, both in terms of python and in terms of scheme. For example, for your python function composition example f(g(h(x))), shell piping would look like this: cat x | h | g | f | view - while in scheme it would be like python, just a little different: (f (g (h x))) Essentially the point is, by using existing syntax and language facilities, can we mimic the seamless shell piping workflow (e.g., with no annoying nesting). I think one issue that python or scheme will have is that, when something goes wrong, debugging the problem would still be very annoying in python/scheme, whereas in shell piping, you simply remove some tail-end processing commands to view and earlier output, then fix your issue, then reintroduce the tail-end commands that you removed, possibility with some modifications. So "view -" acts as a debugger, not just for visual inspection of your report. Anyway this is an interesting area. Keep up the good work. I would just like to mention that this is related, in fact it is, part of a much larger paradigm of programming called dataflow programming, sometimes called stream processing. And also has connection with reactive programming. (which if you lookup in wikipedia [1] falls into a larger paradigm called declarative programming). [1] https://en.wikipedia.org/wiki/Programming_paradigm https://en.wikipedia.org/wiki/Programming_paradigm
- vram22 8y agoYes, good points about the debugging issues. Thanks for the encouragement and links. Will check them out.