Fantastic! It would be interesting to combine pipelines into something that branches off into various categories. For e.g., you could split the lines of acces

Fantastic! It would be interesting to combine pipelines into something that branches off into various categories. For e.g., you could split the lines of access-log file into IP addresses and size of requests and fork off separate processing threads: one for obtaining unique set of IP addresses and another for summing up. It seems like all that is needed for describing a pipeline is: (a) a queue for input (b) a processing program that's connected to the queue (c) (possibly multiple) queues for output (d) a topology connecting the processing programs (e) a job scheduler. At the face of it, it looks similar to Apple's Automator and Matt Welsh's PhD thesis on SEDA: * http://www.eecs.harvard.edu/~mdw/proj/seda/ * Paper: http://www.eecs.harvard.edu/~mdw/papers/seda-sosp01.pdf EDIT: Formatting