An overview of a problem that I run into is the need to pair some input, validate that input, query the system based on the input, expose the intermediate s

An overview of a problem that I run into is the need to pair some input, validate that input, query the system based on the input, expose the intermediate stages for inspection and modification by arbitrary consumers, conditionally modify the underlying system in some way, and provide well defined semantics around the outcome while handling failure recovery, distributing the processing into multiple, cancelable stages, and sharing the system context and other forms of memory with concurrent operations. This sounds like a dream for data analysis.