NiFi is also very useful for visually chaining together scripts (e.g. foo | bar | baz). In NiFi you can use ExecuteProcess and ExecuteStreamCommand to basically

NiFi is also very useful for visually chaining together scripts (e.g. foo | bar | baz). In NiFi you can use ExecuteProcess and ExecuteStreamCommand to basically do what pipes (|) does. Further, the intermediate data between each process is backed by a WAL. You can also mix and match your pipeline, for example you can mix-in some of your custom scripts, and periodically use built-in processors that are provided to do additional stuff (e.g. PutKafka).