3 ms·
DataFlow programming[1] (which NiFi) is created to do is NOT the same as your typical extraction/transform/load (ETL) tool with a nice user interface! Wikipedi
by nl 11y ago
DataFlow programming[1] (which NiFi) is created to do is NOT the same as your typical extraction/transform/load (ETL) tool with a nice user interface!
Wikipedia says Dataflow programming languages share some features of functional languages, and were generally developed in order to bring some functional concepts to a language more suitable for numeric processing.
The thing closest to Dataflow that is most commonly used is the concept of DAG operations in Spark, but Dataflow usually makes time windowing a first level concept. Spark Streaming is moving towards this type thing.
It is true that there is overlap with ETL tools, but that undersells what Dataflow is.
[1] https://en.wikipedia.org/wiki/Dataflow_programming https://en.wikipedia.org/wiki/Dataflow_programming
- abrookewood 11y agoCan you expand on how they are different? The first thing I thought of after reading about this was that it was primarily an ETL engine.
- nl 11y agoSo ETL is mostly about getting data into some kind of processing system. That means there are lots of functions for dealing with things like csv data and polling directories and transforming json data into flat structures and... etc. Dataflow is a programming model for performing actions on data. A system that implements dataflow programming will probably have functions to load external data into the structure needed for the system, but it isn't primarily about moving data to another system. For example, Google Dataflow[1] has functions for reading files etc, but there aren't really the huge number of things for cleaning and processing data that a real ETL system has. Instead, you load the data into the system, and then process it for a specific task. [1] https://cloud.google.com/dataflow/what-is-google-cloud-dataflow#Sdks https://cloud.google.com/dataflow/what-is-google-cloud-dataf...