4 ms·
So basically Apache has two separate projects doing the same thing -- NiFi and Airflow? Or are they significantly different?
by BerislavLopac 8y ago
So basically Apache has two separate projects doing the same thing -- NiFi and Airflow? Or are they significantly different?
- cleansy 8y agoThey are, but the significance depends on your view point. Both provide some sort of "ETL capability". For me the deciding factors were if I wanted to have bulk operations over many records or basically a state machine for individual records. NiFi works great on the latter, as far as my experience goes, Airflow for "get whole data file and process it as a whole and then move to the next processing step". EDIT: plus, of course, things like your tech landscape. NiFi is Java and Airflow Python. If all of your tools are around one or the other, you can get both to work for the same use cases, but as mentioned above you will have more or less implementation work to do.
- marcinzm 8y ago>So basically Apache has two separate projects doing the same thing As someone else said Airflow and NiFi are decently different. However, Apache having multiple projects that overlap is pretty common. Streaming you have Flink, Spark and Storm. Batch processing you have Flink, Spark, Hadoop MapReduce, Hive, and Tez. Serialization you have Avro and Thrift. Columnar storage you have ORC and Parquet. Batch ETL management you have Oozie and Airflow. NoSQL you have Cassandra and HBase. That's just off the top of my head.