3 ms·

Recently I started to use Apache NiFi[1] for everything that does not required too complex operations. It's pretty much what this framework does, just with an U

Recently I started to use Apache NiFi[1] for everything that does not required too complex operations. It's pretty much what this framework does, just with an UI and a lot of monitoring features. However, one downside is the massive RAM consumption. 1GB of RAM even if it does pretty much nothing is quite a bill to start off with. 1: https://nifi.apache.org/ https://nifi.apache.org/

ekianjo 8y ago
Yes, but Nifi is pretty good at what it does. 1GB RAM cost is nothing compared to the time it saves you in the end to make very robust data flows.
- cleansy 8y ago
  Of course 1GB RAM is nothing nowadays, but this was something that I wanted to point out since sometimes you have constrained resources available or need to calculate what machine size you want to use. NiFi might not run on an AWS t2.micro instance. Whereas Apache Airflow does.
  - ekianjo 8y ago
    When you use Nifi usually you are not in a budget constrained environment. Because CPU wise it takes its toll too and you need more than one core to be comfy, so micro instances are out anyway.
- srean 8y ago
  I would rather use the 1 GB closer to the problem I want to solve.
BerislavLopac 8y ago
So basically Apache has two separate projects doing the same thing -- NiFi and Airflow? Or are they significantly different?
- cleansy 8y ago
  They are, but the significance depends on your view point. Both provide some sort of "ETL capability". For me the deciding factors were if I wanted to have bulk operations over many records or basically a state machine for individual records. NiFi works great on the latter, as far as my experience goes, Airflow for "get whole data file and process it as a whole and then move to the next processing step". EDIT: plus, of course, things like your tech landscape. NiFi is Java and Airflow Python. If all of your tools are around one or the other, you can get both to work for the same use cases, but as mentioned above you will have more or less implementation work to do.
- marcinzm 8y ago
  >So basically Apache has two separate projects doing the same thing As someone else said Airflow and NiFi are decently different. However, Apache having multiple projects that overlap is pretty common. Streaming you have Flink, Spark and Storm. Batch processing you have Flink, Spark, Hadoop MapReduce, Hive, and Tez. Serialization you have Avro and Thrift. Columnar storage you have ORC and Parquet. Batch ETL management you have Oozie and Airflow. NoSQL you have Cassandra and HBase. That's just off the top of my head.

ekianjo 8y ago

Yes, but Nifi is pretty good at what it does. 1GB RAM cost is nothing compared to the time it saves you in the end to make very robust data flows.

cleansy 8y ago
Of course 1GB RAM is nothing nowadays, but this was something that I wanted to point out since sometimes you have constrained resources available or need to calculate what machine size you want to use. NiFi might not run on an AWS t2.micro instance. Whereas Apache Airflow does.
- ekianjo 8y ago
  When you use Nifi usually you are not in a budget constrained environment. Because CPU wise it takes its toll too and you need more than one core to be comfy, so micro instances are out anyway.
srean 8y ago
I would rather use the 1 GB closer to the problem I want to solve.

BerislavLopac 8y ago

So basically Apache has two separate projects doing the same thing -- NiFi and Airflow? Or are they significantly different?

cleansy 8y ago
They are, but the significance depends on your view point. Both provide some sort of "ETL capability". For me the deciding factors were if I wanted to have bulk operations over many records or basically a state machine for individual records. NiFi works great on the latter, as far as my experience goes, Airflow for "get whole data file and process it as a whole and then move to the next processing step". EDIT: plus, of course, things like your tech landscape. NiFi is Java and Airflow Python. If all of your tools are around one or the other, you can get both to work for the same use cases, but as mentioned above you will have more or less implementation work to do.
marcinzm 8y ago
>So basically Apache has two separate projects doing the same thing As someone else said Airflow and NiFi are decently different. However, Apache having multiple projects that overlap is pretty common. Streaming you have Flink, Spark and Storm. Batch processing you have Flink, Spark, Hadoop MapReduce, Hive, and Tez. Serialization you have Avro and Thrift. Columnar storage you have ORC and Parquet. Batch ETL management you have Oozie and Airflow. NoSQL you have Cassandra and HBase. That's just off the top of my head.