4 ms·

If you're considering Apache NiFi, you should also look at Apache Airflow and Uber Cadence to decide what model would work best for you.

by yawz 6y ago

ekianjo 6y ago
They do have totally different use cases, so it should be fairly quick to decide which one is for you.
- mac01021 6y ago
  Can you explain what the different use cases are? In particular, what is Airflow good at that Nifi is not? I have used Nifi a little bit and Airflow not at all. Reading the home pages of the two products, it's hard for me to know when it would be more appropriate to use Airflow than Nifi. They both schedule jobs and move data according to control flow topologies that you build in a GUI, right?
  - lsofzz 6y ago
    From watching the presentation on Youtube (https://youtu.be/sQCgtCoZyFQ https://youtu.be/sQCgtCoZyFQ), it seems NiFi is geared towards acquisition of data, gluing/batching/massaging the flow between systems and providing the necessary interfaces to downstream systems; - Would love to see the ability to develop custom NiFi processors in Go/Rust/Elixir etc. - XML is a big pain in the rear. - Being container-aware is big win. Stateless is even better. I see a good opportunity there for users like me to explorer NiFi's capability in the future. > They both schedule jobs and move data according to control flow topologies that you build in a GUI, right? Airflow on the other hand is designed to run scheduled jobs (whether it be batched or otherwise). The 'job' can really be anything - build / data processing pipelines, system configuration management pipelines and so on. In Airflow parlance, one can create connected DAG's as pipelines that massage the data in a way you intend it to. They both share some commonalities but I do gravitate towards their use cases being subtly different and an important one highlighted above.

4 ms·

If you're considering Apache NiFi, you should also look at Apache Airflow and Uber Cadence to decide what model would work best for you.

by yawz 6y ago

If you're considering Apache NiFi, you should also look at Apache Airflow and Uber Cadence to decide what model would work best for you.

ekianjo 6y ago
They do have totally different use cases, so it should be fairly quick to decide which one is for you.
- mac01021 6y ago
  Can you explain what the different use cases are? In particular, what is Airflow good at that Nifi is not? I have used Nifi a little bit and Airflow not at all. Reading the home pages of the two products, it's hard for me to know when it would be more appropriate to use Airflow than Nifi. They both schedule jobs and move data according to control flow topologies that you build in a GUI, right?
  - lsofzz 6y ago
    From watching the presentation on Youtube (https://youtu.be/sQCgtCoZyFQ https://youtu.be/sQCgtCoZyFQ), it seems NiFi is geared towards acquisition of data, gluing/batching/massaging the flow between systems and providing the necessary interfaces to downstream systems; - Would love to see the ability to develop custom NiFi processors in Go/Rust/Elixir etc. - XML is a big pain in the rear. - Being container-aware is big win. Stateless is even better. I see a good opportunity there for users like me to explorer NiFi's capability in the future. > They both schedule jobs and move data according to control flow topologies that you build in a GUI, right? Airflow on the other hand is designed to run scheduled jobs (whether it be batched or otherwise). The 'job' can really be anything - build / data processing pipelines, system configuration management pipelines and so on. In Airflow parlance, one can create connected DAG's as pipelines that massage the data in a way you intend it to. They both share some commonalities but I do gravitate towards their use cases being subtly different and an important one highlighted above.