4 ms·

Is there something similar for Apache projects?

by syats 6y ago

adwww 6y ago
Ha good luck explaining in one sentance the difference between Flume, Spark, Storm, NiFi, Camel, Apex, Flink, Beam.....
- faizshah 6y ago
  I’ll try. Heres what u should know from these: Fast, Reliable, Stream Processing - Flink Data Science on Big Data - Spark Reliable Distributed Log Aggregation - Flume Low Code Data Flows w/ GUI - NiFi Zapier for Enterprise Software - Camel A Single API for Batch and Stream Jobs. Execute on Spark, Flink, managed services etc. - Beam Never heard of anyone use Apex over Flink. Storm community has branched off into Flink, Spark Streaming, Heron and Cloud Dataflow. Explaining Spark vs Flink is quite hard tho.
  - yazaddaruvala 6y ago
    From the little I know, Spark was built as a Batch job solution. Flink was built as a Streaming solution. Since Spark needed to adapt, they leveraged micro-batching to operate as a streaming solution. They are very similar today. Maybe some remnants of the original design make Spark "slightly less appropriate" for pure streaming usecases, or possibly less able to iterate on future features/optimizations, but so-far that line of reasoning has been only speculation.
    - faizshah 6y ago
      So Flink differentiates in stream processing in two major ways: - Flink guarantees exactly once stream processing through a barrier checkpointing system. - Flink has very detailed APIs for handling state and doing stateful stream processing. Additionally, Flink has one of the most active communities in the Apache Software Foundation: https://blogs.apache.org/foundation/entry/the-apache-software-foundation-announces55 https://blogs.apache.org/foundation/entry/the-apache-softwar... In situations where you want to create a streaming application with persistent state you might choose Flink over spark streaming.
  - adwww 6y ago
    Wow that's actually a pretty good effort. We use Camel, Spark and Beam to do much the same thing at my work, so despite using them all semi regularly, I've never known why you'd pick one over the other.
gunnarmorling 6y ago
My awesome open-source data engineering list [1] covers a few Apach projects. Contributions welcome! [1] https://github.com/gunnarmorling/awesome-opensource-data-engineering https://github.com/gunnarmorling/awesome-opensource-data-eng...

Is there something similar for Apache projects?

by syats 6y ago

Is there something similar for Apache projects?

adwww 6y ago

Ha good luck explaining in one sentance the difference between Flume, Spark, Storm, NiFi, Camel, Apex, Flink, Beam.....

faizshah 6y ago
I’ll try. Heres what u should know from these: Fast, Reliable, Stream Processing - Flink Data Science on Big Data - Spark Reliable Distributed Log Aggregation - Flume Low Code Data Flows w/ GUI - NiFi Zapier for Enterprise Software - Camel A Single API for Batch and Stream Jobs. Execute on Spark, Flink, managed services etc. - Beam Never heard of anyone use Apex over Flink. Storm community has branched off into Flink, Spark Streaming, Heron and Cloud Dataflow. Explaining Spark vs Flink is quite hard tho.
- yazaddaruvala 6y ago
  From the little I know, Spark was built as a Batch job solution. Flink was built as a Streaming solution. Since Spark needed to adapt, they leveraged micro-batching to operate as a streaming solution. They are very similar today. Maybe some remnants of the original design make Spark "slightly less appropriate" for pure streaming usecases, or possibly less able to iterate on future features/optimizations, but so-far that line of reasoning has been only speculation.
  - faizshah 6y ago
    So Flink differentiates in stream processing in two major ways: - Flink guarantees exactly once stream processing through a barrier checkpointing system. - Flink has very detailed APIs for handling state and doing stateful stream processing. Additionally, Flink has one of the most active communities in the Apache Software Foundation: https://blogs.apache.org/foundation/entry/the-apache-software-foundation-announces55 https://blogs.apache.org/foundation/entry/the-apache-softwar... In situations where you want to create a streaming application with persistent state you might choose Flink over spark streaming.
- adwww 6y ago
  Wow that's actually a pretty good effort. We use Camel, Spark and Beam to do much the same thing at my work, so despite using them all semi regularly, I've never known why you'd pick one over the other.

gunnarmorling 6y ago

My awesome open-source data engineering list [1] covers a few Apach projects. Contributions welcome! [1] https://github.com/gunnarmorling/awesome-opensource-data-engineering https://github.com/gunnarmorling/awesome-opensource-data-eng...