5 ms·
Seems like this would duplicate a rather large chunk of Apache Crunch, which implements Google Flume nearly exactly as far as public API is concerned. As far as
by melted 11y ago
Seems like this would duplicate a rather large chunk of Apache Crunch, which implements Google Flume nearly exactly as far as public API is concerned. As far as I can tell, Google Dataflow is also a variation on top of Google Flume. It would be helpful if they could elucidate why this project would not be redundant under the Apache umbrella.
- arkadiyt 11y agoThis sentence captures everything that I hate about the Apache projects and the Hadoop ecosystem in particular.
- asdfologist 11y agoYes, competition is bad and people should never try to create better ways of doing the same thing.
- Eduard 11y agoWhat kind of problems do you see with redundancy?
- melted 11y agoBasically it looks like it's nearly the same API with minor variations.
- chubot 11y agoI was skeptical too, until I read the paper. The motivation is actually explained really well. It's different than Flume.
- throwaway6497 11y agoAre you referring to the DataFlow paper at http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf
- chubot 11y agoYes, I like how the paper is written, and the work is solid IMO, having used most of the predecessors. The annoying thing is footnote 1, where they explain that they coopted the term "dataflow model". That's just plain dumb, since that word is used in so many other settings, especially big data settings (TensorFlow is also a dataflow system, but both the data types and model are completely different, etc.). So I can see why someone would just dismiss it as yet another framework, sort of like how all these JavaScript frameworks that are the hot thing one year and then legacy code the next. If they had used a more specific name, I think it would help the perception a lot. The main point is batch vs streaming, which is still big pain point, and being precise about the event time vs event processing time.
- nl 11y agoApache explicitly allows multiple overlapping projects. See, for example Apache Storm/Spark/Flink. Also worth noting is that the implementation matters at least as much as the API. In this case there are substantial differences: Data flow is a DSL, not a Java API, and it is designed for streaming data. It is unclear if Crunch handles the streaming case, but it talks a lot about Map/Reduce which makes me think it isn't the primary usecase.
- mey 11y agoApache Ant + Ivy vs Apache Maven vs Apache Buildr Apache Tapestry vs Apache Wicket vs Apache MyFaces those jump to mind, but I'm sure there are plenty of other overlaps in the Apache sphere. They certainly embrace it.
- rryan 11y agoIt's quite different from the original Flume paper as Dataflow supports expressing streaming computations in addition to batch in a unified set of APIs. It's like a combination of Flume and Millwheel.
- squarecog 11y agoNote how the creator of Apache Crunch is on the Dataflow committer list.