10 ms·

Google proposes its Dataflow batch/stream tech to the Apache Incubator

191 points by crb 11y ago

obulpathi 11y ago
It would be awesome to have the code portable across various big data engines.
fhoffa 11y ago
Note that this proposal is being back not only by Google, but also Cloudera, Data Artisans, Talend, Cask, PayPal, ... Some other posts on the announcement: http://googlecloudplatform.blogspot.com/2016/01/Dataflow-and-open-source-proposal-to-join-the-Apache-Incubator.html http://googlecloudplatform.blogspot.com/2016/01/Dataflow-and... http://blog.cloudera.com/blog/2016/01/spark-dataflow-joins-googles-dataflow-sdk/ http://blog.cloudera.com/blog/2016/01/spark-dataflow-joins-g... http://data-artisans.com/dataflow-proposed-as-apache-incubator-project/ http://data-artisans.com/dataflow-proposed-as-apache-incubat... http://blog.cask.co/2016/01/cask-anticipates-googles-dataflow-to-flourish-in-apache/ http://blog.cask.co/2016/01/cask-anticipates-googles-dataflo...
- ericand 11y ago
  Also Talend blog: https://www.talend.com/blog/2016/01/20/talend-joins-google-to-propose-dataflow-as-an-asf-incubator-project https://www.talend.com/blog/2016/01/20/talend-joins-google-t...
melted 11y ago
Seems like this would duplicate a rather large chunk of Apache Crunch, which implements Google Flume nearly exactly as far as public API is concerned. As far as I can tell, Google Dataflow is also a variation on top of Google Flume. It would be helpful if they could elucidate why this project would not be redundant under the Apache umbrella.
- arkadiyt 11y ago
  This sentence captures everything that I hate about the Apache projects and the Hadoop ecosystem in particular.
  - asdfologist 11y ago
    Yes, competition is bad and people should never try to create better ways of doing the same thing.
- Eduard 11y ago
  What kind of problems do you see with redundancy?
  - melted 11y ago
    Basically it looks like it's nearly the same API with minor variations.
    - chubot 11y ago
      I was skeptical too, until I read the paper. The motivation is actually explained really well. It's different than Flume.
      - throwaway6497 11y ago
        Are you referring to the DataFlow paper at http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf
        chubot 11y ago
        Yes, I like how the paper is written, and the work is solid IMO, having used most of the predecessors. The annoying thing is footnote 1, where they explain that they coopted the term "dataflow model". That's just plain dumb, since that word is used in so many other settings, especially big data settings (TensorFlow is also a dataflow system, but both the data types and model are completely different, etc.). So I can see why someone would just dismiss it as yet another framework, sort of like how all these JavaScript frameworks that are the hot thing one year and then legacy code the next. If they had used a more specific name, I think it would help the perception a lot. The main point is batch vs streaming, which is still big pain point, and being precise about the event time vs event processing time.
- nl 11y ago
  Apache explicitly allows multiple overlapping projects. See, for example Apache Storm/Spark/Flink. Also worth noting is that the implementation matters at least as much as the API. In this case there are substantial differences: Data flow is a DSL, not a Java API, and it is designed for streaming data. It is unclear if Crunch handles the streaming case, but it talks a lot about Map/Reduce which makes me think it isn't the primary usecase.
  - mey 11y ago
    Apache Ant + Ivy vs Apache Maven vs Apache Buildr Apache Tapestry vs Apache Wicket vs Apache MyFaces those jump to mind, but I'm sure there are plenty of other overlaps in the Apache sphere. They certainly embrace it.
- rryan 11y ago
  It's quite different from the original Flume paper as Dataflow supports expressing streaming computations in addition to batch in a unified set of APIs. It's like a combination of Flume and Millwheel.
- squarecog 11y ago
  Note how the creator of Apache Crunch is on the Dataflow committer list.
ericand 11y ago
O'Reilly post also released today references the Apache Dataflow submission: https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102 https://www.oreilly.com/ideas/the-world-beyond-batch-streami...
- spenrose 11y ago
  The author is one of the DataFlow committers.
mindprince 11y ago
> While Google has previously published papers describing some of its technologies, Google decided to take a different approach with Dataflow. Google open-sourced the SDK and model alongside commercialization of the idea and ahead of publishing papers on the topic. A large number of ASF projects in the Big Data space are inspired by Google's publications. Good to see Google finally taking the lead and coming out with code.
xcelq 11y ago
Can we hope to see a google like search engine open source? I'm just waiting for this day to happen.
Wonnk13 11y ago
what are the best resources to learn about streaming, dataflow, etc? Not necessarily the Google implementations, but the core concepts backing them.
- jcbeard 11y ago
  Wrote this awhile back on streaming and dataflow: http://www.jonathanbeard.io/blog/2015/09/19/streaming-and-dataflow.html http://www.jonathanbeard.io/blog/2015/09/19/streaming-and-da... There are literally hundreds of data-flow streaming implementations. Quite sad that Google has named their system "Dataflow" which literally search bombs any other dataflow like system out there. Many of these systems are more data-flow in semantics only but more transaction-like under the hood. Quite nice for simple check-pointing/fault tolerance but not so great for performance. There's also some interesting work form Dongarra's lab on data-flow task sets (very OmpSs like). The earliest system was by Jack Dennis. Either way, I hope my blog post helps answer your questions.
  - Wonnk13 11y ago
    very cool, thanks much!
- harlanji 11y ago
  Some of the best material I've read recently comes from the Confluent blug, esp. Martin Klepmann. The views are tilted toward Kafka and Samza since the founders are the same people, though they are both Apache projects. The article that blew my mind was "Turning the Database Inside Out": https://martin.kleppmann.com/2015/03/04/turning-the-database-inside-out.html https://martin.kleppmann.com/2015/03/04/turning-the-database... . Doesn't encompass the full space, but the architectural implications when combined with CQRS/Event Sourcing models are huge. Samza's architecture and API embodies a lot of the important ideas at a lower level than Storm; while it may not be the easiest to use in practice, the concepts and documentation translate.
  - Wonnk13 11y ago
    cool, thank you! Def need to pickup a copy of Klepman's book
sysk 11y ago
Can anyone ELI5 what it means for an open source project to become an Apache project? Why doesn't Google just push the code on Github?
- zitterbewegung 11y ago
  They have pushed it to GitHub I think they are giving it to apache to be more vendor neutral.
- oh_sigh 11y ago
  It's about who maintains it.
  - sysk 11y ago
    Does that mean that Google will stop maintaining the project?
    - wener 11y ago
      A lot of Apache project was submitted by venders and maintained by themself,goto ASF just say anyone can involve and do things with it.
    - chimerasaurus 11y ago
      There are several organizations included on the proposal, including Google, who will still be actively involved in the project, if accepted.
- nl 11y ago
  It is about control. Donating something to the ASF (Apache Software Foundation) means you lose control and ownership of the copyright.
- chimerasaurus 11y ago
  Going to start backwards on this one. :) The ASF proposal contains a few different components (SDK and runners) all of which have lived on GitHub for awhile (the proposal has links if you're interested.) If accepted as an incubating project, the code would still live on GitHub like other Apache Incubator projects. tl;dr/ELI5 The goal is to grow an open community, have consistent (and open) processes, and be part of a larger OSS ecosystem. When projects live on GitHub, the code is open but project processes, etc. may not be.
BenoitP 11y ago
Where does Dataflow stands? Is it only a wrapper, trying to define a standard API for combining stream producers, datastores, and stream engines?
- vgt 11y ago
  It's a Batch+Stream unified processing model, an SDK. Idea is you can code up your pipeline in Dataflow and have your choice of where to run it - Spark, Flink, etc. Google Cloud Dataflow is a fully-managed service that executes Dataflow pipelines and has nice value adds on top like fault tolerance and auto-optimization. (Disclosure: I work on BigQuery, not Dataflow)

191 points by crb 11y ago

It would be awesome to have the code portable across various big data engines.

Note that this proposal is being back not only by Google, but also Cloudera, Data Artisans, Talend, Cask, PayPal, ... Some other posts on the announcement: http://googlecloudplatform.blogspot.com/2016/01/Dataflow-and-open-source-proposal-to-join-the-Apache-Incubator.html http://googlecloudplatform.blogspot.com/2016/01/Dataflow-and... http://blog.cloudera.com/blog/2016/01/spark-dataflow-joins-googles-dataflow-sdk/ http://blog.cloudera.com/blog/2016/01/spark-dataflow-joins-g... http://data-artisans.com/dataflow-proposed-as-apache-incubator-project/ http://data-artisans.com/dataflow-proposed-as-apache-incubat... http://blog.cask.co/2016/01/cask-anticipates-googles-dataflow-to-flourish-in-apache/ http://blog.cask.co/2016/01/cask-anticipates-googles-dataflo...

ericand 11y ago
Also Talend blog: https://www.talend.com/blog/2016/01/20/talend-joins-google-to-propose-dataflow-as-an-asf-incubator-project https://www.talend.com/blog/2016/01/20/talend-joins-google-t...

melted 11y ago

Seems like this would duplicate a rather large chunk of Apache Crunch, which implements Google Flume nearly exactly as far as public API is concerned. As far as I can tell, Google Dataflow is also a variation on top of Google Flume. It would be helpful if they could elucidate why this project would not be redundant under the Apache umbrella.

arkadiyt 11y ago
This sentence captures everything that I hate about the Apache projects and the Hadoop ecosystem in particular.
- asdfologist 11y ago
  Yes, competition is bad and people should never try to create better ways of doing the same thing.
Eduard 11y ago
What kind of problems do you see with redundancy?
- melted 11y ago
  Basically it looks like it's nearly the same API with minor variations.
  - chubot 11y ago
    I was skeptical too, until I read the paper. The motivation is actually explained really well. It's different than Flume.
    - throwaway6497 11y ago
      Are you referring to the DataFlow paper at http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf
      - chubot 11y ago
        Yes, I like how the paper is written, and the work is solid IMO, having used most of the predecessors. The annoying thing is footnote 1, where they explain that they coopted the term "dataflow model". That's just plain dumb, since that word is used in so many other settings, especially big data settings (TensorFlow is also a dataflow system, but both the data types and model are completely different, etc.). So I can see why someone would just dismiss it as yet another framework, sort of like how all these JavaScript frameworks that are the hot thing one year and then legacy code the next. If they had used a more specific name, I think it would help the perception a lot. The main point is batch vs streaming, which is still big pain point, and being precise about the event time vs event processing time.
nl 11y ago
Apache explicitly allows multiple overlapping projects. See, for example Apache Storm/Spark/Flink. Also worth noting is that the implementation matters at least as much as the API. In this case there are substantial differences: Data flow is a DSL, not a Java API, and it is designed for streaming data. It is unclear if Crunch handles the streaming case, but it talks a lot about Map/Reduce which makes me think it isn't the primary usecase.
- mey 11y ago
  Apache Ant + Ivy vs Apache Maven vs Apache Buildr Apache Tapestry vs Apache Wicket vs Apache MyFaces those jump to mind, but I'm sure there are plenty of other overlaps in the Apache sphere. They certainly embrace it.
rryan 11y ago
It's quite different from the original Flume paper as Dataflow supports expressing streaming computations in addition to batch in a unified set of APIs. It's like a combination of Flume and Millwheel.
squarecog 11y ago
Note how the creator of Apache Crunch is on the Dataflow committer list.

ericand 11y ago

O'Reilly post also released today references the Apache Dataflow submission: https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102 https://www.oreilly.com/ideas/the-world-beyond-batch-streami...

spenrose 11y ago
The author is one of the DataFlow committers.

mindprince 11y ago

> While Google has previously published papers describing some of its technologies, Google decided to take a different approach with Dataflow. Google open-sourced the SDK and model alongside commercialization of the idea and ahead of publishing papers on the topic. A large number of ASF projects in the Big Data space are inspired by Google's publications. Good to see Google finally taking the lead and coming out with code.

xcelq 11y ago

Can we hope to see a google like search engine open source? I'm just waiting for this day to happen.

Wonnk13 11y ago

what are the best resources to learn about streaming, dataflow, etc? Not necessarily the Google implementations, but the core concepts backing them.

jcbeard 11y ago
Wrote this awhile back on streaming and dataflow: http://www.jonathanbeard.io/blog/2015/09/19/streaming-and-dataflow.html http://www.jonathanbeard.io/blog/2015/09/19/streaming-and-da... There are literally hundreds of data-flow streaming implementations. Quite sad that Google has named their system "Dataflow" which literally search bombs any other dataflow like system out there. Many of these systems are more data-flow in semantics only but more transaction-like under the hood. Quite nice for simple check-pointing/fault tolerance but not so great for performance. There's also some interesting work form Dongarra's lab on data-flow task sets (very OmpSs like). The earliest system was by Jack Dennis. Either way, I hope my blog post helps answer your questions.
- Wonnk13 11y ago
  very cool, thanks much!
harlanji 11y ago
Some of the best material I've read recently comes from the Confluent blug, esp. Martin Klepmann. The views are tilted toward Kafka and Samza since the founders are the same people, though they are both Apache projects. The article that blew my mind was "Turning the Database Inside Out": https://martin.kleppmann.com/2015/03/04/turning-the-database-inside-out.html https://martin.kleppmann.com/2015/03/04/turning-the-database... . Doesn't encompass the full space, but the architectural implications when combined with CQRS/Event Sourcing models are huge. Samza's architecture and API embodies a lot of the important ideas at a lower level than Storm; while it may not be the easiest to use in practice, the concepts and documentation translate.
- Wonnk13 11y ago
  cool, thank you! Def need to pickup a copy of Klepman's book

sysk 11y ago

Can anyone ELI5 what it means for an open source project to become an Apache project? Why doesn't Google just push the code on Github?

zitterbewegung 11y ago
They have pushed it to GitHub I think they are giving it to apache to be more vendor neutral.
oh_sigh 11y ago
It's about who maintains it.
- sysk 11y ago
  Does that mean that Google will stop maintaining the project?
  - wener 11y ago
    A lot of Apache project was submitted by venders and maintained by themself,goto ASF just say anyone can involve and do things with it.
  - chimerasaurus 11y ago
    There are several organizations included on the proposal, including Google, who will still be actively involved in the project, if accepted.
nl 11y ago
It is about control. Donating something to the ASF (Apache Software Foundation) means you lose control and ownership of the copyright.
chimerasaurus 11y ago
Going to start backwards on this one. :) The ASF proposal contains a few different components (SDK and runners) all of which have lived on GitHub for awhile (the proposal has links if you're interested.) If accepted as an incubating project, the code would still live on GitHub like other Apache Incubator projects. tl;dr/ELI5 The goal is to grow an open community, have consistent (and open) processes, and be part of a larger OSS ecosystem. When projects live on GitHub, the code is open but project processes, etc. may not be.

BenoitP 11y ago

Where does Dataflow stands? Is it only a wrapper, trying to define a standard API for combining stream producers, datastores, and stream engines?

vgt 11y ago
It's a Batch+Stream unified processing model, an SDK. Idea is you can code up your pipeline in Dataflow and have your choice of where to run it - Spark, Flink, etc. Google Cloud Dataflow is a fully-managed service that executes Dataflow pipelines and has nice value adds on top like fault tolerance and auto-optimization. (Disclosure: I work on BigQuery, not Dataflow)