2 ms·

Is this basically like Apache Spark in its programming model?

by jey 12y ago

espeed 12y ago
Yes, it's like Spark (http://spark.apache.org/ http://spark.apache.org/) and SparkStreaming (http://spark.apache.org/streaming/ http://spark.apache.org/streaming/) combined. Here are the relevant papers... * FlumeJava (iterative, data-parallel pipelines like Spark): http://pages.cs.wisc.edu/~akella/CS838/F12/838-CloudPapers/FlumeJava.pdf http://pages.cs.wisc.edu/~akella/CS838/F12/838-CloudPapers/F... * MillWheel (fault-tolerant stream processing like SparkStreaming): http://research.google.com/pubs/pub41378.html http://research.google.com/pubs/pub41378.html Pointers to the IO blog posts... * "Reimagining developer productivity and data analytics in the cloud" http://googlecloudplatform.blogspot.com/2014/06/reimagining-developer-productivity-and-data-analytics-in-the-cloud-news-from-google-io.html http://googlecloudplatform.blogspot.com/2014/06/reimagining-... * "Sneak peek: Google Cloud Dataflow, a Cloud-native data processing service" http://googlecloudplatform.blogspot.com/2014/06/sneak-peek-google-cloud-dataflow-a-cloud-native-data-processing-service.html http://googlecloudplatform.blogspot.com/2014/06/sneak-peek-g... The Dataflow-specific talks at Google IO 2014... * Big data, the Cloud way: Accelerated and simplified https://www.youtube.com/watch?v=Y0Z58YQSXv0 https://www.youtube.com/watch?v=Y0Z58YQSXv0 * The dawn of "Fast Data" https://www.youtube.com/watch?v=TnLiEWglqHk https://www.youtube.com/watch?v=TnLiEWglqHk * Predicting the future with the Google Cloud Platform https://www.youtube.com/watch?v=YyvvxFeADh8 https://www.youtube.com/watch?v=YyvvxFeADh8 * Keynote (starts at Urs Hölzle's segment on Google Cloud) https://www.youtube.com/watch?v=wtLJPvx7-ys#t=6932 https://www.youtube.com/watch?v=wtLJPvx7-ys#t=6932
- jey 12y ago
  Cool. Does this mean Google is moving away toward languages that allow for easier use and serialization of closures than in C++ and Java? (For example, Spark uses Scala natively.)
  - espeed 12y ago
    Dataflow is language agnostic. The Java API is being released first, and more languages will follow.

Is this basically like Apache Spark in its programming model?

by jey 12y ago

Is this basically like Apache Spark in its programming model?

Yes, it's like Spark (http://spark.apache.org/ http://spark.apache.org/) and SparkStreaming (http://spark.apache.org/streaming/ http://spark.apache.org/streaming/) combined. Here are the relevant papers... * FlumeJava (iterative, data-parallel pipelines like Spark): http://pages.cs.wisc.edu/~akella/CS838/F12/838-CloudPapers/FlumeJava.pdf http://pages.cs.wisc.edu/~akella/CS838/F12/838-CloudPapers/F... * MillWheel (fault-tolerant stream processing like SparkStreaming): http://research.google.com/pubs/pub41378.html http://research.google.com/pubs/pub41378.html Pointers to the IO blog posts... * "Reimagining developer productivity and data analytics in the cloud" http://googlecloudplatform.blogspot.com/2014/06/reimagining-developer-productivity-and-data-analytics-in-the-cloud-news-from-google-io.html http://googlecloudplatform.blogspot.com/2014/06/reimagining-... * "Sneak peek: Google Cloud Dataflow, a Cloud-native data processing service" http://googlecloudplatform.blogspot.com/2014/06/sneak-peek-google-cloud-dataflow-a-cloud-native-data-processing-service.html http://googlecloudplatform.blogspot.com/2014/06/sneak-peek-g... The Dataflow-specific talks at Google IO 2014... * Big data, the Cloud way: Accelerated and simplified https://www.youtube.com/watch?v=Y0Z58YQSXv0 https://www.youtube.com/watch?v=Y0Z58YQSXv0 * The dawn of "Fast Data" https://www.youtube.com/watch?v=TnLiEWglqHk https://www.youtube.com/watch?v=TnLiEWglqHk * Predicting the future with the Google Cloud Platform https://www.youtube.com/watch?v=YyvvxFeADh8 https://www.youtube.com/watch?v=YyvvxFeADh8 * Keynote (starts at Urs Hölzle's segment on Google Cloud) https://www.youtube.com/watch?v=wtLJPvx7-ys#t=6932 https://www.youtube.com/watch?v=wtLJPvx7-ys#t=6932

jey 12y ago
Cool. Does this mean Google is moving away toward languages that allow for easier use and serialization of closures than in C++ and Java? (For example, Spark uses Scala natively.)
- espeed 12y ago
  Dataflow is language agnostic. The Java API is being released first, and more languages will follow.