4 ms·

Google Dumps MapReduce in Favor of New Hyper-Scale Analytics System

41 points by posharma 12y ago

wyager 12y ago
>Google Replaces MapReduce With New Hyper-Scale Cloud Analytics System Buzzword... overload!
- msane 12y ago
  Hey sometimes you just have to do whatever it takes to synergize global channels on virtual platforms. You know, really aggregate extensible markets with repurposed leading-edge metrics.
  - deleted 12y ago
    [deleted]
  - dclusin 12y ago
    But how will these leading-edge metrics enable us to deliver paradigm shifting solutions to our customers while simultaneously reducing costs and increasing operational efficiency?
    - source99 12y ago
      While making the world a better place!
    - msane 12y ago
      That's the best part - it's automated with a context-sensitive customizability infrastructure. That means you can expand any targeted benchmark in a demand-driven mesh.
CHY872 12y ago
Sounds a bit like Dryad?
_dark_matter_ 12y ago
>“We don’t really use MapReduce anymore” This is not even true. They have recently published research that involved using MapReduce in their own systems. Example: http://research.google.com/pubs/pub41376.html http://research.google.com/pubs/pub41376.html
- kiyoto 12y ago
  As someone who has one foot in marketing and one foot in development (I am a developer evangelist), here is what's happening. 1. Google's infrastructure is evolving atop MapReduce (FlumeJava/MillWheel). 2. Google's PR decided to call it "not using MapReduce anymore" because in marketing, "beyond <current fad>" sounds really cool. 3. The rest of the world's PR/press/marketing fall for Google's clever PR. Either way, it is great to see Google making its core technology accessible as part of their PaaS =)
  - t1m 12y ago
    It would be more great if they open sourced it.
  - zobzu 12y ago
    hyperscale man! cyberleetscale would have been better. IMO. Thanks for the clarifications :)
  - username223 12y ago
    > 3. The rest of the world's PR/press/marketing fall for Google's clever PR. If "hyper-scale Analytics" counts as "clever PR," then eating paste counts as "clever food choice." It's not worth my time to decode the BS.
  - rsync 12y ago
    I know what I'm supposed to do when my taxi driver starts recommending stocks to me ... But what do I do when people start describing themselves as "developer evangelists" ? That has to be some kind of sell signal, right ?
    - deleted 12y ago
basyt 12y ago
Its fundamentally the same thing as MapReduce isn't it? Can someone explain the differences to me please? There isn't much of use in the article
- njharman 12y ago
  This has "cloud" prefixed to name of every component. So, obviously, is better. Also, they're selling it. So, ya know, marketing trumps engineering.
- dyoo1979 12y ago
  You'll probably want to read the FlumeJava paper. http://pages.cs.wisc.edu/~akella/CS838/F12/838-CloudPapers/FlumeJava.pdf http://pages.cs.wisc.edu/~akella/CS838/F12/838-CloudPapers/F... Citation: http://dl.acm.org/citation.cfm?id=1806638 http://dl.acm.org/citation.cfm?id=1806638 The key word is pipeline. If you have some analysis that runs in several stages, you'll be taking the output of one stage, and connecting it to the next. If you want to compose multiple phases, chained together, raw MapReduce isn't going to help you very much with the chaining. What's described in the paper is a way to do the chaining in a nice way. The system will take care of writing the raw MapReduces for you. But it'll also do a lot of work on the interconnections between your stages as well.
- espeed 12y ago
  MapReduce wasn't designed for iterative algorithms or streaming data, whereas Google Dataflow and Spark (http://spark.apache.org/ http://spark.apache.org/) make iterative algoritms easy. It's a much simpler programming paradigm, and it allows you to do iterative graph-processing and machine-learning algos (http://spark.apache.org/mllib/ http://spark.apache.org/mllib/) that are impractical on MapReduce. For example, Spark provides the primitives needed to build GraphX (http://amplab.github.io/graphx/ http://amplab.github.io/graphx/, http://spark.apache.org/graphx/ http://spark.apache.org/graphx/), which is essentially GraphLab on Spark.
miralabs 12y ago
"said it got too cumbersome once the size of the data reached a few petabytes." I dont think there's a lot of companies where data would reach this huge. Anyone has any idea on how large a typical warehousing database is?
- jamesaguilar 12y ago
  What do you mean, warehousing? Like, item tracking inside an actual warehouse? Hard to imagine spending more than a kB per unique item -- more per SKU, but less per individual object -- so even if you have 1M items being tracked, the total size would only be a gigabyte. Even if you had a billion unique things in your store, the resulting database would still fit on a single flash drive.
  - seanp2k2 12y ago
    Given the context, it seems like "warehouse" in parent's comment was more specifically "data warehouse": http://en.m.wikipedia.org/wiki/Data_warehouse http://en.m.wikipedia.org/wiki/Data_warehouse While I couldn't quickly find anything that speaks to any kind of average or normal size of a data warehouse, this article mentions Facebook's being around 300PB: https://code.facebook.com/posts/229861827208629/scaling-the-facebook-data-warehouse-to-300-pb/ https://code.facebook.com/posts/229861827208629/scaling-the-...
- frozenport 12y ago
  Its about what they need and not what other companies need.
infocollector 12y ago
More snake oil? :-)
jey 12y ago
Is this basically like Apache Spark in its programming model?
- espeed 12y ago
  Yes, it's like Spark (http://spark.apache.org/ http://spark.apache.org/) and SparkStreaming (http://spark.apache.org/streaming/ http://spark.apache.org/streaming/) combined. Here are the relevant papers... * FlumeJava (iterative, data-parallel pipelines like Spark): http://pages.cs.wisc.edu/~akella/CS838/F12/838-CloudPapers/FlumeJava.pdf http://pages.cs.wisc.edu/~akella/CS838/F12/838-CloudPapers/F... * MillWheel (fault-tolerant stream processing like SparkStreaming): http://research.google.com/pubs/pub41378.html http://research.google.com/pubs/pub41378.html Pointers to the IO blog posts... * "Reimagining developer productivity and data analytics in the cloud" http://googlecloudplatform.blogspot.com/2014/06/reimagining-developer-productivity-and-data-analytics-in-the-cloud-news-from-google-io.html http://googlecloudplatform.blogspot.com/2014/06/reimagining-... * "Sneak peek: Google Cloud Dataflow, a Cloud-native data processing service" http://googlecloudplatform.blogspot.com/2014/06/sneak-peek-google-cloud-dataflow-a-cloud-native-data-processing-service.html http://googlecloudplatform.blogspot.com/2014/06/sneak-peek-g... The Dataflow-specific talks at Google IO 2014... * Big data, the Cloud way: Accelerated and simplified https://www.youtube.com/watch?v=Y0Z58YQSXv0 https://www.youtube.com/watch?v=Y0Z58YQSXv0 * The dawn of "Fast Data" https://www.youtube.com/watch?v=TnLiEWglqHk https://www.youtube.com/watch?v=TnLiEWglqHk * Predicting the future with the Google Cloud Platform https://www.youtube.com/watch?v=YyvvxFeADh8 https://www.youtube.com/watch?v=YyvvxFeADh8 * Keynote (starts at Urs Hölzle's segment on Google Cloud) https://www.youtube.com/watch?v=wtLJPvx7-ys#t=6932 https://www.youtube.com/watch?v=wtLJPvx7-ys#t=6932
  - jey 12y ago
    Cool. Does this mean Google is moving away toward languages that allow for easier use and serialization of closures than in C++ and Java? (For example, Spark uses Scala natively.)
    - espeed 12y ago
      Dataflow is language agnostic. The Java API is being released first, and more languages will follow.
t1m 12y ago
Is it just me or is using 'cloud' in their product names just shark jumping. Seriously.
entrusted 12y ago
The main takeaway from this article is that the author, Yevgeniy Sverdlik, has demonstrably never worked with distributed computing systems. The rest is buzzwords propping up sweeping ridiculous conclusions.
jpgvm 12y ago
Yay, marketing, yay.
neckbeard 12y ago
For those that missed it, similar discussion on this a couple of days ago: https://news.ycombinator.com/item?id=7947782 https://news.ycombinator.com/item?id=7947782
- dang 12y ago
  Yes, it's the same story, so we'll call this thread a dupe.
capkutay 12y ago
This sounds more like Google app engine's shot at amazon kinesis than anything else.

41 points by posharma 12y ago

wyager 12y ago

>Google Replaces MapReduce With New Hyper-Scale Cloud Analytics System Buzzword... overload!

msane 12y ago
Hey sometimes you just have to do whatever it takes to synergize global channels on virtual platforms. You know, really aggregate extensible markets with repurposed leading-edge metrics.
- deleted 12y ago
  [deleted]
- dclusin 12y ago
  But how will these leading-edge metrics enable us to deliver paradigm shifting solutions to our customers while simultaneously reducing costs and increasing operational efficiency?
  - source99 12y ago
    While making the world a better place!
  - msane 12y ago
    That's the best part - it's automated with a context-sensitive customizability infrastructure. That means you can expand any targeted benchmark in a demand-driven mesh.

CHY872 12y ago

Sounds a bit like Dryad?

_dark_matter_ 12y ago

>“We don’t really use MapReduce anymore” This is not even true. They have recently published research that involved using MapReduce in their own systems. Example: http://research.google.com/pubs/pub41376.html http://research.google.com/pubs/pub41376.html

kiyoto 12y ago
As someone who has one foot in marketing and one foot in development (I am a developer evangelist), here is what's happening. 1. Google's infrastructure is evolving atop MapReduce (FlumeJava/MillWheel). 2. Google's PR decided to call it "not using MapReduce anymore" because in marketing, "beyond <current fad>" sounds really cool. 3. The rest of the world's PR/press/marketing fall for Google's clever PR. Either way, it is great to see Google making its core technology accessible as part of their PaaS =)
- t1m 12y ago
  It would be more great if they open sourced it.
- zobzu 12y ago
  hyperscale man! cyberleetscale would have been better. IMO. Thanks for the clarifications :)
- username223 12y ago
  > 3. The rest of the world's PR/press/marketing fall for Google's clever PR. If "hyper-scale Analytics" counts as "clever PR," then eating paste counts as "clever food choice." It's not worth my time to decode the BS.
- rsync 12y ago
  I know what I'm supposed to do when my taxi driver starts recommending stocks to me ... But what do I do when people start describing themselves as "developer evangelists" ? That has to be some kind of sell signal, right ?
  - deleted 12y ago

basyt 12y ago

Its fundamentally the same thing as MapReduce isn't it? Can someone explain the differences to me please? There isn't much of use in the article

njharman 12y ago
This has "cloud" prefixed to name of every component. So, obviously, is better. Also, they're selling it. So, ya know, marketing trumps engineering.
dyoo1979 12y ago
You'll probably want to read the FlumeJava paper. http://pages.cs.wisc.edu/~akella/CS838/F12/838-CloudPapers/FlumeJava.pdf http://pages.cs.wisc.edu/~akella/CS838/F12/838-CloudPapers/F... Citation: http://dl.acm.org/citation.cfm?id=1806638 http://dl.acm.org/citation.cfm?id=1806638 The key word is pipeline. If you have some analysis that runs in several stages, you'll be taking the output of one stage, and connecting it to the next. If you want to compose multiple phases, chained together, raw MapReduce isn't going to help you very much with the chaining. What's described in the paper is a way to do the chaining in a nice way. The system will take care of writing the raw MapReduces for you. But it'll also do a lot of work on the interconnections between your stages as well.
espeed 12y ago
MapReduce wasn't designed for iterative algorithms or streaming data, whereas Google Dataflow and Spark (http://spark.apache.org/ http://spark.apache.org/) make iterative algoritms easy. It's a much simpler programming paradigm, and it allows you to do iterative graph-processing and machine-learning algos (http://spark.apache.org/mllib/ http://spark.apache.org/mllib/) that are impractical on MapReduce. For example, Spark provides the primitives needed to build GraphX (http://amplab.github.io/graphx/ http://amplab.github.io/graphx/, http://spark.apache.org/graphx/ http://spark.apache.org/graphx/), which is essentially GraphLab on Spark.

miralabs 12y ago

"said it got too cumbersome once the size of the data reached a few petabytes." I dont think there's a lot of companies where data would reach this huge. Anyone has any idea on how large a typical warehousing database is?

jamesaguilar 12y ago
What do you mean, warehousing? Like, item tracking inside an actual warehouse? Hard to imagine spending more than a kB per unique item -- more per SKU, but less per individual object -- so even if you have 1M items being tracked, the total size would only be a gigabyte. Even if you had a billion unique things in your store, the resulting database would still fit on a single flash drive.
- seanp2k2 12y ago
  Given the context, it seems like "warehouse" in parent's comment was more specifically "data warehouse": http://en.m.wikipedia.org/wiki/Data_warehouse http://en.m.wikipedia.org/wiki/Data_warehouse While I couldn't quickly find anything that speaks to any kind of average or normal size of a data warehouse, this article mentions Facebook's being around 300PB: https://code.facebook.com/posts/229861827208629/scaling-the-facebook-data-warehouse-to-300-pb/ https://code.facebook.com/posts/229861827208629/scaling-the-...
frozenport 12y ago
Its about what they need and not what other companies need.

infocollector 12y ago

More snake oil? :-)

jey 12y ago

Is this basically like Apache Spark in its programming model?

espeed 12y ago
Yes, it's like Spark (http://spark.apache.org/ http://spark.apache.org/) and SparkStreaming (http://spark.apache.org/streaming/ http://spark.apache.org/streaming/) combined. Here are the relevant papers... * FlumeJava (iterative, data-parallel pipelines like Spark): http://pages.cs.wisc.edu/~akella/CS838/F12/838-CloudPapers/FlumeJava.pdf http://pages.cs.wisc.edu/~akella/CS838/F12/838-CloudPapers/F... * MillWheel (fault-tolerant stream processing like SparkStreaming): http://research.google.com/pubs/pub41378.html http://research.google.com/pubs/pub41378.html Pointers to the IO blog posts... * "Reimagining developer productivity and data analytics in the cloud" http://googlecloudplatform.blogspot.com/2014/06/reimagining-developer-productivity-and-data-analytics-in-the-cloud-news-from-google-io.html http://googlecloudplatform.blogspot.com/2014/06/reimagining-... * "Sneak peek: Google Cloud Dataflow, a Cloud-native data processing service" http://googlecloudplatform.blogspot.com/2014/06/sneak-peek-google-cloud-dataflow-a-cloud-native-data-processing-service.html http://googlecloudplatform.blogspot.com/2014/06/sneak-peek-g... The Dataflow-specific talks at Google IO 2014... * Big data, the Cloud way: Accelerated and simplified https://www.youtube.com/watch?v=Y0Z58YQSXv0 https://www.youtube.com/watch?v=Y0Z58YQSXv0 * The dawn of "Fast Data" https://www.youtube.com/watch?v=TnLiEWglqHk https://www.youtube.com/watch?v=TnLiEWglqHk * Predicting the future with the Google Cloud Platform https://www.youtube.com/watch?v=YyvvxFeADh8 https://www.youtube.com/watch?v=YyvvxFeADh8 * Keynote (starts at Urs Hölzle's segment on Google Cloud) https://www.youtube.com/watch?v=wtLJPvx7-ys#t=6932 https://www.youtube.com/watch?v=wtLJPvx7-ys#t=6932
- jey 12y ago
  Cool. Does this mean Google is moving away toward languages that allow for easier use and serialization of closures than in C++ and Java? (For example, Spark uses Scala natively.)
  - espeed 12y ago
    Dataflow is language agnostic. The Java API is being released first, and more languages will follow.

t1m 12y ago

Is it just me or is using 'cloud' in their product names just shark jumping. Seriously.

entrusted 12y ago

The main takeaway from this article is that the author, Yevgeniy Sverdlik, has demonstrably never worked with distributed computing systems. The rest is buzzwords propping up sweeping ridiculous conclusions.

jpgvm 12y ago

Yay, marketing, yay.

neckbeard 12y ago

For those that missed it, similar discussion on this a couple of days ago: https://news.ycombinator.com/item?id=7947782 https://news.ycombinator.com/item?id=7947782

dang 12y ago
Yes, it's the same story, so we'll call this thread a dupe.

capkutay 12y ago

This sounds more like Google app engine's shot at amazon kinesis than anything else.