8 ms·

> This morning, at their I/O Conference, Google revealed > that they’re not using Map-Reduce to process data > internally at all any more. This is incorrec

> This morning, at their I/O Conference, Google revealed > that they’re not using Map-Reduce to process data > internally at all any more. This is incorrect, so monumentally so that I couldn't continue reading. It's as if the author had opened an article about climate change with "Now that advancing glaciers have rendered Algeria uninhabitable..." MapReduce doesn't work well for low-latency pipelines because it's got a high fixed overhead, but it's still the undisputed king of medium-latency and latency-insensitive workloads.

thrownaway2424 12y ago
"For one, we don't really use Map-Reduce any more." https://www.youtube.com/watch?v=wtLJPvx7-ys#t=9454 https://www.youtube.com/watch?v=wtLJPvx7-ys#t=9454
sylvinus 12y ago
Not so sure about this. I was at the conference and captured the exact wording from Urs, "We don't really use MapReduce anymore" (https://twitter.com/sylvinus/status/481864384981913601 https://twitter.com/sylvinus/status/481864384981913601)
dsymonds 12y ago
Though it seems to be quoting Urs accurately enough. All I can guess is that he meant MR isn't used for some particular context. MR is most definitely still heavily used inside Google.
- mtdewcmu 12y ago
  Well, you would expect there to be legacy applications around for a while, if they decided not to use MR in future algorithms.
- jrockway 12y ago
  Apparently the context was streaming analytics, which is a workload that MapReduce is poorly suited to.
jbigelow76 12y ago
You may want to correct Urs Hölzle, Senior VP of Technical Infrastructure at Google, then or at least tell him to choose his words better. From today's I/O keynote video https://www.youtube.com/watch?v=wtLJPvx7-ys#t=9454 https://www.youtube.com/watch?v=wtLJPvx7-ys#t=9454 This is the exact quote: "... and today even when you use map-reduce, which we invented over a decade ago, it's still cumbersome to write and maintain analytics pipelines, and if you want streaming analytics you are out of luck. And in most systems once you have more than a few petabytes they kind of break down. So we've done analytics at scale for awhile and we've learned a few things. FOR ONE, WE DON'T REALLY USE MAP-REDUCE ANYMORE. It's great for simple jobs but it gets too cumbersome as you build pipelines, and everything is an analytics pipeline." emphasis mine Of course the word "really" in the middle of the sentence gives semantic wiggle room, but it's still a pretty big statement.
- gdy 12y ago
  >"FOR ONE, WE DON'T REALLY USE MAP-REDUCE ANYMORE" And this is said in the context of talking about streaming analytics.
  - jbigelow76 12y ago
    But Urs also said, paraphrasing this time, that once you get into petabytes of information everything pretty much becomes streaming analytics. Since I would assume that any non-trivial service that Google provides is in that petabyte neighborhood it explains why he would say that Google isn't using MR anymore.
- gaius 12y ago
  I am pretty sure that Google didn't invent map-reduce, which has been around since the 1970s at least. This guy may work for Google, but he's a clown.
  - seanmcdirmid 12y ago
    How many big data jobs were being processed by MapReduce in the 70s, 80s, early 90s? Ya, that's right: none. Sanjay and Jeff were the first to apply the combination of map-shuffle-and-reduce as we know it today to big data processing. Also, Urs Holzle is not a clown.

thrownaway2424 12y ago

"For one, we don't really use Map-Reduce any more." https://www.youtube.com/watch?v=wtLJPvx7-ys#t=9454 https://www.youtube.com/watch?v=wtLJPvx7-ys#t=9454

sylvinus 12y ago

Not so sure about this. I was at the conference and captured the exact wording from Urs, "We don't really use MapReduce anymore" (https://twitter.com/sylvinus/status/481864384981913601 https://twitter.com/sylvinus/status/481864384981913601)

dsymonds 12y ago

Though it seems to be quoting Urs accurately enough. All I can guess is that he meant MR isn't used for some particular context. MR is most definitely still heavily used inside Google.

mtdewcmu 12y ago
Well, you would expect there to be legacy applications around for a while, if they decided not to use MR in future algorithms.
jrockway 12y ago
Apparently the context was streaming analytics, which is a workload that MapReduce is poorly suited to.

jbigelow76 12y ago

You may want to correct Urs Hölzle, Senior VP of Technical Infrastructure at Google, then or at least tell him to choose his words better. From today's I/O keynote video https://www.youtube.com/watch?v=wtLJPvx7-ys#t=9454 https://www.youtube.com/watch?v=wtLJPvx7-ys#t=9454 This is the exact quote: "... and today even when you use map-reduce, which we invented over a decade ago, it's still cumbersome to write and maintain analytics pipelines, and if you want streaming analytics you are out of luck. And in most systems once you have more than a few petabytes they kind of break down. So we've done analytics at scale for awhile and we've learned a few things. FOR ONE, WE DON'T REALLY USE MAP-REDUCE ANYMORE. It's great for simple jobs but it gets too cumbersome as you build pipelines, and everything is an analytics pipeline." emphasis mine Of course the word "really" in the middle of the sentence gives semantic wiggle room, but it's still a pretty big statement.

gdy 12y ago
>"FOR ONE, WE DON'T REALLY USE MAP-REDUCE ANYMORE" And this is said in the context of talking about streaming analytics.
- jbigelow76 12y ago
  But Urs also said, paraphrasing this time, that once you get into petabytes of information everything pretty much becomes streaming analytics. Since I would assume that any non-trivial service that Google provides is in that petabyte neighborhood it explains why he would say that Google isn't using MR anymore.
gaius 12y ago
I am pretty sure that Google didn't invent map-reduce, which has been around since the 1970s at least. This guy may work for Google, but he's a clown.
- seanmcdirmid 12y ago
  How many big data jobs were being processed by MapReduce in the 70s, 80s, early 90s? Ya, that's right: none. Sanjay and Jeff were the first to apply the combination of map-shuffle-and-reduce as we know it today to big data processing. Also, Urs Holzle is not a clown.