3 ms·

On the Death of Map-Reduce at Google

47 points by aburan28 12y ago

ot 12y ago
Previous discussion: https://news.ycombinator.com/item?id=7947782 https://news.ycombinator.com/item?id=7947782
geofft 12y ago
The link makes more sense without the anchor: http://the-paper-trail.org/blog/the-elephant-was-a-trojan-horse-on-the-death-of-map-reduce-at-google/ http://the-paper-trail.org/blog/the-elephant-was-a-trojan-ho...
enneff 12y ago
> _So it’s no surprise to hear that Google have retired Map-Reduce._ As a Googler my reaction to this statement is "LOL". We still do a ton of map reduces.
- _ak 12y ago
  I'm not sure if you're allowed to talk about it, but... I remember a talk by Rob Pike where he mentioned that Sawzall was one of the first major MR users within Google. But IIRC, Sawzall has been retired at Google some time ago, and the successor to it was developed in Go. Does the successor follow the same or similar principles in terms of parallelization, or did you follow another approach?
  - enneff 12y ago
    The successor to Sawzall is basically a framework for writing mapreduces in Go.
- stingraycharles 12y ago
  Do new applications make use of it, though? "Deprecated" might be a better term to describe it.
  - enneff 12y ago
    Yes, new mapreduces are written every day.
rwmj 12y ago
I love how the old research into this is just ignored. There is a ton of work which has showed for 20 years or more that map-reduce is just a subset of interesting parallel computation patterns. eg: http://pubs.doc.ic.ac.uk/parallel-skeleton/ http://pubs.doc.ic.ac.uk/parallel-skeleton/
threeseed 12y ago
Businesses are already mixing map reduce with other approaches. On the Hadoop platform we have Spark, YARN, Storm etc which all work on the existing infrastructure. Map reduce however is still perfectly suited for many batch processes.
dundun 12y ago
MR is actually growing at Google still and heavily used by legacy systems and experienced engineers. Compared to newer frameworks like those described in the FlumeJava and MillWheel papers, MR's growth is flat. This is also happening in the Hadoop ecosystem too: if you're writing JavaMR by hand, you're probably spending more time and writing less efficient jobs than what you might get by an optimized pig/hive job with tez under the hood. Or through something in the Cascading or Crunch family, which provides useful abstractions on top of MR or other execution engines. Then there are also a lot of tools popping up that take some if the use cases that were shoehorned into MR which are more natural outside, like ML/iterative computation through Spark. MR isn't dying inside or outside Google, it's just being abstracted away.
MrTortoise 12y ago
I'm kind of late to this map-reduce party in terms of large scale data processing. What is the paradigm shifting towards. The point about MR being a trivial subset seems rather obvious to me. Hadoop for instance is still in a phase of getting a lot of adoption yet is being implied as being a dead man walking. Raises a lot of very interesting postivie questions for me

47 points by aburan28 12y ago

ot 12y ago

Previous discussion: https://news.ycombinator.com/item?id=7947782 https://news.ycombinator.com/item?id=7947782

The link makes more sense without the anchor: http://the-paper-trail.org/blog/the-elephant-was-a-trojan-horse-on-the-death-of-map-reduce-at-google/ http://the-paper-trail.org/blog/the-elephant-was-a-trojan-ho...

enneff 12y ago

> _So it’s no surprise to hear that Google have retired Map-Reduce._ As a Googler my reaction to this statement is "LOL". We still do a ton of map reduces.

_ak 12y ago
I'm not sure if you're allowed to talk about it, but... I remember a talk by Rob Pike where he mentioned that Sawzall was one of the first major MR users within Google. But IIRC, Sawzall has been retired at Google some time ago, and the successor to it was developed in Go. Does the successor follow the same or similar principles in terms of parallelization, or did you follow another approach?
- enneff 12y ago
  The successor to Sawzall is basically a framework for writing mapreduces in Go.
stingraycharles 12y ago
Do new applications make use of it, though? "Deprecated" might be a better term to describe it.
- enneff 12y ago
  Yes, new mapreduces are written every day.

rwmj 12y ago

I love how the old research into this is just ignored. There is a ton of work which has showed for 20 years or more that map-reduce is just a subset of interesting parallel computation patterns. eg: http://pubs.doc.ic.ac.uk/parallel-skeleton/ http://pubs.doc.ic.ac.uk/parallel-skeleton/

threeseed 12y ago

Businesses are already mixing map reduce with other approaches. On the Hadoop platform we have Spark, YARN, Storm etc which all work on the existing infrastructure. Map reduce however is still perfectly suited for many batch processes.

dundun 12y ago

MR is actually growing at Google still and heavily used by legacy systems and experienced engineers. Compared to newer frameworks like those described in the FlumeJava and MillWheel papers, MR's growth is flat. This is also happening in the Hadoop ecosystem too: if you're writing JavaMR by hand, you're probably spending more time and writing less efficient jobs than what you might get by an optimized pig/hive job with tez under the hood. Or through something in the Cascading or Crunch family, which provides useful abstractions on top of MR or other execution engines. Then there are also a lot of tools popping up that take some if the use cases that were shoehorned into MR which are more natural outside, like ML/iterative computation through Spark. MR isn't dying inside or outside Google, it's just being abstracted away.

MrTortoise 12y ago

I'm kind of late to this map-reduce party in terms of large scale data processing. What is the paradigm shifting towards. The point about MR being a trivial subset seems rather obvious to me. Hadoop for instance is still in a phase of getting a lot of adoption yet is being implied as being a dead man walking. Raises a lot of very interesting postivie questions for me