10 ms·
The Elephant was a Trojan Horse: On the Death of Map-Reduce at Google
- jmillikin 12y ago> This morning, at their I/O Conference, Google revealed > that they’re not using Map-Reduce to process data > internally at all any more. This is incorrect, so monumentally so that I couldn't continue reading. It's as if the author had opened an article about climate change with "Now that advancing glaciers have rendered Algeria uninhabitable..." MapReduce doesn't work well for low-latency pipelines because it's got a high fixed overhead, but it's still the undisputed king of medium-latency and latency-insensitive workloads.
- thrownaway2424 12y ago"For one, we don't really use Map-Reduce any more." https://www.youtube.com/watch?v=wtLJPvx7-ys#t=9454 https://www.youtube.com/watch?v=wtLJPvx7-ys#t=9454
- sylvinus 12y agoNot so sure about this. I was at the conference and captured the exact wording from Urs, "We don't really use MapReduce anymore" (https://twitter.com/sylvinus/status/481864384981913601 https://twitter.com/sylvinus/status/481864384981913601)
- dsymonds 12y agoThough it seems to be quoting Urs accurately enough. All I can guess is that he meant MR isn't used for some particular context. MR is most definitely still heavily used inside Google.
- jbigelow76 12y agoYou may want to correct Urs Hölzle, Senior VP of Technical Infrastructure at Google, then or at least tell him to choose his words better. From today's I/O keynote video https://www.youtube.com/watch?v=wtLJPvx7-ys#t=9454 https://www.youtube.com/watch?v=wtLJPvx7-ys#t=9454 This is the exact quote: "... and today even when you use map-reduce, which we invented over a decade ago, it's still cumbersome to write and maintain analytics pipelines, and if you want streaming analytics you are out of luck. And in most systems once you have more than a few petabytes they kind of break down. So we've done analytics at scale for awhile and we've learned a few things. FOR ONE, WE DON'T REALLY USE MAP-REDUCE ANYMORE. It's great for simple jobs but it gets too cumbersome as you build pipelines, and everything is an analytics pipeline." emphasis mine Of course the word "really" in the middle of the sentence gives semantic wiggle room, but it's still a pretty big statement.
- gdy 12y ago>"FOR ONE, WE DON'T REALLY USE MAP-REDUCE ANYMORE" And this is said in the context of talking about streaming analytics.
- jbigelow76 12y agoBut Urs also said, paraphrasing this time, that once you get into petabytes of information everything pretty much becomes streaming analytics. Since I would assume that any non-trivial service that Google provides is in that petabyte neighborhood it explains why he would say that Google isn't using MR anymore.
- gaius 12y agoI am pretty sure that Google didn't invent map-reduce, which has been around since the 1970s at least. This guy may work for Google, but he's a clown.
- seanmcdirmid 12y agoHow many big data jobs were being processed by MapReduce in the 70s, 80s, early 90s? Ya, that's right: none. Sanjay and Jeff were the first to apply the combination of map-shuffle-and-reduce as we know it today to big data processing. Also, Urs Holzle is not a clown.
- dpritchett 12y agoAs a non-PhD who doesn't work at Google I'm having trouble reading between the lines. What exactly is the offered improvement here? Decoupled (compared to Hadoop) systems with distributed data and JIT processing?
- EdwardDiego 12y agoThe offered improvement is, I guess, process Google-sized 'Big Data' without the significant issues of Hadoop style clusters. Beyond that, there's not enough details. To give you an example, data import speed has always been an issue with Hadoop - Facebook are quite proud that they can import 320TB of data into their Hadoop clusters in a day.
- tsotha 12y agoYes, it all seems very hand-wavey to me. "Map reduce is finished, and it's been replaced by... some other thing."
- deleted 12y ago[deleted]
- rdtsc 12y agoParadigm shift -- new type of cloud processing engine + making it available to the outside.
- EdwardDiego 12y agoMake sure you crystallise your synergies while going forward.
- tupshin 12y agoProjects like Apache Spark have demonstrated the power of a more complex DAG (Directed Acyclic Graph) approach that allows for more precise control over the data-processing flow, compared with the simpler execution model of M/R. All of the major Hadoop vendors are pivoting, and simultaneously adding support for Spark (which can work with the Hadoop stack, but is not part of it), while also supporting the development of one or more technologies that are trying to retrofit Hadoop into a more powerful model, such as Tez, from Hortonworks.
- timr 12y ago"and the real contribution from Google in this area was arguably GFS, not Map-Reduce" This, a million times over. Map-reduce is not difficult to implement. Implementing a distributed, petabyte-scale filesystem to hold the data being accessed by thousands of workers is what's difficult. It's just a shame that Hadoop (and HDFS) are what we're saddled with in the outside world. They're a total disaster area, from configuration difficulty to memory usage to speed to monitoring. But since HDFS is the only commonly used distributed FS, you're pretty much bound to Hadoop (and the rest of the horrible Apache ecosystem). The world needs a good, stable, well-written distributed filesystem (ideally, one not written in a bloated language designed for remote controls and set-top boxes).
- wyager 12y ago>ideally, one not written in a bloated language designed for remote controls and set-top boxes I don't know what you'd have to be smoking to use Java in a remote control.
- riffraff 12y agojava was designed for that http://en.wikipedia.org/wiki/Oak_(programming_language) http://en.wikipedia.org/wiki/Oak_(programming_language)
- e12e 12y agoIndeed. And it runs today (on among other things) sim cards. See eg: "Defcon 21 - The Secret Life of SIM Cards" https://www.youtube.com/watch?v=31D94QOo2gY https://www.youtube.com/watch?v=31D94QOo2gY
- wyager 12y agoI know. My comment still stands.
- tptacek 12y agoIt really doesn't, in that it doesn't actually build an argument of any sort.
- coldcode 12y agoComputing darwinism. Many things are tried and most of them fail, but each try teaches something useful for the next generation.
- deleted 12y ago[deleted]
- ohashi 12y agoSo what's replacing it in the open source world? What are some real world problems Map Reduce is being used for now that some other solution is better suited for?
- toast0 12y agoA lot of problems people are solving with Hadoop and friends are best solved by getting a big box with lots of ram and lots of SSDs. Current generation dual processor socket server boards go up to 1 TB of ram, a few years ago that kind of ram would require several servers; and the Haswell EP Xeons, expected towards the end of the year will support even more ram.
- EdwardDiego 12y agoArgh, there are far too many firms who have Hadoop clusters who don't really need them.
- ohashi 12y agoSo you're essentially saying it was a stop-gap solution until hardware caught up for many use cases.
- chris_va 12y ago'The company stopped using the system “years ago.”' Hm, as a former Google engineer, that statement (from the journalist) is not accurate. Though the definition of map reduce is malleable, so it's hard to say what was meant in the first place.
- deleted 12y ago[deleted]
- nevi-me 12y agoDidn't Urs mean that it's not used for analytics anymore?
- riffraff 12y ago> It will also be no surprise to me when, eventually, Hadoop does the same, and the elephant is finally given its dotage. AFAIU, hadoop has already moved to support different workloads than MR when it introduced YARN[0] [0] http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yar...
- ambrood 12y agoThat isn't "hadoop supporting different workloads". That is a Hadoop cluster supporting different frameworks. They are two different things.
- deleted 12y ago[deleted]
- _pmf_ 12y agoMap-reduce is a brute force approach. It's obvious that intelligent, specific data handing strategies are orders of magnitude better for production systems if there's the engineering power to pull it off.
- jeffdavis 12y agoThe author discussed the data flow model -- would you consider that to be a "specific data handling strategy"? Also, why do you think developing for a map-reduce framework requires fewer engineering resources than developing for, e.g. a dataflow framework?
- rlpb 12y agoSo they're not using MapReduce, a particular implementation of the map/reduce concept. But are they still using the concept having just changed their implementation? I've read the article and this is still unclear to me; I think commentators are conflating the two.
- erikano 12y ago>[P]aradoxically, it’s easier to build robust, fault-tolerant systems from unreliable components[.] Giving this some thought, I think I see a parallell here to how most the rest of the nature works; the "components" that make up a creature appears to often be as simple as it can afford to be.