6 ms·
You may want to correct Urs Hölzle, Senior VP of Technical Infrastructure at Google, then or at least tell him to choose his words better. From today's I/O key
by jbigelow76 12y ago
You may want to correct Urs Hölzle, Senior VP of Technical Infrastructure at Google, then or at least tell him to choose his words better.
From today's I/O keynote video https://www.youtube.com/watch?v=wtLJPvx7-ys#t=9454 https://www.youtube.com/watch?v=wtLJPvx7-ys#t=9454
This is the exact quote:
"... and today even when you use map-reduce, which we invented over a decade ago, it's still cumbersome to write and maintain analytics pipelines, and if you want streaming analytics you are out of luck. And in most systems once you have more than a few petabytes they kind of break down. So we've done analytics at scale for awhile and we've learned a few things. FOR ONE, WE DON'T REALLY USE MAP-REDUCE ANYMORE. It's great for simple jobs but it gets too cumbersome as you build pipelines, and everything is an analytics pipeline."
emphasis mine
Of course the word "really" in the middle of the sentence gives semantic wiggle room, but it's still a pretty big statement.
- gdy 12y ago>"FOR ONE, WE DON'T REALLY USE MAP-REDUCE ANYMORE" And this is said in the context of talking about streaming analytics.
- jbigelow76 12y agoBut Urs also said, paraphrasing this time, that once you get into petabytes of information everything pretty much becomes streaming analytics. Since I would assume that any non-trivial service that Google provides is in that petabyte neighborhood it explains why he would say that Google isn't using MR anymore.
- gaius 12y agoI am pretty sure that Google didn't invent map-reduce, which has been around since the 1970s at least. This guy may work for Google, but he's a clown.
- seanmcdirmid 12y agoHow many big data jobs were being processed by MapReduce in the 70s, 80s, early 90s? Ya, that's right: none. Sanjay and Jeff were the first to apply the combination of map-shuffle-and-reduce as we know it today to big data processing. Also, Urs Holzle is not a clown.
- gaius 12y ago"Big data" is not a thing, and neither is "the cloud", while I'm here.
- seanmcdirmid 12y agoWell, then, you really don't understand the value of their contribution, which you have in your mind is just "map" and "reduce."
- walshemj 12y agoBritish Telecom used map reduce in billing systems for the dialcom (telecom gold) platform in the 80's - that was on the largest (non black) prime minicomputer site in the UK. Back then 17x 750's would be roughly the same as one the 5k plus clusters that yahoo etal use. We even sold the system to NZ telecom
- seanmcdirmid 12y agoInteresting. What kind of distributed file system were they using?
- supermatt 12y agoA dfs isn't a requirement for map/reduce.
- seanmcdirmid 12y agoFrom http://en.wikipedia.org/wiki/MapReduce http://en.wikipedia.org/wiki/MapReduce: > MapReduce is a programming model and an associated implementation for processing and generating large data sets with a parallel, distributed algorithm on a cluster. ... > The "MapReduce System" (also called "infrastructure" or "framework") orchestrates by marshalling the distributed servers, running the various tasks in parallel, managing all communications and data transfers between the various parts of the system, and providing for redundancy and fault tolerance. ... > The name MapReduce originally referred to the proprietary Google technology but has since been genericized. So it would be quite impossible to have a MapReduce system without distributed computing infrastructure; even if you were doing mapping and reducing, it wouldn't be MapReduce.