8 ms·
I am pretty sure that Google didn't invent map-reduce, which has been around since the 1970s at least. This guy may work for Google, but he's a clown.
by gaius 12y ago
I am pretty sure that Google didn't invent map-reduce, which has been around since the 1970s at least.
This guy may work for Google, but he's a clown.
- seanmcdirmid 12y agoHow many big data jobs were being processed by MapReduce in the 70s, 80s, early 90s? Ya, that's right: none. Sanjay and Jeff were the first to apply the combination of map-shuffle-and-reduce as we know it today to big data processing. Also, Urs Holzle is not a clown.
- gaius 12y ago"Big data" is not a thing, and neither is "the cloud", while I'm here.
- seanmcdirmid 12y agoWell, then, you really don't understand the value of their contribution, which you have in your mind is just "map" and "reduce."
- walshemj 12y agoBritish Telecom used map reduce in billing systems for the dialcom (telecom gold) platform in the 80's - that was on the largest (non black) prime minicomputer site in the UK. Back then 17x 750's would be roughly the same as one the 5k plus clusters that yahoo etal use. We even sold the system to NZ telecom
- seanmcdirmid 12y agoInteresting. What kind of distributed file system were they using?
- supermatt 12y agoA dfs isn't a requirement for map/reduce.
- seanmcdirmid 12y agoFrom http://en.wikipedia.org/wiki/MapReduce http://en.wikipedia.org/wiki/MapReduce: > MapReduce is a programming model and an associated implementation for processing and generating large data sets with a parallel, distributed algorithm on a cluster. ... > The "MapReduce System" (also called "infrastructure" or "framework") orchestrates by marshalling the distributed servers, running the various tasks in parallel, managing all communications and data transfers between the various parts of the system, and providing for redundancy and fault tolerance. ... > The name MapReduce originally referred to the proprietary Google technology but has since been genericized. So it would be quite impossible to have a MapReduce system without distributed computing infrastructure; even if you were doing mapping and reducing, it wouldn't be MapReduce.
- supermatt 12y agoI see no mention of a distributed file system there. Local storage is not a requirement of distributed processing.
- 1stop 12y agoHow do you do distributed processing without a distributed filesystem? Do you mean you'd load the filesystem into memory and send it to the "processors"?
- supermatt 12y agoThe data could be stored on a network device, such as a file server or database, for example. It could indeed be local, but it needn't be distributed. In the example GP gave, the data could possibly have been stored in a database queried using segmentation via consistent hashing (a basic way to distribute jobs across a known number of workers).
- walshemj 12y agowhat does falsely claiming that google invented MR make him then ?
- gaius 12y agoI see the Google fanboys and wannabes are out in force on this thread.
- seanmcdirmid 12y agoI see the crazies are out trying to redefine MapReduce as just being map and reduce and completely missing the point. But whatever, they've probably never seen big data loads and are definitely not involved in the industry.
- gaius 12y agoOoh, scary big data. I could run your workloads in Excel without breaking a sweat. But go on kidding yourself.
- seanmcdirmid 12y agoI don't think Excel scales to 10 or 100 TB of data.
- gaius 12y agoIn all seriousness tho', I was running data sets that big in Oracle, in 2006. You can see why I don't take "big data" seriously.
- ithkuil 12y agoThere's certainly a hype around big data nowadays, often even up to the point of being ridiculous. The point is that people are starting to use this term to describe something that it's not even technical anymore, let alone describe the actual amount of data: merely using data to drive decision making. This is not a new thing [0], yet there is a clear trend that shows how this kind of dependency is shifting from being auxiliary to being generative; some of the reasons are: 1. cheaper computing and storage power 2. increased computing literacy among scientists and not. 3. increased availability of digitalised content in many areas that capture human behaviour. When there's request, there's opportunity for business. One thing that is new and big about Big Data is the market. It should be called "Big Market (of data)". It's an overloaded term. IMHO it's counterproductive to let the hype around Big Data as a business term pollute the discussion about what contribution Google and others have made in the field of computer science and data processing. So what did Google really invent? Obviously the name and concept behind MapReduce wasn't new. Nor the fact that they did start to process large amounts of data. Size and growth are two key factors here. Although it's possible that the NIH syndrome affected Google, it's possible that existing products just weren't able to solve those two requirements. It's difficult to tell exactly how large given that the Google is not very keen at releasing numbers, although it's possible to find some announcements like [1] "Google processed about 24 petabytes of data per day in 2009". 20P is 10000 times more that 200 T. Stop to think a moment what does 10000 mean. It's enough to completely change the problem, almost any problem. A room full of people becomes a metropolis; an US annual low wage salary becomes 100 million dollars, more than the annual spending of Palau [2]. Well, it's silly to make those comparison, but it's hard to think about anything that scaled by 10000 doesn't change profoundly. Hell, this absurdly long post is well under 10k! To stay in the realm of computer science, processor performance didn't increase by a factor of 10000 since PDP-11 from 1978 to Xeon from 2005 [3]. Working at that scale poses unique problems, and that's where real the contributions to the advancement of the field made by the engineers and the engineering culture at Google are placed. If anything, just knowing it's possible and having some accounts on what they focused on is inspiring. This is the Big Data I care about. It's not about fanboyism. It's cool, it's real, it's rare. Arguing who invented the map reduce mechanics is like arguing that hierarchical filesystems where already there hence any progress made in that area by countless engineers is just trivial. [0] Historical perspective: James Gleick , http://en.wikipedia.org/wiki/The_Information:_A_History,_a_Theory,_a_Flood http://en.wikipedia.org/wiki/The_Information:_A_History,_a_T... [1] http://dl.acm.org/citation.cfm?doid=1327452.1327492 http://dl.acm.org/citation.cfm?doid=1327452.1327492 [2] https://www.cia.gov/library/publications/the-world-factbook/geos/ps.html https://www.cia.gov/library/publications/the-world-factbook/... [3] http://www.cs.columbia.edu/~sedwards/classes/2012/3827-spring/advanced-arch-2011.pdf http://www.cs.columbia.edu/~sedwards/classes/2012/3827-sprin...
- dbc1012 12y agoI don't know about Mr Holzle but you're wrong about map/reduce. I'm aware of two significant counterexamples. I'm sure there are others. Teradata's been doing map/reduce in their proprietary DBC 1012 AMP clusters since the 80's, providing analytical data warehousing for some of the world's largest companies[1]. Walmart used them to globally optimize their inventory. MPI systems have been supporting distributed map/reduce operations since the early 90's (see MPI_REDUCE[2]). 1- http://www.cs.rutgers.edu/~rmartin/teaching/fall99/papers/teradata-DBC.pdf http://www.cs.rutgers.edu/~rmartin/teaching/fall99/papers/te... 2- http://www.mpi-forum.org/docs/mpi-1.0/mpi-10.ps http://www.mpi-forum.org/docs/mpi-1.0/mpi-10.ps
- rbanffy 12y agoWhat was big data in the 70s, 80s and 90s? We just didn't call it map-reduce at the time.