7 ms·

Map Reduce: A simple introduction (2010)

117 points by awjr 13y ago

usujason 13y ago
This is a good explanation and I would love to share it but the poor spelling and grammar makes it un-shareable.
- nocman 13y ago
  Perfectionist much? While I can agree that pointing out the spelling and/or grammar mistakes could be constructive, calling the article "un-shareable" because of them seems terribly over-the-top.
  - snoman 13y ago
    Unless you're already familiar with the material, spelling and grammar lends credibility to the content of the article. Similar to a code smell, it makes you ask "are you sure you know what you're doing?"
    - nocman 13y ago
      While I agree that numerous spelling and grammar mistakes can indicate a lack of maturity in writing skills, I also know that some of the most brilliant people I've ever worked with were absolutely terrible at spelling. One guy in particular was an amazing programmer. He produced libraries of code that were fast, efficient, and easy to read and understand. They were also very well documented. I know, because I went through the documentation and fixed the dozens and dozens of spelling errors. If I didn't know otherwise, I'd have thought he made a game out of how poorly he spelled words. I can appreciate numerous spelling/grammar errors making you analyze an article with a bit more scrutiny. However, I can't think of a single example of an article with those kind of errors where I couldn't figure out from the content whether I thought the author really knew what they were talking about. At any rate, I think there is a big difference between an article needing a little bit more scrutiny, and the article being "unsharable".
      - usujason 13y ago
        Just my personal opinion, no disrespect intended. I have personally authored posts that needed a bit more time on the proofreading table and I did worry that people would question my authority on the subject due to grammatical errors in the article.
        nocman 13y ago
        Fair enough. Just to be clear, I don't want to give the impression that I don't think spelling/grammar matter (even for blog posts). I just think it is easy to get so pedantic about it that you place far too much weight on them. I have my own set of pet peeves, and am probably guilty of allowing the violation of one of them to taint my view of an article too quickly and too often. And of course, there are always those articles that are so bad grammatically that it looks like a first-grader wrote them (but I don't think that's the kind of article we were discussing).
- awjr 13y ago
  To be honest, I don't think (and I'm going off the author's name) that english is his first language. Personally, I posted this to HN as it succinctly gave me a quick overview of MR. I try not to judge an article/blog post on the quality of the grammar when the ideas they are trying to convey are solid and useful. There are some exceptionally fantastic minds out there that are able to succinctly explain, sometimes difficult, subjects. I appreciate them for at least trying, and if they explained it in a language that is not their first language, all the more respect to them.
- srcreigh 13y ago
  The only place I noticed spelling errors was in the email from the CEO... and spelling errors there seemed to fit.
triplesec 13y ago
This is a fantastic explanation. anyone got any favourite uses of this / other resources to share of similar clarity?
- pdpi 13y ago
  I'm a big fan of (and occasional poster to) Reddit's Explain Like I'm Five, where you can ask questions and people are supposed to answer them in simple, accessible terms (if not in the "Little Timmy" literal 5yo sort of speech) You can find it at http://www.reddit.com/r/explainlikeimfive http://www.reddit.com/r/explainlikeimfive
- oliwary 13y ago
  I really like this one: http://www.michaelnielsen.org/ddi/how-the-bitcoin-protocol-actually-works/ http://www.michaelnielsen.org/ddi/how-the-bitcoin-protocol-a... He designs his own cryptocoin to explain the choices made by bitcoin and how it works.
mirajshah 13y ago
Great explanation, quite cheeky as well :)
RankingMember 13y ago
At first I thought you were making fun of the CEO with the misspellings. Then I continued reading and realized you just don't use spell check. ;) Interesting article, thanks.
jacquesm 13y ago
I love the punchline, besides the ridiculous requirements and the deadline the way it is dealt with upon completion is way too close to comfort, I may have worked at some point in the distant past for that particular boss.
briffle 13y ago
Map Reduce seems very interesting, but every example I have seen explains it in terms of counting frequency of words in documents. I would love to have someone explain it with an actual business example. I can't think of many real world uses where counting the frequency of words would matter to most businesses. (besides maybe some analysis of log files)
- svachalek 13y ago
  The earliest search engines could be described as pretty much exactly this map-reduce.
- bretthoerner 13y ago
  Here's a tiny twist on WordCount that many, many companies use MR for (often their first foot in the water): you have many gigabytes or terabytes of web access logs and need to bill customers via some API scheme like /api/3001?x=1&y=2 where "3001" there is the customer ID. You add some regex and now you're doing "WordCount" on the customer ID (per day, month, etc).
- pacala 13y ago
  You can implement SQL on top of MapReduce.
- agibsonccc 13y ago
  Let's use a simple data mining example with logs. You want to count the number of users by region visiting your website to understand where your users are coming from. In your log files you have the IP which you can use to geolocate those users. These counts can be caluclated in parallel. We do this by first "mapping" each line in a log file extracting out the IP. The output of map will be a geolocated hash by region. Reduce's goal is to calculate an answer based on these keys. If we think about the output of map being the geo location, we can then collect a count by geo located hash of where our users are coming from. Map: Map the raw input to some key value space Reduce: Reduce the output of the map to some aggregate result, think of this like the sql aggregation functions Hope that helps!
- patrickmay 13y ago
sebnukem2 13y ago
The many typos of the text make it almost difficult to read...
deutronium 13y ago
Am I right in thinking MapReduce seems to be going out of fashion somewhat as even Google themselves have moved towards using something they term Millwheel. Which is a stream based processing system. I have seen one paper on a streaming MapReduce solution though.
- bcuccioli 13y ago
  Wow. I worked on Millwheel as a summer intern the summer before last. At the time it was a team of about 11 people. I'm honestly pretty surprised to see this comment as I thought it was just a small internal research project. Have you seen any references to it in the wild other than the Google Research paper?
  - deutronium 13y ago
    Oh maybe I'm wrong, I really thought I saw something that said it was used for the index creation. I'm just having a look over the papers I've read. They do definitely seem to have switched from MapReduce though at least - http://www.theregister.co.uk/2010/09/09/google_caffeine_explained/ http://www.theregister.co.uk/2010/09/09/google_caffeine_expl...
- brooksd 13y ago
  Hadoop is busy replacing MapReduce with Yez. MapReduce was a great first implementation but we need something that performs better.
sarreph 13y ago
So this is essentially analogous to division of labour efficiencies. Or am I missing something?
- patrickmay 13y ago
  It's more a technique/architecture for parallelizing highly parallelizable tasks.

117 points by awjr 13y ago

usujason 13y ago

This is a good explanation and I would love to share it but the poor spelling and grammar makes it un-shareable.

nocman 13y ago
Perfectionist much? While I can agree that pointing out the spelling and/or grammar mistakes could be constructive, calling the article "un-shareable" because of them seems terribly over-the-top.
- snoman 13y ago
  Unless you're already familiar with the material, spelling and grammar lends credibility to the content of the article. Similar to a code smell, it makes you ask "are you sure you know what you're doing?"
  - nocman 13y ago
    While I agree that numerous spelling and grammar mistakes can indicate a lack of maturity in writing skills, I also know that some of the most brilliant people I've ever worked with were absolutely terrible at spelling. One guy in particular was an amazing programmer. He produced libraries of code that were fast, efficient, and easy to read and understand. They were also very well documented. I know, because I went through the documentation and fixed the dozens and dozens of spelling errors. If I didn't know otherwise, I'd have thought he made a game out of how poorly he spelled words. I can appreciate numerous spelling/grammar errors making you analyze an article with a bit more scrutiny. However, I can't think of a single example of an article with those kind of errors where I couldn't figure out from the content whether I thought the author really knew what they were talking about. At any rate, I think there is a big difference between an article needing a little bit more scrutiny, and the article being "unsharable".
    - usujason 13y ago
      Just my personal opinion, no disrespect intended. I have personally authored posts that needed a bit more time on the proofreading table and I did worry that people would question my authority on the subject due to grammatical errors in the article.
      - nocman 13y ago
        Fair enough. Just to be clear, I don't want to give the impression that I don't think spelling/grammar matter (even for blog posts). I just think it is easy to get so pedantic about it that you place far too much weight on them. I have my own set of pet peeves, and am probably guilty of allowing the violation of one of them to taint my view of an article too quickly and too often. And of course, there are always those articles that are so bad grammatically that it looks like a first-grader wrote them (but I don't think that's the kind of article we were discussing).
awjr 13y ago
To be honest, I don't think (and I'm going off the author's name) that english is his first language. Personally, I posted this to HN as it succinctly gave me a quick overview of MR. I try not to judge an article/blog post on the quality of the grammar when the ideas they are trying to convey are solid and useful. There are some exceptionally fantastic minds out there that are able to succinctly explain, sometimes difficult, subjects. I appreciate them for at least trying, and if they explained it in a language that is not their first language, all the more respect to them.
srcreigh 13y ago
The only place I noticed spelling errors was in the email from the CEO... and spelling errors there seemed to fit.

triplesec 13y ago

This is a fantastic explanation. anyone got any favourite uses of this / other resources to share of similar clarity?

pdpi 13y ago
I'm a big fan of (and occasional poster to) Reddit's Explain Like I'm Five, where you can ask questions and people are supposed to answer them in simple, accessible terms (if not in the "Little Timmy" literal 5yo sort of speech) You can find it at http://www.reddit.com/r/explainlikeimfive http://www.reddit.com/r/explainlikeimfive
oliwary 13y ago
I really like this one: http://www.michaelnielsen.org/ddi/how-the-bitcoin-protocol-actually-works/ http://www.michaelnielsen.org/ddi/how-the-bitcoin-protocol-a... He designs his own cryptocoin to explain the choices made by bitcoin and how it works.

mirajshah 13y ago

Great explanation, quite cheeky as well :)

RankingMember 13y ago

At first I thought you were making fun of the CEO with the misspellings. Then I continued reading and realized you just don't use spell check. ;) Interesting article, thanks.

jacquesm 13y ago

I love the punchline, besides the ridiculous requirements and the deadline the way it is dealt with upon completion is way too close to comfort, I may have worked at some point in the distant past for that particular boss.

briffle 13y ago

Map Reduce seems very interesting, but every example I have seen explains it in terms of counting frequency of words in documents. I would love to have someone explain it with an actual business example. I can't think of many real world uses where counting the frequency of words would matter to most businesses. (besides maybe some analysis of log files)

svachalek 13y ago
The earliest search engines could be described as pretty much exactly this map-reduce.
bretthoerner 13y ago
Here's a tiny twist on WordCount that many, many companies use MR for (often their first foot in the water): you have many gigabytes or terabytes of web access logs and need to bill customers via some API scheme like /api/3001?x=1&y=2 where "3001" there is the customer ID. You add some regex and now you're doing "WordCount" on the customer ID (per day, month, etc).
pacala 13y ago
You can implement SQL on top of MapReduce.
agibsonccc 13y ago
Let's use a simple data mining example with logs. You want to count the number of users by region visiting your website to understand where your users are coming from. In your log files you have the IP which you can use to geolocate those users. These counts can be caluclated in parallel. We do this by first "mapping" each line in a log file extracting out the IP. The output of map will be a geolocated hash by region. Reduce's goal is to calculate an answer based on these keys. If we think about the output of map being the geo location, we can then collect a count by geo located hash of where our users are coming from. Map: Map the raw input to some key value space Reduce: Reduce the output of the map to some aggregate result, think of this like the sql aggregation functions Hope that helps!
patrickmay 13y ago

sebnukem2 13y ago

The many typos of the text make it almost difficult to read...

deutronium 13y ago

Am I right in thinking MapReduce seems to be going out of fashion somewhat as even Google themselves have moved towards using something they term Millwheel. Which is a stream based processing system. I have seen one paper on a streaming MapReduce solution though.

bcuccioli 13y ago
Wow. I worked on Millwheel as a summer intern the summer before last. At the time it was a team of about 11 people. I'm honestly pretty surprised to see this comment as I thought it was just a small internal research project. Have you seen any references to it in the wild other than the Google Research paper?
- deutronium 13y ago
  Oh maybe I'm wrong, I really thought I saw something that said it was used for the index creation. I'm just having a look over the papers I've read. They do definitely seem to have switched from MapReduce though at least - http://www.theregister.co.uk/2010/09/09/google_caffeine_explained/ http://www.theregister.co.uk/2010/09/09/google_caffeine_expl...
brooksd 13y ago
Hadoop is busy replacing MapReduce with Yez. MapReduce was a great first implementation but we need something that performs better.

sarreph 13y ago

So this is essentially analogous to division of labour efficiencies. Or am I missing something?

patrickmay 13y ago
It's more a technique/architecture for parallelizing highly parallelizable tasks.