3 ms·

Does anyone see any obvious documentation on defining your own reduce function? All the example use the built in "aggregate," which looks like it refers to the

Does anyone see any obvious documentation on defining your own reduce function? All the example use the built in "aggregate," which looks like it refers to the Hadoop aggregate sub-package. The lines output by the map functions start with the name of a Hadoop class within the aggregate package, which must tell it how to aggregate.

gfunk911 18y ago
The PDF here seems to be a good start. http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2297&categoryID=265 http://developer.amazonwebservices.com/connect/entry.jspa?ex...
rjurney 18y ago
Map/Reduce is really hard. As you can tell, this thing takes the operational pain out of Hadoop... not the thinkin part :) Beyond something like this: http://hadoop.apache.org/core/docs/current/mapred_tutorial.html http://hadoop.apache.org/core/docs/current/mapred_tutorial.h... there isn't too much in the way of docs on how to think in map/reduce that I'm aware of, but I found the classes in the Pig tutorial to be quite helpful. edit: oh, guy above me has great link!
- pskomoroch 18y ago
  Here is another great resource for how to "think" in terms of mapreduce (joins, keyswaps etc): http://www.cloudera.com/hadoop-training-basic http://www.cloudera.com/hadoop-training-basic
pskomoroch 18y ago
The item similarity code example that uses the api has 4 python streaming mapreduce steps with custom reducers: http://elasticmapreduce.s3.amazonaws.com/samples/similarity/similarity.py http://elasticmapreduce.s3.amazonaws.com/samples/similarity/... AWS just put a pointer to the code bucket on s3, I'll add a link to a zip file with the full source including a script for converting the raw netflix data to something more friendly to Hadoop streaming. The article is here: http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2294&categoryID=265 http://developer.amazonwebservices.com/connect/entry.jspa?ex... Check out the resources link at the end, the Cloudera Hadoop Hacks site has a few good intros to custom reduers in java, ruby, and python.
- pskomoroch 18y ago
  Here is a direct link to the full code example from the article: http://elasticmapreduce.s3.amazonaws.com/samples/similarity/similarity.zip http://elasticmapreduce.s3.amazonaws.com/samples/similarity/... You will get much better performance out of Hadoop if you do this in Java, but I thought python would be more readable for a wider audience. Hadoop 18.3 doesn't have numerical sort in streaming (see https://issues.apache.org/jira/browse/HADOOP-2302 https://issues.apache.org/jira/browse/HADOOP-2302 ), so the code is slightly more complicated than it needs to be. Dumbo will also probably be supported in the next release, then the code example would be even simpler.

gfunk911 18y ago

The PDF here seems to be a good start. http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2297&categoryID=265 http://developer.amazonwebservices.com/connect/entry.jspa?ex...

rjurney 18y ago

Map/Reduce is really hard. As you can tell, this thing takes the operational pain out of Hadoop... not the thinkin part :) Beyond something like this: http://hadoop.apache.org/core/docs/current/mapred_tutorial.html http://hadoop.apache.org/core/docs/current/mapred_tutorial.h... there isn't too much in the way of docs on how to think in map/reduce that I'm aware of, but I found the classes in the Pig tutorial to be quite helpful. edit: oh, guy above me has great link!

pskomoroch 18y ago
Here is another great resource for how to "think" in terms of mapreduce (joins, keyswaps etc): http://www.cloudera.com/hadoop-training-basic http://www.cloudera.com/hadoop-training-basic

pskomoroch 18y ago

The item similarity code example that uses the api has 4 python streaming mapreduce steps with custom reducers: http://elasticmapreduce.s3.amazonaws.com/samples/similarity/similarity.py http://elasticmapreduce.s3.amazonaws.com/samples/similarity/... AWS just put a pointer to the code bucket on s3, I'll add a link to a zip file with the full source including a script for converting the raw netflix data to something more friendly to Hadoop streaming. The article is here: http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2294&categoryID=265 http://developer.amazonwebservices.com/connect/entry.jspa?ex... Check out the resources link at the end, the Cloudera Hadoop Hacks site has a few good intros to custom reduers in java, ruby, and python.

pskomoroch 18y ago
Here is a direct link to the full code example from the article: http://elasticmapreduce.s3.amazonaws.com/samples/similarity/similarity.zip http://elasticmapreduce.s3.amazonaws.com/samples/similarity/... You will get much better performance out of Hadoop if you do this in Java, but I thought python would be more readable for a wider audience. Hadoop 18.3 doesn't have numerical sort in streaming (see https://issues.apache.org/jira/browse/HADOOP-2302 https://issues.apache.org/jira/browse/HADOOP-2302 ), so the code is slightly more complicated than it needs to be. Dumbo will also probably be supported in the next release, then the code example would be even simpler.