3 ms·

The item similarity code example that uses the api has 4 python streaming mapreduce steps with custom reducers: http://elasticmapreduce.s3.amazonaws.com/sample

The item similarity code example that uses the api has 4 python streaming mapreduce steps with custom reducers: http://elasticmapreduce.s3.amazonaws.com/samples/similarity/similarity.py http://elasticmapreduce.s3.amazonaws.com/samples/similarity/... AWS just put a pointer to the code bucket on s3, I'll add a link to a zip file with the full source including a script for converting the raw netflix data to something more friendly to Hadoop streaming. The article is here: http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2294&categoryID=265 http://developer.amazonwebservices.com/connect/entry.jspa?ex... Check out the resources link at the end, the Cloudera Hadoop Hacks site has a few good intros to custom reduers in java, ruby, and python.

pskomoroch 18y ago
Here is a direct link to the full code example from the article: http://elasticmapreduce.s3.amazonaws.com/samples/similarity/similarity.zip http://elasticmapreduce.s3.amazonaws.com/samples/similarity/... You will get much better performance out of Hadoop if you do this in Java, but I thought python would be more readable for a wider audience. Hadoop 18.3 doesn't have numerical sort in streaming (see https://issues.apache.org/jira/browse/HADOOP-2302 https://issues.apache.org/jira/browse/HADOOP-2302 ), so the code is slightly more complicated than it needs to be. Dumbo will also probably be supported in the next release, then the code example would be even simpler.