3 ms·

Does anyone know how this framework stacks up against Dumbo (Last.fm) and/or Pydoop? I've used Dumbo before and had great luck. With Dumbo, the only problem is

by kevnin 13y ago

Does anyone know how this framework stacks up against Dumbo (Last.fm) and/or Pydoop? I've used Dumbo before and had great luck. With Dumbo, the only problem is the lack of consolidated documentation. Much of the knowledge is lost in the maintainer's blog

stevejohnson 13y ago
I don't know a thing about Pydoop, but here's what makes mrjob unique with particular respect to Dumbo: First of all, mrjob has a lot of documentation. Some things are too hard to find in the haystack, but that's more a problem of organization than content. (Always looking for suggestions on how to improve that!) As for the rest... mrjob tries to make all parts of Hadoop and Amazon Elastic MapReduce jobs easy to write, run, and test. It can run without Hadoop at all, it can use your own Hadoop cluster, or it will automatically handle all the pain of shipping a job to Amazon. mrjob gives you a single API to use across all those platforms and a simple way to switch between them via the command line. It helps you manage file dependencies and greps the myriad logs for errors. Dumbo (and AFAICT Pydoop) handle the Python-Hadoop bridge, but don't do all that file management and config mishmash. They also don't provide a "Hadoop simulator" for your automated tests. If you want to use one of those tools, you need a Hadoop cluster to run your code, and they won't help you move dependencies and input files around. The flip side to that is that mrjob doesn't give you the same level of access to Hadoop APIs that Dumbo and Pydoop do. It's simplified a great deal. But that hasn't stopped several companies, including Yelp, from using it for day-to-day heavy lifting. And in my opinion, the simplification is a good thing for most cases. I should also mention that Dumbo can be faster if you use typedbytes. In theory mrjob could support typedbytes too, but we haven't gotten a patch we were happy with yet. In the end, mrjob's goal is for these to all give you the same result: > python my_job.py -r local input.txt > output > python my_job.py -r hadoop input.txt > output # assuming Hadoop configs > python my_job.py -r emr input.txt > output # assuming AWS configs
mattj 13y ago
(original author of mrjob here) Steve's post is 100% correct. I originally wrote mrjob as an internal tool at yelp out of my frustration with using dumbo for multi-step jobs. Specifically, I found myself writing the same incantation of "wrap a mapper / reducer function with an encoding scheme" over and over again. I tried to add protocol support into dumbo (so you could specify that your job reads json, uses pickle for intermediate data, and writes thrift), but I had a hard time working with the dumbo codebase (disclaimer: I haven't looked at it since, so it might be easy to do this now). I also wanted to represent mappers and reducers as python generators, which makes writing memory-performant steps natural (eg you normally want to rely on the shuffle / sort to perform the hard work of aggregating by key). Finally, I wanted my jobs to be easy to test both from unittest and from the command line - debugging hadoop streaming jobs is way more of a pain in the ass than it should be.

by kevnin 13y ago

stevejohnson 13y ago

I don't know a thing about Pydoop, but here's what makes mrjob unique with particular respect to Dumbo: First of all, mrjob has a lot of documentation. Some things are too hard to find in the haystack, but that's more a problem of organization than content. (Always looking for suggestions on how to improve that!) As for the rest... mrjob tries to make all parts of Hadoop and Amazon Elastic MapReduce jobs easy to write, run, and test. It can run without Hadoop at all, it can use your own Hadoop cluster, or it will automatically handle all the pain of shipping a job to Amazon. mrjob gives you a single API to use across all those platforms and a simple way to switch between them via the command line. It helps you manage file dependencies and greps the myriad logs for errors. Dumbo (and AFAICT Pydoop) handle the Python-Hadoop bridge, but don't do all that file management and config mishmash. They also don't provide a "Hadoop simulator" for your automated tests. If you want to use one of those tools, you need a Hadoop cluster to run your code, and they won't help you move dependencies and input files around. The flip side to that is that mrjob doesn't give you the same level of access to Hadoop APIs that Dumbo and Pydoop do. It's simplified a great deal. But that hasn't stopped several companies, including Yelp, from using it for day-to-day heavy lifting. And in my opinion, the simplification is a good thing for most cases. I should also mention that Dumbo can be faster if you use typedbytes. In theory mrjob could support typedbytes too, but we haven't gotten a patch we were happy with yet. In the end, mrjob's goal is for these to all give you the same result: > python my_job.py -r local input.txt > output > python my_job.py -r hadoop input.txt > output # assuming Hadoop configs > python my_job.py -r emr input.txt > output # assuming AWS configs

mattj 13y ago

(original author of mrjob here) Steve's post is 100% correct. I originally wrote mrjob as an internal tool at yelp out of my frustration with using dumbo for multi-step jobs. Specifically, I found myself writing the same incantation of "wrap a mapper / reducer function with an encoding scheme" over and over again. I tried to add protocol support into dumbo (so you could specify that your job reads json, uses pickle for intermediate data, and writes thrift), but I had a hard time working with the dumbo codebase (disclaimer: I haven't looked at it since, so it might be easy to do this now). I also wanted to represent mappers and reducers as python generators, which makes writing memory-performant steps natural (eg you normally want to rely on the shuffle / sort to perform the hard work of aggregating by key). Finally, I wanted my jobs to be easy to test both from unittest and from the command line - debugging hadoop streaming jobs is way more of a pain in the ass than it should be.