4 ms·

A Guide to Python Frameworks for Hadoop

93 points by laserson 14y ago

monstrado 14y ago
A while back I did some tests to compare the Hadoop Streaming library to the Dumbo library. I have to agree, if your MapReduce jobs are not crazy complicated, and don't require chaining...it's best to just write it using the raw Streaming library.
bcbrown 14y ago
The only framework I've used of the ones discussed is Hadoop Streaming with Python. For our use case (rapid prototyping of statistical analytics on several-terabyte structured data), it worked perfectly and was almost frictionless. As the article calls out, we had to detect the boundaries between keys manually, but that didn't add much complexity. We called into Python through Hive scripts. Our group had little prior experience; some had used Python before, no one had used Hive/Hadoop much or at all, but we were all productive within a day of ramp-up time. I'm positive we were more productive than if we'd implemented everything in Java, even for the people more experienced in Java than Python. If I have any future projects requiring similar analysis, I'd like to use Hive with Hadoop Streaming and Python again.
stevejohnson 14y ago
Former[1] primary mrjob maintainer here, thanks for the shout-out! I'd like to make a couple of notes and corrections. In particular, I thank you for recommending mrjob for EMR usage, as it's something we've made a point of trying to be the best at. First of all, all of these frameworks use Hadoop Streaming. As mentioned in the mrjob 0.4-dev docs [2] (pardon the run-on sentence): "Although Hadoop is primarly designed to work with JVM code, it supports other languages via Hadoop Streaming, a special jar which calls an arbitrary program as a subprocess, passing input via stdin and gathering results via stdout." mrjob's role is to give you a structured way to write Hadoop Streaming jobs in Python (or, recently, any language). When your task runs, it's taking input and output the same way as your raw Python example does, except mrjob is passing it through to the methods you've defined after running each line through a deserialization function. It picks the code to run based on command line arguments such as --mapper, --reducer, etc. The output of the code is again serialized. The methods of [de]serialization are defined declaritively in your class, as you showed. So why did you find mrjob to be slower than bare Hadoop Streaming? I don't know! In theory, you're running approximately the same code between the mrjob and the bare Python versions of your script. If anyone has time to dig into this and find out where that time is being spent, I would be grateful. Results should be sent to the issue tracker on the Github page [3]. Feel free to ask clarifying questions. I realize I may not be explaining this effectively to people unfamiliar with the ins and outs of Python MapReduce frameworks. I'm thinking of organizing the 2nd "mrjob hackathon" in the near future, so please ping me if you're interested in contributing to an easy-to-handle lots-of-low-hanging-fruit OSS project. (Particularly if you're a Rubyist, because we have an experimental way to use Ruby with it.) [1] mrjob is maintained by Yelp, where I worked until recently. It's still under active development, though it's slowed somewhat since Dave and I moved on. [2] http://mrjob.readthedocs.org/en/latest/guides/concepts.html#hadoop-streaming-and-mrjob http://mrjob.readthedocs.org/en/latest/guides/concepts.html#... [3] http://www.github.com/yelp/mrjob/issues/new http://www.github.com/yelp/mrjob/issues/new
- stevejohnson 14y ago
  One last thing I forgot to mention: there was a typedbytes support pull request for mrjob at one point, but it was behind the dev branch enough that it wasn't practical to merge. It is possible (probable?) that typedbytes support will make it into a future release if an interested party could put in the time. I would have done it myself if I had been able to make head or tail of the typedbytes documentation. Also, if you're just starting out or want to look over the docs, I'd recommend using the dev version hosted on readthedocs instead of the PyPI version, as the author did: http://mrjob.readthedocs.org/en/latest/index.html http://mrjob.readthedocs.org/en/latest/index.html
  - laserson 14y ago
    I did only a bit of profiling on mrjob, but it seemed that the ser/de was significantly slowing down the overall computation.
    - stevejohnson 14y ago
      Here's the code that's doing the actual line parsing: https://github.com/Yelp/mrjob/blob/master/mrjob/protocol.py#L144 https://github.com/Yelp/mrjob/blob/master/mrjob/protocol.py#... It's just splitting on tab for input and re-joining on tab for output. Some extra logic for lines without tabs. EDIT: The example in the post is using JSON for communication between intermediate steps, while the Hadoop Streaming example is using a custom delimiter format. So this isn't really a fair comparison; the mrjob example could just as easily use the same efficient intermediate format.
      - laserson 14y ago
        Here is the profiling output: https://gist.github.com/4478737 https://gist.github.com/4478737 The input is RawProtocol, which simply splits on tab. But after that, mrjob defaults to using JSON internally, and this is causing a lot of slowdown.
        14y ago
throwaway54-762 14y ago
Slightly tangential, but I'd like to shamelessly plug my HDFS client library (with nice Python bindings)[0]. If you want to access files on HDFS from your Python tasks outside of your typical map / shuffle inputs from the streaming API, it might be handy? It doesn't go through the JVM (the library is in C), so it might save a little latency for short Python tasks. [0]: https://github.com/cemeyer/hadoofus https://github.com/cemeyer/hadoofus Also, I'm pretty new to publishing my own open source libraries. If people would be so kind, I'd love some constructive criticism. Thanks HN!
- stevejohnson 14y ago
  The main thing you need to do is publish your Python package to PyPI. This should fill in the gaps in your Python packaging knowledge: http://guide.python-distribute.org/ http://guide.python-distribute.org/
  - throwaway54-762 14y ago
    It's a C library, though — is that acceptable for PyPI? Thanks for the feedback!
    - stevejohnson 14y ago
      Yep - if it's a Python library that can be 'python setup.py install'ed, then it should be on PyPI. That's true of modules requiring C extensions.
seany 14y ago
You can do hbase stuff via streaming as well with this library https://github.com/vanship82/hadoop-hbase-streaming https://github.com/vanship82/hadoop-hbase-streaming

93 points by laserson 14y ago

A while back I did some tests to compare the Hadoop Streaming library to the Dumbo library. I have to agree, if your MapReduce jobs are not crazy complicated, and don't require chaining...it's best to just write it using the raw Streaming library.

bcbrown 14y ago

The only framework I've used of the ones discussed is Hadoop Streaming with Python. For our use case (rapid prototyping of statistical analytics on several-terabyte structured data), it worked perfectly and was almost frictionless. As the article calls out, we had to detect the boundaries between keys manually, but that didn't add much complexity. We called into Python through Hive scripts. Our group had little prior experience; some had used Python before, no one had used Hive/Hadoop much or at all, but we were all productive within a day of ramp-up time. I'm positive we were more productive than if we'd implemented everything in Java, even for the people more experienced in Java than Python. If I have any future projects requiring similar analysis, I'd like to use Hive with Hadoop Streaming and Python again.

stevejohnson 14y ago

Former[1] primary mrjob maintainer here, thanks for the shout-out! I'd like to make a couple of notes and corrections. In particular, I thank you for recommending mrjob for EMR usage, as it's something we've made a point of trying to be the best at. First of all, all of these frameworks use Hadoop Streaming. As mentioned in the mrjob 0.4-dev docs [2] (pardon the run-on sentence): "Although Hadoop is primarly designed to work with JVM code, it supports other languages via Hadoop Streaming, a special jar which calls an arbitrary program as a subprocess, passing input via stdin and gathering results via stdout." mrjob's role is to give you a structured way to write Hadoop Streaming jobs in Python (or, recently, any language). When your task runs, it's taking input and output the same way as your raw Python example does, except mrjob is passing it through to the methods you've defined after running each line through a deserialization function. It picks the code to run based on command line arguments such as --mapper, --reducer, etc. The output of the code is again serialized. The methods of [de]serialization are defined declaritively in your class, as you showed. So why did you find mrjob to be slower than bare Hadoop Streaming? I don't know! In theory, you're running approximately the same code between the mrjob and the bare Python versions of your script. If anyone has time to dig into this and find out where that time is being spent, I would be grateful. Results should be sent to the issue tracker on the Github page [3]. Feel free to ask clarifying questions. I realize I may not be explaining this effectively to people unfamiliar with the ins and outs of Python MapReduce frameworks. I'm thinking of organizing the 2nd "mrjob hackathon" in the near future, so please ping me if you're interested in contributing to an easy-to-handle lots-of-low-hanging-fruit OSS project. (Particularly if you're a Rubyist, because we have an experimental way to use Ruby with it.) [1] mrjob is maintained by Yelp, where I worked until recently. It's still under active development, though it's slowed somewhat since Dave and I moved on. [2] http://mrjob.readthedocs.org/en/latest/guides/concepts.html#hadoop-streaming-and-mrjob http://mrjob.readthedocs.org/en/latest/guides/concepts.html#... [3] http://www.github.com/yelp/mrjob/issues/new http://www.github.com/yelp/mrjob/issues/new

stevejohnson 14y ago
One last thing I forgot to mention: there was a typedbytes support pull request for mrjob at one point, but it was behind the dev branch enough that it wasn't practical to merge. It is possible (probable?) that typedbytes support will make it into a future release if an interested party could put in the time. I would have done it myself if I had been able to make head or tail of the typedbytes documentation. Also, if you're just starting out or want to look over the docs, I'd recommend using the dev version hosted on readthedocs instead of the PyPI version, as the author did: http://mrjob.readthedocs.org/en/latest/index.html http://mrjob.readthedocs.org/en/latest/index.html
- laserson 14y ago
  I did only a bit of profiling on mrjob, but it seemed that the ser/de was significantly slowing down the overall computation.
  - stevejohnson 14y ago
    Here's the code that's doing the actual line parsing: https://github.com/Yelp/mrjob/blob/master/mrjob/protocol.py#L144 https://github.com/Yelp/mrjob/blob/master/mrjob/protocol.py#... It's just splitting on tab for input and re-joining on tab for output. Some extra logic for lines without tabs. EDIT: The example in the post is using JSON for communication between intermediate steps, while the Hadoop Streaming example is using a custom delimiter format. So this isn't really a fair comparison; the mrjob example could just as easily use the same efficient intermediate format.
    - laserson 14y ago
      Here is the profiling output: https://gist.github.com/4478737 https://gist.github.com/4478737 The input is RawProtocol, which simply splits on tab. But after that, mrjob defaults to using JSON internally, and this is causing a lot of slowdown.
      - 14y ago

throwaway54-762 14y ago

Slightly tangential, but I'd like to shamelessly plug my HDFS client library (with nice Python bindings)[0]. If you want to access files on HDFS from your Python tasks outside of your typical map / shuffle inputs from the streaming API, it might be handy? It doesn't go through the JVM (the library is in C), so it might save a little latency for short Python tasks. [0]: https://github.com/cemeyer/hadoofus https://github.com/cemeyer/hadoofus Also, I'm pretty new to publishing my own open source libraries. If people would be so kind, I'd love some constructive criticism. Thanks HN!

stevejohnson 14y ago
The main thing you need to do is publish your Python package to PyPI. This should fill in the gaps in your Python packaging knowledge: http://guide.python-distribute.org/ http://guide.python-distribute.org/
- throwaway54-762 14y ago
  It's a C library, though — is that acceptable for PyPI? Thanks for the feedback!
  - stevejohnson 14y ago
    Yep - if it's a Python library that can be 'python setup.py install'ed, then it should be on PyPI. That's true of modules requiring C extensions.

seany 14y ago

You can do hbase stuff via streaming as well with this library https://github.com/vanship82/hadoop-hbase-streaming https://github.com/vanship82/hadoop-hbase-streaming