3 ms·
The only framework I've used of the ones discussed is Hadoop Streaming with Python. For our use case (rapid prototyping of statistical analytics on several-tera
by bcbrown 14y ago
The only framework I've used of the ones discussed is Hadoop Streaming with Python. For our use case (rapid prototyping of statistical analytics on several-terabyte structured data), it worked perfectly and was almost frictionless.
As the article calls out, we had to detect the boundaries between keys manually, but that didn't add much complexity. We called into Python through Hive scripts. Our group had little prior experience; some had used Python before, no one had used Hive/Hadoop much or at all, but we were all productive within a day of ramp-up time. I'm positive we were more productive than if we'd implemented everything in Java, even for the people more experienced in Java than Python.
If I have any future projects requiring similar analysis, I'd like to use Hive with Hadoop Streaming and Python again.