4 ms·
To be clear, Disco is a full replacement for Hadoop (including MapReduce, HDFS, etc). I've heard good things, but you also leave the (rather large) Hadoop ecos
by bretthoerner 16y ago
To be clear, Disco is a full replacement for Hadoop (including MapReduce, HDFS, etc). I've heard good things, but you also leave the (rather large) Hadoop ecosystem.
As a Python developer I've found Dumbo / mrjob to be easy to use.
- boyter 16y agoThere is that. For those curious Disco has an Erlang/Python base. It does seem to be getting better too. I also found it easier to set up a Map/Reduce cluster in it then Hadoop.
- coderdude 16y agoWhich do you prefer using, Dumbo or mrjob?
- bretthoerner 16y agoThey're pretty similar, it's hard to make map reduce very complex in Python. What mrjob has going for it is that it's very easy to run a job on Amazon's EMR (Elastic Map Reduce) so you don't have to do any Hadoop setup. You can run Dumbo on EMR, too, it's just more manual (last I checked). That said, Dumbo has been around longer and seems to be smarter about input/output formats. It seems that by default it can ready just about any type of Hadoop format (look up SequenceFiles, for example) and by default it'll output compressed SequenceFiles, too, which help save space and I/O.