4 ms·
Hadoop is the problem, MapReduce is not the problem. Having used both Hadoop and Disco, I can say that Disco is by far a net positive on all projects I used it
by sitkack 13y ago
Hadoop is the problem, MapReduce is not the problem. Having used both Hadoop and Disco, I can say that Disco is by far a net positive on all projects I used it on. And the overhead to coding it in Disco vs single node is about an extra 30 minutes. You can start with working single node and go multinode w/o much effort.
http://discoproject.org http://discoproject.org
Hadoop on the other hand is a huge, massive pain in the ass. And I am a Hadoop consultant. I recommend that most customers NOT use it.
- edraferi 13y agoCool project, thanks for the link.
- travisoliphant 13y agoThis is 100% true. Hadoop gets way too much attention given the other useful solutions that exist out there. I have known people to use Disco successfully on several hundred node cluster. You can also interact Disco with IPython parallel much more easily. This is why we include it in Linux versions of Anaconda.
- hhuuggoo 13y agoagreed, disco is really great becuase you can easily get access to the file descriptor which has your data, then you can load it however way you want