3 ms·
Hadoop streaming is definitely a great way to use Hadoop and actually provided a lot of the early inspiration for Pachyderm. The thing that I always found diffi
by jdoliner 12y ago
Hadoop streaming is definitely a great way to use Hadoop and actually provided a lot of the early inspiration for Pachyderm. The thing that I always found difficult when using it was dependency management. If I wanted to use libraries it was a pain to get them distributed across all of the machines in the cluster and invariably I'd get one machine with a different version of something that would mess things up. This problem was the impetus for using Docker to do MapReduce.
- lclarkmichalek 12y agoOr, you known, docker run -i and use streaming still :) Edit: that said, there's a ton of stuff you could work on there. Not convinced I'd abandon hadoop so quickly just to avoid wrapping my scripts in `docker run --rm -i foo ..`, but that's not the only api that's possible. But even then, not sure I'd want to tie myself so closely to docker still :)
- buryat 12y agoYou can specify files & archives that have to be copied to nodes before executing the job: > The -files and -archives options allow you to make files and archives available to the tasks. The argument is a URI to the file or archive that you have already uploaded to HDFS. http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/HadoopStreaming.html#Working_with_Large_Files_and_Archives http://hadoop.apache.org/docs/current/hadoop-mapreduce-clien...