4 ms·

> Companies that want to get serious about data analysis often have to hire elite programmers who specialize in writing Hadoop MapReduce jobs. There's no need

by buryat 12y ago

> Companies that want to get serious about data analysis often have to hire elite programmers who specialize in writing Hadoop MapReduce jobs. There's no need to do so, you can use Hadoop Streaming (http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/HadoopStreaming.html http://hadoop.apache.org/docs/current/hadoop-mapreduce-clien...) and anything that works with stdin/stdout.

jdoliner 12y ago
Hadoop streaming is definitely a great way to use Hadoop and actually provided a lot of the early inspiration for Pachyderm. The thing that I always found difficult when using it was dependency management. If I wanted to use libraries it was a pain to get them distributed across all of the machines in the cluster and invariably I'd get one machine with a different version of something that would mess things up. This problem was the impetus for using Docker to do MapReduce.
- lclarkmichalek 12y ago
  Or, you known, docker run -i and use streaming still :) Edit: that said, there's a ton of stuff you could work on there. Not convinced I'd abandon hadoop so quickly just to avoid wrapping my scripts in `docker run --rm -i foo ..`, but that's not the only api that's possible. But even then, not sure I'd want to tie myself so closely to docker still :)
- buryat 12y ago
  You can specify files & archives that have to be copied to nodes before executing the job: > The -files and -archives options allow you to make files and archives available to the tasks. The argument is a URI to the file or archive that you have already uploaded to HDFS. http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/HadoopStreaming.html#Working_with_Large_Files_and_Archives http://hadoop.apache.org/docs/current/hadoop-mapreduce-clien...

4 ms·

> Companies that want to get serious about data analysis often have to hire elite programmers who specialize in writing Hadoop MapReduce jobs. There's no need

by buryat 12y ago

jdoliner 12y ago
Hadoop streaming is definitely a great way to use Hadoop and actually provided a lot of the early inspiration for Pachyderm. The thing that I always found difficult when using it was dependency management. If I wanted to use libraries it was a pain to get them distributed across all of the machines in the cluster and invariably I'd get one machine with a different version of something that would mess things up. This problem was the impetus for using Docker to do MapReduce.
- lclarkmichalek 12y ago
  Or, you known, docker run -i and use streaming still :) Edit: that said, there's a ton of stuff you could work on there. Not convinced I'd abandon hadoop so quickly just to avoid wrapping my scripts in `docker run --rm -i foo ..`, but that's not the only api that's possible. But even then, not sure I'd want to tie myself so closely to docker still :)
- buryat 12y ago
  You can specify files & archives that have to be copied to nodes before executing the job: > The -files and -archives options allow you to make files and archives available to the tasks. The argument is a URI to the file or archive that you have already uploaded to HDFS. http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/HadoopStreaming.html#Working_with_Large_Files_and_Archives http://hadoop.apache.org/docs/current/hadoop-mapreduce-clien...