4 ms·
What toolset are you using that you can run both locally and on a Hadoop cluster?
by azylman 12y ago
What toolset are you using that you can run both locally and on a Hadoop cluster?
- arjie 12y agoI believe he mentioned it. The Hadoop streaming mode.
- mdaniel 12y agoAlmost all of them? The vocabulary of the grandparent comment implies they are using hadoop's streaming mode, and thus one can use a map-reduce streaming abstraction such as MRJob or just plain stdin/stdout; both will work locally and in cluster mode. Or, if static typing is more agreeable to your development process, running hadoop in "single machine cluster" mode is relatively painless. The same goes for other distributed processing frameworks like Spark.