3 ms·
Can anyone tell me how, as a sole developer, it's possible to gain real-world experience with distributed Hadoop and Spark given the massive computing resources
by cutler 9y ago
Can anyone tell me how, as a sole developer, it's possible to gain real-world experience with distributed Hadoop and Spark given the massive computing resources required? It just seems like a closed shop to me.
- teraflop 9y agoDefine "massive". You can learn all the most important aspects of the APIs and programming model on a single machine, either by running in "local" (non-distributed) mode or by running a few VMs to simulate a real cluster. And you can spin up a cluster of 100 machines on either AWS or GCE for about a dollar per hour.
- jkestelyn 9y agoGCP offers a $300 credit that expires after 1 year. It's a good way to get your feet wet with Hadoop and Spark via Cloud Dataproc. (Google Cloud employee speaking.)