4 ms·
I do the same on Google Compute Engine, except without the auto-terminate and scaling :( However, Google's bdutil has a great set of shell scripts which auto s
by MoOmer 11y ago
I do the same on Google Compute Engine, except without the auto-terminate and scaling :(
However, Google's bdutil has a great set of shell scripts which auto setup the environment; and, with minimal changes you can set up the exact Scala/Spark versions you need.
The fact that I (just one dude) can set up a pipeline and chomp through TBs of data on clusters with TBs of memory over the course of hours still keeps me in awe of the advances of both GCE and AWS.
I'll have to give EMR/AWS a shot!
- jeffreysmith 11y agoI have nothing against GCE. There's certainly innovation going on there. The new dataflow system certainly includes some very exciting and powerful ideas: https://cloud.google.com/dataflow/ https://cloud.google.com/dataflow/ The empowerment of these platforms is something that I'm very excited about. Spark will allow you to go from some fairly basic processing of small files on your laptop to a massive cluster that will process huge amounts of data very efficiently. And EMR makes all of that even easier. This is something that I hope to convey in my upcoming book: http://www.reactivemachinelearning.com/ http://www.reactivemachinelearning.com/ One of the ideas that I'm playing with is that big data and small data are basically the same. You should assume that you have an infinite amount of data, and then you'll build your system to handle whatever comes at it. Even if you end up not having a ton of data, you won't be sorry you used awesome tools like Spark.