3 ms·
Not sure that you understand what you're saying. You don't need a new OS/Platform/Ecosystem to run Hadoop/Spark. It is literally just half a dozen Java apps th
by threeseed 9y ago
Not sure that you understand what you're saying.
You don't need a new OS/Platform/Ecosystem to run Hadoop/Spark. It is literally just half a dozen Java apps that you run on typical hardware on your regular Linux OS. That is specifically what it was designed for. So rather than write your own Java apps and distribute it yourself why not use a platform that is proven, performant and with Spark will allow you to do things that you could never write yourself.
- makapuf 9y agoThe OS/Platform/Ecosystem was a reference to the post I was replying to. To be clearer, I don't think you need such things when you just need one or maybe a few servers talking together when you remove the cost of parallelism (see article about COST) and all the hadoop machinery. I have an OS, platform and ecosystem, it's called unix and works for 95% of my needs. Frankly I've seen my current project be implemented partly on single node (to be replicated for later scalability) and partly multiprocess python. The python is maybe 10x speedier and has 10% the code and run complexity. We tried to run it on a shared cluster of hundred nodes for giggles, the python finished running before the other one started launching. Not taking into account the setup / maintenance. The COST model is a good one indeed. So I'm roughly paraphrasing the article (which I point anyone talking about needing big data to, just before asking how big their data exactly is.)