3 ms·

Honestly, you also must factor out the price of distributed computing, and take into account the fact that processing power and disk size have increased a lot i

Honestly, you also must factor out the price of distributed computing, and take into account the fact that processing power and disk size have increased a lot in that timeframe also. Then you don't need a new OS/Platform/Ecosystem with FS (hdfs)/ Scheduler / Applications to maintain, host and secure to compute thing that could be done with local scripts, or even java / cython programs. or distribute on a few servers what can (often) be distributed easily. that still covers most of the use cases.

threeseed 9y ago
Not sure that you understand what you're saying. You don't need a new OS/Platform/Ecosystem to run Hadoop/Spark. It is literally just half a dozen Java apps that you run on typical hardware on your regular Linux OS. That is specifically what it was designed for. So rather than write your own Java apps and distribute it yourself why not use a platform that is proven, performant and with Spark will allow you to do things that you could never write yourself.
- makapuf 9y ago
  The OS/Platform/Ecosystem was a reference to the post I was replying to. To be clearer, I don't think you need such things when you just need one or maybe a few servers talking together when you remove the cost of parallelism (see article about COST) and all the hadoop machinery. I have an OS, platform and ecosystem, it's called unix and works for 95% of my needs. Frankly I've seen my current project be implemented partly on single node (to be replicated for later scalability) and partly multiprocess python. The python is maybe 10x speedier and has 10% the code and run complexity. We tried to run it on a shared cluster of hundred nodes for giggles, the python finished running before the other one started launching. Not taking into account the setup / maintenance. The COST model is a good one indeed. So I'm roughly paraphrasing the article (which I point anyone talking about needing big data to, just before asking how big their data exactly is.)

threeseed 9y ago

Not sure that you understand what you're saying. You don't need a new OS/Platform/Ecosystem to run Hadoop/Spark. It is literally just half a dozen Java apps that you run on typical hardware on your regular Linux OS. That is specifically what it was designed for. So rather than write your own Java apps and distribute it yourself why not use a platform that is proven, performant and with Spark will allow you to do things that you could never write yourself.

makapuf 9y ago
The OS/Platform/Ecosystem was a reference to the post I was replying to. To be clearer, I don't think you need such things when you just need one or maybe a few servers talking together when you remove the cost of parallelism (see article about COST) and all the hadoop machinery. I have an OS, platform and ecosystem, it's called unix and works for 95% of my needs. Frankly I've seen my current project be implemented partly on single node (to be replicated for later scalability) and partly multiprocess python. The python is maybe 10x speedier and has 10% the code and run complexity. We tried to run it on a shared cluster of hundred nodes for giggles, the python finished running before the other one started launching. Not taking into account the setup / maintenance. The COST model is a good one indeed. So I'm roughly paraphrasing the article (which I point anyone talking about needing big data to, just before asking how big their data exactly is.)