4 ms·

Honestly this is an outdated article. In 2017 Hadoop is a more like OS/Platform/Ecosystem with FS (hdfs)/ Scheduler / Applications etc. Spark, Presto, Hive now

Honestly this is an outdated article. In 2017 Hadoop is a more like OS/Platform/Ecosystem with FS (hdfs)/ Scheduler / Applications etc. Spark, Presto, Hive now ensure that you no longer have to write Map-Reduce jobs. While I understand the message about 600 MB being enough to fit in memory and speed offered by command line tools etc. Yet its better to just use Spark (which has local mode with convenient in-memory caching) so that when you have to create a company wide data handling/processing system you can just "plug-in" your code into a Spark/Presto/Hadoop cluster. Finally if you are truly looking for speed while maintaining portability, these days I would recommend using Docker containers with external volumes created on tmpfs thus providing both speed of an in-memory implementation while being agnostic to both OS and FS.

barrkel 9y ago
Yup. Hadoop is a DIY database engine ecosystem; the value is in the ecosystem and the choice of plugging in different bits, all tied together around a common core in HDFS and possibly YARN.
makapuf 9y ago
Honestly, you also must factor out the price of distributed computing, and take into account the fact that processing power and disk size have increased a lot in that timeframe also. Then you don't need a new OS/Platform/Ecosystem with FS (hdfs)/ Scheduler / Applications to maintain, host and secure to compute thing that could be done with local scripts, or even java / cython programs. or distribute on a few servers what can (often) be distributed easily. that still covers most of the use cases.
- threeseed 9y ago
  Not sure that you understand what you're saying. You don't need a new OS/Platform/Ecosystem to run Hadoop/Spark. It is literally just half a dozen Java apps that you run on typical hardware on your regular Linux OS. That is specifically what it was designed for. So rather than write your own Java apps and distribute it yourself why not use a platform that is proven, performant and with Spark will allow you to do things that you could never write yourself.
  - makapuf 9y ago
    The OS/Platform/Ecosystem was a reference to the post I was replying to. To be clearer, I don't think you need such things when you just need one or maybe a few servers talking together when you remove the cost of parallelism (see article about COST) and all the hadoop machinery. I have an OS, platform and ecosystem, it's called unix and works for 95% of my needs. Frankly I've seen my current project be implemented partly on single node (to be replicated for later scalability) and partly multiprocess python. The python is maybe 10x speedier and has 10% the code and run complexity. We tried to run it on a shared cluster of hundred nodes for giggles, the python finished running before the other one started launching. Not taking into account the setup / maintenance. The COST model is a good one indeed. So I'm roughly paraphrasing the article (which I point anyone talking about needing big data to, just before asking how big their data exactly is.)
deleted 9y ago
[deleted]
fulafel 9y ago
But if you start with 600 MB, you have quite a bit of headroom before you grow out of memory on a single server. Normal off the shelf servers fit 3-6 TB, which 5 orders of magniture up from 600 MB. Even AWS EC2 has 2 TB instances.

barrkel 9y ago

Yup. Hadoop is a DIY database engine ecosystem; the value is in the ecosystem and the choice of plugging in different bits, all tied together around a common core in HDFS and possibly YARN.

makapuf 9y ago

Honestly, you also must factor out the price of distributed computing, and take into account the fact that processing power and disk size have increased a lot in that timeframe also. Then you don't need a new OS/Platform/Ecosystem with FS (hdfs)/ Scheduler / Applications to maintain, host and secure to compute thing that could be done with local scripts, or even java / cython programs. or distribute on a few servers what can (often) be distributed easily. that still covers most of the use cases.

threeseed 9y ago
Not sure that you understand what you're saying. You don't need a new OS/Platform/Ecosystem to run Hadoop/Spark. It is literally just half a dozen Java apps that you run on typical hardware on your regular Linux OS. That is specifically what it was designed for. So rather than write your own Java apps and distribute it yourself why not use a platform that is proven, performant and with Spark will allow you to do things that you could never write yourself.
- makapuf 9y ago
  The OS/Platform/Ecosystem was a reference to the post I was replying to. To be clearer, I don't think you need such things when you just need one or maybe a few servers talking together when you remove the cost of parallelism (see article about COST) and all the hadoop machinery. I have an OS, platform and ecosystem, it's called unix and works for 95% of my needs. Frankly I've seen my current project be implemented partly on single node (to be replicated for later scalability) and partly multiprocess python. The python is maybe 10x speedier and has 10% the code and run complexity. We tried to run it on a shared cluster of hundred nodes for giggles, the python finished running before the other one started launching. Not taking into account the setup / maintenance. The COST model is a good one indeed. So I'm roughly paraphrasing the article (which I point anyone talking about needing big data to, just before asking how big their data exactly is.)

deleted 9y ago

[deleted]

fulafel 9y ago

But if you start with 600 MB, you have quite a bit of headroom before you grow out of memory on a single server. Normal off the shelf servers fit 3-6 TB, which 5 orders of magniture up from 600 MB. Even AWS EC2 has 2 TB instances.