4 ms·

> Hadoop solved a hard problem that nobody had. This baseless assertion is patently wrong on so many levels. Building computing clusters on COTS hardware is a

> Hadoop solved a hard problem that nobody had. This baseless assertion is patently wrong on so many levels. Building computing clusters on COTS hardware is a very mundane problem. Running processing jobs on data shards is a very mundane problem. Scaling COTS clusters transparently is a very mundane problem.

stingraycharles 7y ago
Insert "almost" into parent's sentence and it becomes correct. Many people use/used Hadoop for problems that did not warrant the overhead and complexity that comes with Hadoop. I've seen it countless times with my own eyes that people pre-emptively use tools like Hadoop and Spark because of a chance that they will hit a massive scale in the future. This happens in both startups and enterprises alike: people like to think they have big problems too often.
- jabl 7y ago
  > This happens in both startups and enterprises alike: people like to think they have big problems too often. A.k.a. resume-driven development. Having Hadoop on your CV looks sexier than awk.
  - arethuza 7y ago
    I was at a conference once where someone was presenting and describing something where they used Hadoop to do pretty simple processing on ~30MB of data and the audience appeared to be lapping it up. I just sat there thinking I could probably run what they did on my phone. And just to be clear - it wasn't a PoC or a demo.
- coredog64 7y ago
  This. A prior employer spent millions on a Cloudera install that had 5 worker nodes and was being used to service a bunch of generic jobs. Worse still, it didn’t even use HDFS and we eventually got sick of the crappy embedded Zookeeper/Kafka setup.
- geezerjay 7y ago
  > Insert "almost" into parent's sentence and it becomes correct. No, it still remains astonishingly wrong. Even container orchestration platforms are being adapted to provide the same service that Hadoop has been providing for years, and no one in their right mind would claim that running processing jobs on the cloud is a problem that almost no one has.
danpalmer 7y ago
There are broadly 3 levels of the amount of data that people have: - Fits on one computer (most of the market) - Fits on several computers (most of the rest) - Requires a significant cluster of machines (50+ to store it) Hadoop only really solves the last one. It has huge overheads in terms of speed and in terms of resources and headcount to run it properly, so it only makes sense at a particular scale. It's like a mainframe – most companies shouldn't buy one. If you add to this the fact that Hadoop was about batch processing, and its "realtime" capabilities were poor, there really aren't that many potential customers, and many of the potential customers would rather run it in-house, or build their own system.
- thrower123 7y ago
  > - Fits on one computer (most of the market) There's one category above that, which is "Fits in memory" and that is a huge chunk of the market. I've seen first hand people getting way too cute and complicated planning for scale, and then it works out that they don't even have more than a couple of GB of data.
  - _dps 7y ago
    I remember distinctly a conversation I had in early 2012 about what I jokingly called "big memory" as killing off the (then ubiquitous) "big data" trend. Most people's sensibilities have not adjusted to a world where getting 64GB of memory in a server is a triviality, and you can 8x that without too much effort or cost. Unless you're storing media, or you are truly "web-scale", your business data will very likely fit in 512GB.
  - danpalmer 7y ago
    I've heard of a startup spending many days of contractor time to optimise their database so they didn't have to go from the 4GB Heroku database to the 8GB one. I've heard of a company refusing to purchase an external drive for an employee so they could process a handful of ~50GB datasets on their MacBook Air – instead forcing them to use "the cloud" or constantly download and backup datasets. I've heard of companies doing extensive work to set up Hadoop to process a few GB. Roughly I'd suggest that "fits on my laptop" is <1TB, "fits in memory" is < 1TB, "fits on one computer" is < 10TB, "fits on a small cluster" is <100TB, and "might be worth Hadoop" is >100TB. I could be too low on these though.

stingraycharles 7y ago

Insert "almost" into parent's sentence and it becomes correct. Many people use/used Hadoop for problems that did not warrant the overhead and complexity that comes with Hadoop. I've seen it countless times with my own eyes that people pre-emptively use tools like Hadoop and Spark because of a chance that they will hit a massive scale in the future. This happens in both startups and enterprises alike: people like to think they have big problems too often.

jabl 7y ago
> This happens in both startups and enterprises alike: people like to think they have big problems too often. A.k.a. resume-driven development. Having Hadoop on your CV looks sexier than awk.
- arethuza 7y ago
  I was at a conference once where someone was presenting and describing something where they used Hadoop to do pretty simple processing on ~30MB of data and the audience appeared to be lapping it up. I just sat there thinking I could probably run what they did on my phone. And just to be clear - it wasn't a PoC or a demo.
coredog64 7y ago
This. A prior employer spent millions on a Cloudera install that had 5 worker nodes and was being used to service a bunch of generic jobs. Worse still, it didn’t even use HDFS and we eventually got sick of the crappy embedded Zookeeper/Kafka setup.
geezerjay 7y ago
> Insert "almost" into parent's sentence and it becomes correct. No, it still remains astonishingly wrong. Even container orchestration platforms are being adapted to provide the same service that Hadoop has been providing for years, and no one in their right mind would claim that running processing jobs on the cloud is a problem that almost no one has.

danpalmer 7y ago

There are broadly 3 levels of the amount of data that people have: - Fits on one computer (most of the market) - Fits on several computers (most of the rest) - Requires a significant cluster of machines (50+ to store it) Hadoop only really solves the last one. It has huge overheads in terms of speed and in terms of resources and headcount to run it properly, so it only makes sense at a particular scale. It's like a mainframe – most companies shouldn't buy one. If you add to this the fact that Hadoop was about batch processing, and its "realtime" capabilities were poor, there really aren't that many potential customers, and many of the potential customers would rather run it in-house, or build their own system.

thrower123 7y ago
> - Fits on one computer (most of the market) There's one category above that, which is "Fits in memory" and that is a huge chunk of the market. I've seen first hand people getting way too cute and complicated planning for scale, and then it works out that they don't even have more than a couple of GB of data.
- _dps 7y ago
  I remember distinctly a conversation I had in early 2012 about what I jokingly called "big memory" as killing off the (then ubiquitous) "big data" trend. Most people's sensibilities have not adjusted to a world where getting 64GB of memory in a server is a triviality, and you can 8x that without too much effort or cost. Unless you're storing media, or you are truly "web-scale", your business data will very likely fit in 512GB.
- danpalmer 7y ago
  I've heard of a startup spending many days of contractor time to optimise their database so they didn't have to go from the 4GB Heroku database to the 8GB one. I've heard of a company refusing to purchase an external drive for an employee so they could process a handful of ~50GB datasets on their MacBook Air – instead forcing them to use "the cloud" or constantly download and backup datasets. I've heard of companies doing extensive work to set up Hadoop to process a few GB. Roughly I'd suggest that "fits on my laptop" is <1TB, "fits in memory" is < 1TB, "fits on one computer" is < 10TB, "fits on a small cluster" is <100TB, and "might be worth Hadoop" is >100TB. I could be too low on these though.