3 ms·

Yes. Very very very few people have data as big as "the internet" and the need for speed (which was the reason Google developed a look of their distributed tool

Yes. Very very very few people have data as big as "the internet" and the need for speed (which was the reason Google developed a look of their distributed tooling) that you get with these frameworks. And it really only makes sense to have permanent infrastructure for distributed computing if you're constantly using it. Like if you're constantly rebuilding an index of the internet. Which most people aren't. For occasional reindexing jobs, I've personally had success with Kubernetes. Our customer facing services were deployed with it, so it was trivial to bump up the number of nodes, and then schedule a bunch of workers containers, then roll everything down when the job is complete. No need to learn the ins-and-outs of Hadoop.

threeseed 7y ago
It is very, very, very common for companies to have enough data to need a distributed approach to ETL and especially common if they are doing any machine learning which is most of them. In telcos you have network telemetry data. In supermarkets and retail you have purchase data and often credit card gateway data. In banking and finance you obviously have transaction data. And with Kubernetes you still need a compute framework. Like I don't know the standard in the industry the Hadoop Spark framework.
- huffmsa 7y ago
  A lot of companies think they need machine learning because the management consultants tell them they do (and hey, we can set it up for you!). And those that are "doing machine learning", often aren't doing it very well and have trouble applying it to the business. Note I'm not saying all. Just 99%, like the parents comment referred to. That leftover 1% are the companies you can name off the top of your head. ExxonMobil, Target, Chase, Visa, etc.
throwaway082729 7y ago
Which is probably why these Hadoop companies have very few customers and can probably never grow bigger beyond a certain point.

threeseed 7y ago

It is very, very, very common for companies to have enough data to need a distributed approach to ETL and especially common if they are doing any machine learning which is most of them. In telcos you have network telemetry data. In supermarkets and retail you have purchase data and often credit card gateway data. In banking and finance you obviously have transaction data. And with Kubernetes you still need a compute framework. Like I don't know the standard in the industry the Hadoop Spark framework.

huffmsa 7y ago
A lot of companies think they need machine learning because the management consultants tell them they do (and hey, we can set it up for you!). And those that are "doing machine learning", often aren't doing it very well and have trouble applying it to the business. Note I'm not saying all. Just 99%, like the parents comment referred to. That leftover 1% are the companies you can name off the top of your head. ExxonMobil, Target, Chase, Visa, etc.

throwaway082729 7y ago

Which is probably why these Hadoop companies have very few customers and can probably never grow bigger beyond a certain point.