4 ms·

I have come across quite a few articles that mention that for 99% of 'big' data problems, Hadoop and the like are an overkill. Simple tools, with a beefy machin

I have come across quite a few articles that mention that for 99% of 'big' data problems, Hadoop and the like are an overkill. Simple tools, with a beefy machine is just as sufficient for the task. Is that the reality of today? Personally I too feel that distributed computing is an overkill for most 'big' data problems.

dkersten 7y ago
I've heard people say that a huge portion of the "big" data is garbage, throwing more garbage at something won't make it better, so you'd get much more out of it by cutting it down to a more manageable amount of good data, which you can then easily process in postgres or whatever. But even if you "collect everything and sort it out later", in my own personal experience and in what I've read here on HN, you can go a long long way before you need to reach for the power tools. What most companies call "big data" is typically not that much (in quantity and in velocity). Most companies don't have tens or hundreds of terabytes of data. For example, I'm currently processing timeseries data in postgres using the timescaledb extension, which makes it perform very well. Still too early to state numbers, but its looking promising so far and if their claims are true, then I won't need anything else. We will see :)
hobofan 7y ago
It really depends on how "big" your data is. If you have data that is actually big (= doesn't fit onto a machine), then Hadoop is a reasonable candidate. Otherwise you are fine with a lot simpler tools. This has probably been true for quite some time, and companies are just slowly realizing now that their relevant data isn't actually that big. On the other hand computing power and storage has still grown in the last years, and less resource intensive ETL has become more acessible, so the bar for "big" data has been raised quite a bit.
- more-coffee 7y ago
  About 5 years ago I worked at a small, business oriented telco. The biggest ETL was processing external CDRs, applying call tariffs, and creating bills and reports. A previous developer had been adamant about using a Hadoop cluster to process this, storing all of it in Cassandra NoSQL. The idea was interesting, but it didn't quite work out. At some point he left the company and we had to do something about the pipeline, as it was crashing most of the time. We did some calculations, and figured that with the right approach, a good old MySQL+PHP solution would do the trick. And it did. Having switched jobs myself in the meantime, I'm happy knowing the system is still running, and finding people to maintain it is relatively easy.
  - threeseed 7y ago
    Maybe that developer knew something unique about that use case i.e. data needs were expected to grow or they had plans to use it for Data Science (very common in telcos). And you just weren't aware of it. It's always easy to pass judgement at technology choices but in my experience they are often made with the best intentions based on requirements that not everyone is aware of.
    - huffmsa 7y ago
      Doesn't matter what he "might have known or intended to do", he didn't build a solution which met the specifications the business needed.
      - threeseed 7y ago
        You really didn't get my point did you. We don't know about the specifications because we don't work there and since the OP said he wasn't there he might not know either. And the fact is that requirements regularly change over time. The point is that it's really easy to judge when you aren't there and are privy to all the facts.
        huffmsa 7y ago
        But we do know the OPs solution met the specifications, and is still in use, meaning that those specifications haven't changed too much. And you're damn right I'm going to be judgemental of someone not only promised the moon, but abandoned the work when the rocket exploded on the pad and someone else had to clean up.
    - em500 7y ago
      More likely he knew that the pay for a hadoop / bigdata specialist was a lot better than for a mysql specialist, especially 5 years ago.
      - huffmsa 7y ago
manigandham 7y ago
Yes it's still true. Now with stream processing frameworks and infinite cloud storage, you can have a single big machine churn through TBs of data easily.
huffmsa 7y ago
Yes. Very very very few people have data as big as "the internet" and the need for speed (which was the reason Google developed a look of their distributed tooling) that you get with these frameworks. And it really only makes sense to have permanent infrastructure for distributed computing if you're constantly using it. Like if you're constantly rebuilding an index of the internet. Which most people aren't. For occasional reindexing jobs, I've personally had success with Kubernetes. Our customer facing services were deployed with it, so it was trivial to bump up the number of nodes, and then schedule a bunch of workers containers, then roll everything down when the job is complete. No need to learn the ins-and-outs of Hadoop.
- threeseed 7y ago
  It is very, very, very common for companies to have enough data to need a distributed approach to ETL and especially common if they are doing any machine learning which is most of them. In telcos you have network telemetry data. In supermarkets and retail you have purchase data and often credit card gateway data. In banking and finance you obviously have transaction data. And with Kubernetes you still need a compute framework. Like I don't know the standard in the industry the Hadoop Spark framework.
  - huffmsa 7y ago
    A lot of companies think they need machine learning because the management consultants tell them they do (and hey, we can set it up for you!). And those that are "doing machine learning", often aren't doing it very well and have trouble applying it to the business. Note I'm not saying all. Just 99%, like the parents comment referred to. That leftover 1% are the companies you can name off the top of your head. ExxonMobil, Target, Chase, Visa, etc.
- throwaway082729 7y ago
  Which is probably why these Hadoop companies have very few customers and can probably never grow bigger beyond a certain point.
chungleong 7y ago
99% of businesses don't experience exponential growth. But hope springs eternal in the human breast.
gdulli 7y ago
It's not that 99% of big data problems can't or shouldn't be solved by those tools. It's that hype makes everyone think their problem falls under the category of big data when it might not. Hitting a performance or stability limit with MySQL with an unsophisticated schema/architecture does not mean you have big data. But that's the scenario that's common.

dkersten 7y ago

I've heard people say that a huge portion of the "big" data is garbage, throwing more garbage at something won't make it better, so you'd get much more out of it by cutting it down to a more manageable amount of good data, which you can then easily process in postgres or whatever. But even if you "collect everything and sort it out later", in my own personal experience and in what I've read here on HN, you can go a long long way before you need to reach for the power tools. What most companies call "big data" is typically not that much (in quantity and in velocity). Most companies don't have tens or hundreds of terabytes of data. For example, I'm currently processing timeseries data in postgres using the timescaledb extension, which makes it perform very well. Still too early to state numbers, but its looking promising so far and if their claims are true, then I won't need anything else. We will see :)

hobofan 7y ago

It really depends on how "big" your data is. If you have data that is actually big (= doesn't fit onto a machine), then Hadoop is a reasonable candidate. Otherwise you are fine with a lot simpler tools. This has probably been true for quite some time, and companies are just slowly realizing now that their relevant data isn't actually that big. On the other hand computing power and storage has still grown in the last years, and less resource intensive ETL has become more acessible, so the bar for "big" data has been raised quite a bit.

more-coffee 7y ago
About 5 years ago I worked at a small, business oriented telco. The biggest ETL was processing external CDRs, applying call tariffs, and creating bills and reports. A previous developer had been adamant about using a Hadoop cluster to process this, storing all of it in Cassandra NoSQL. The idea was interesting, but it didn't quite work out. At some point he left the company and we had to do something about the pipeline, as it was crashing most of the time. We did some calculations, and figured that with the right approach, a good old MySQL+PHP solution would do the trick. And it did. Having switched jobs myself in the meantime, I'm happy knowing the system is still running, and finding people to maintain it is relatively easy.
- threeseed 7y ago
  Maybe that developer knew something unique about that use case i.e. data needs were expected to grow or they had plans to use it for Data Science (very common in telcos). And you just weren't aware of it. It's always easy to pass judgement at technology choices but in my experience they are often made with the best intentions based on requirements that not everyone is aware of.
  - huffmsa 7y ago
    Doesn't matter what he "might have known or intended to do", he didn't build a solution which met the specifications the business needed.
    - threeseed 7y ago
      You really didn't get my point did you. We don't know about the specifications because we don't work there and since the OP said he wasn't there he might not know either. And the fact is that requirements regularly change over time. The point is that it's really easy to judge when you aren't there and are privy to all the facts.
      - huffmsa 7y ago
        But we do know the OPs solution met the specifications, and is still in use, meaning that those specifications haven't changed too much. And you're damn right I'm going to be judgemental of someone not only promised the moon, but abandoned the work when the rocket exploded on the pad and someone else had to clean up.
  - em500 7y ago
    More likely he knew that the pay for a hadoop / bigdata specialist was a lot better than for a mysql specialist, especially 5 years ago.
    - huffmsa 7y ago

manigandham 7y ago

Yes it's still true. Now with stream processing frameworks and infinite cloud storage, you can have a single big machine churn through TBs of data easily.

huffmsa 7y ago

Yes. Very very very few people have data as big as "the internet" and the need for speed (which was the reason Google developed a look of their distributed tooling) that you get with these frameworks. And it really only makes sense to have permanent infrastructure for distributed computing if you're constantly using it. Like if you're constantly rebuilding an index of the internet. Which most people aren't. For occasional reindexing jobs, I've personally had success with Kubernetes. Our customer facing services were deployed with it, so it was trivial to bump up the number of nodes, and then schedule a bunch of workers containers, then roll everything down when the job is complete. No need to learn the ins-and-outs of Hadoop.

threeseed 7y ago
It is very, very, very common for companies to have enough data to need a distributed approach to ETL and especially common if they are doing any machine learning which is most of them. In telcos you have network telemetry data. In supermarkets and retail you have purchase data and often credit card gateway data. In banking and finance you obviously have transaction data. And with Kubernetes you still need a compute framework. Like I don't know the standard in the industry the Hadoop Spark framework.
- huffmsa 7y ago
  A lot of companies think they need machine learning because the management consultants tell them they do (and hey, we can set it up for you!). And those that are "doing machine learning", often aren't doing it very well and have trouble applying it to the business. Note I'm not saying all. Just 99%, like the parents comment referred to. That leftover 1% are the companies you can name off the top of your head. ExxonMobil, Target, Chase, Visa, etc.
throwaway082729 7y ago
Which is probably why these Hadoop companies have very few customers and can probably never grow bigger beyond a certain point.

chungleong 7y ago

99% of businesses don't experience exponential growth. But hope springs eternal in the human breast.

gdulli 7y ago

It's not that 99% of big data problems can't or shouldn't be solved by those tools. It's that hype makes everyone think their problem falls under the category of big data when it might not. Hitting a performance or stability limit with MySQL with an unsophisticated schema/architecture does not mean you have big data. But that's the scenario that's common.