4 ms·

Reasons Not to Use Hadoop for Analytics

7 points by bsg75 15y ago

kylemaxwell 15y ago
Hopefully, anyone pondering a new project would investigate Hadoop (or any other tool) before choosing it. Not built for ad hoc queries? Yeah, that's fairly open. Requires programming ability? Well, yes, that's the intent: it's not a studio-type environment. You need staff to keep it running? If you have thousands of nodes and need them, then you probably should already have the resources for an engineer or two. Maybe I'm missing something, but it sounds like the author tried to fit a tool to his needs rather than choosing a tool that already does so.
- bsg75 15y ago
  A mistake many groups tend to make.
spidaman 15y ago
Hadoop is good at what it's designed for (data processing parallelization built by programmers) and bad at what it's not (ad hoc queries by people who are not programmers). Move along here folks, nothing to see here.
- bunderbunder 15y ago
  I'd constrain the description of what it's good at even further. The kind of data processing parallelization it's designed to support is map/reduce. That's only one out of a wide array of possible ways to organize a distributed processing job, and not all tasks are well-suited to it.
jshen 15y ago
What does ad-hoc queries on extremely large datasets easily and in a reasonable amount of time?
- haberman 15y ago
  Google BigQuery (disclaimer: I work on the BigQuery team): https://developers.google.com/bigquery/ https://developers.google.com/bigquery/
  - jshen 15y ago
    how is this different from hive on hadoop?
    - haberman 15y ago
      Check out "What Hive is NOT" from this page: http://hive.apache.org/docs/r0.8.1/ http://hive.apache.org/docs/r0.8.1/ "Hive is based on Hadoop, which is a batch processing system. As a result, Hive does not and cannot promise low latencies on queries. [...] Hive queries response times for even the smallest jobs can be of the order of several minutes. However for larger jobs (e.g., jobs processing terabytes of data) in general they may run into hours." BigQuery OTOH can query over billions of rows in seconds. It is based on a system called Dremel (http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/pubs/archive/36632.pdf http://static.googleusercontent.com/external_content/untrust...) that is specifically designed for interactive use, as opposed to being a query language built on top of a batch system.
      - jshen 15y ago
        Awesome, thank you, I have some reading to do :)
  - pete_o 15y ago
    Blog post from my brother (we both work at Quantivo) on BigQuery: http://www.quantivo.com/blog/3-big-technical-limitations-googles-bigquery http://www.quantivo.com/blog/3-big-technical-limitations-goo....
    - haberman 15y ago
      This blog post would be more compelling if it made a case for how Quantivo avoids these BigQuery "limitations." Or if it were possible to find any technical detail about Quantivo's architecture without having to give out personal information first (http://www.quantivo.com/resources/wp-behavior-analytics-tech-intro http://www.quantivo.com/resources/wp-behavior-analytics-tech...). BigQuery is based on Dremel, which is an open book: http://research.google.com/pubs/pub36632.html http://research.google.com/pubs/pub36632.html
greesil 15y ago
Sounds like an ad for Quantivo. You can pry MapReduce out of my cold, dead hands.
pete_o 15y ago
I work at Quantivo and we didn't post this but appreciate the shout out. Joe added a technical whitepaper written by another Quantivo staffer to his blog post, check it out.

7 points by bsg75 15y ago

Hopefully, anyone pondering a new project would investigate Hadoop (or any other tool) before choosing it. Not built for ad hoc queries? Yeah, that's fairly open. Requires programming ability? Well, yes, that's the intent: it's not a studio-type environment. You need staff to keep it running? If you have thousands of nodes and need them, then you probably should already have the resources for an engineer or two. Maybe I'm missing something, but it sounds like the author tried to fit a tool to his needs rather than choosing a tool that already does so.

bsg75 15y ago
A mistake many groups tend to make.

spidaman 15y ago

Hadoop is good at what it's designed for (data processing parallelization built by programmers) and bad at what it's not (ad hoc queries by people who are not programmers). Move along here folks, nothing to see here.

bunderbunder 15y ago
I'd constrain the description of what it's good at even further. The kind of data processing parallelization it's designed to support is map/reduce. That's only one out of a wide array of possible ways to organize a distributed processing job, and not all tasks are well-suited to it.

jshen 15y ago

What does ad-hoc queries on extremely large datasets easily and in a reasonable amount of time?

haberman 15y ago
Google BigQuery (disclaimer: I work on the BigQuery team): https://developers.google.com/bigquery/ https://developers.google.com/bigquery/
- jshen 15y ago
  how is this different from hive on hadoop?
  - haberman 15y ago
    Check out "What Hive is NOT" from this page: http://hive.apache.org/docs/r0.8.1/ http://hive.apache.org/docs/r0.8.1/ "Hive is based on Hadoop, which is a batch processing system. As a result, Hive does not and cannot promise low latencies on queries. [...] Hive queries response times for even the smallest jobs can be of the order of several minutes. However for larger jobs (e.g., jobs processing terabytes of data) in general they may run into hours." BigQuery OTOH can query over billions of rows in seconds. It is based on a system called Dremel (http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/pubs/archive/36632.pdf http://static.googleusercontent.com/external_content/untrust...) that is specifically designed for interactive use, as opposed to being a query language built on top of a batch system.
    - jshen 15y ago
      Awesome, thank you, I have some reading to do :)
- pete_o 15y ago
  Blog post from my brother (we both work at Quantivo) on BigQuery: http://www.quantivo.com/blog/3-big-technical-limitations-googles-bigquery http://www.quantivo.com/blog/3-big-technical-limitations-goo....
  - haberman 15y ago
    This blog post would be more compelling if it made a case for how Quantivo avoids these BigQuery "limitations." Or if it were possible to find any technical detail about Quantivo's architecture without having to give out personal information first (http://www.quantivo.com/resources/wp-behavior-analytics-tech-intro http://www.quantivo.com/resources/wp-behavior-analytics-tech...). BigQuery is based on Dremel, which is an open book: http://research.google.com/pubs/pub36632.html http://research.google.com/pubs/pub36632.html

greesil 15y ago

Sounds like an ad for Quantivo. You can pry MapReduce out of my cold, dead hands.

pete_o 15y ago

I work at Quantivo and we didn't post this but appreciate the shout out. Joe added a technical whitepaper written by another Quantivo staffer to his blog post, check it out.