4 ms·

What does ad-hoc queries on extremely large datasets easily and in a reasonable amount of time?

by jshen 15y ago

haberman 15y ago
Google BigQuery (disclaimer: I work on the BigQuery team): https://developers.google.com/bigquery/ https://developers.google.com/bigquery/
- jshen 15y ago
  how is this different from hive on hadoop?
  - haberman 15y ago
    Check out "What Hive is NOT" from this page: http://hive.apache.org/docs/r0.8.1/ http://hive.apache.org/docs/r0.8.1/ "Hive is based on Hadoop, which is a batch processing system. As a result, Hive does not and cannot promise low latencies on queries. [...] Hive queries response times for even the smallest jobs can be of the order of several minutes. However for larger jobs (e.g., jobs processing terabytes of data) in general they may run into hours." BigQuery OTOH can query over billions of rows in seconds. It is based on a system called Dremel (http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/pubs/archive/36632.pdf http://static.googleusercontent.com/external_content/untrust...) that is specifically designed for interactive use, as opposed to being a query language built on top of a batch system.
    - jshen 15y ago
      Awesome, thank you, I have some reading to do :)
- pete_o 15y ago
  Blog post from my brother (we both work at Quantivo) on BigQuery: http://www.quantivo.com/blog/3-big-technical-limitations-googles-bigquery http://www.quantivo.com/blog/3-big-technical-limitations-goo....
  - haberman 15y ago
    This blog post would be more compelling if it made a case for how Quantivo avoids these BigQuery "limitations." Or if it were possible to find any technical detail about Quantivo's architecture without having to give out personal information first (http://www.quantivo.com/resources/wp-behavior-analytics-tech-intro http://www.quantivo.com/resources/wp-behavior-analytics-tech...). BigQuery is based on Dremel, which is an open book: http://research.google.com/pubs/pub36632.html http://research.google.com/pubs/pub36632.html

What does ad-hoc queries on extremely large datasets easily and in a reasonable amount of time?

by jshen 15y ago

What does ad-hoc queries on extremely large datasets easily and in a reasonable amount of time?

Google BigQuery (disclaimer: I work on the BigQuery team): https://developers.google.com/bigquery/ https://developers.google.com/bigquery/

jshen 15y ago
how is this different from hive on hadoop?
- haberman 15y ago
  Check out "What Hive is NOT" from this page: http://hive.apache.org/docs/r0.8.1/ http://hive.apache.org/docs/r0.8.1/ "Hive is based on Hadoop, which is a batch processing system. As a result, Hive does not and cannot promise low latencies on queries. [...] Hive queries response times for even the smallest jobs can be of the order of several minutes. However for larger jobs (e.g., jobs processing terabytes of data) in general they may run into hours." BigQuery OTOH can query over billions of rows in seconds. It is based on a system called Dremel (http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/pubs/archive/36632.pdf http://static.googleusercontent.com/external_content/untrust...) that is specifically designed for interactive use, as opposed to being a query language built on top of a batch system.
  - jshen 15y ago
    Awesome, thank you, I have some reading to do :)
pete_o 15y ago
Blog post from my brother (we both work at Quantivo) on BigQuery: http://www.quantivo.com/blog/3-big-technical-limitations-googles-bigquery http://www.quantivo.com/blog/3-big-technical-limitations-goo....
- haberman 15y ago
  This blog post would be more compelling if it made a case for how Quantivo avoids these BigQuery "limitations." Or if it were possible to find any technical detail about Quantivo's architecture without having to give out personal information first (http://www.quantivo.com/resources/wp-behavior-analytics-tech-intro http://www.quantivo.com/resources/wp-behavior-analytics-tech...). BigQuery is based on Dremel, which is an open book: http://research.google.com/pubs/pub36632.html http://research.google.com/pubs/pub36632.html