3 ms·

What about query latency? 5-8 seconds?

by alecco 14y ago

nl 14y ago
5-8 seconds isn't a problem at all. It's Analytics, not an online database for website authentication or something. It competes with datawarehouse solutions where the typical reporting models involves submitting a job, waiting an hour and then getting a report. I'd expect a lot of the most useful applications running on this will use queries that take many minutes (if not hours) to complete.
- mark_integerdsv 14y ago
  "...waiting an hour..." Uhm... Ke? I work in Business Intelligence and anything over 5 seconds to return a typical (eg: year to date/daily sales) report is unacceptable by my standards. Could you perhaps go I to more detail as to the definition of 'job' in your post? Are we talking about giant year end actuarial runs here or something like that?
  - igrigorik 14y ago
    If you know the exact dimensions you're going to query, then you can construct a data cube to fit your exact scenario up front, and indeed, those will return fast. BigQuery allows you to explore the data without the pain of setting up your star schemas upfront. Think of it this way: you've got XXX Tb of log data, and you have a new question you want to ask it. At this point, you're heading back to Map-Reduce, or Pig/Hive, etc. BigQuery is based on Google's Dremel (check the paper, great read), and has all the operational + performance learnings from wide deployment at Google. Type in your "SQL like" query and BQ takes care of all the rest.. within an order of seconds. tl;dr: you're comparing apples and oranges.
  - nl 14y ago
    No, I'm talking about ad-hoc reports over large datasets. For example, find the original source of all users who bought more than 6 different items over any 6 week period during the last 5 years, then find every web page loaded by IPs form the same subnet as those users in the same time periods.
    - mark_integerdsv 14y ago
      Got it, makes sense.
- alecco 14y ago
  OP ran "simple SELECT queries on the sample Wikipedia dataset", not complex queries.
  - nl 14y ago
    Yeah. That's like complaining a F-22 take a lot longer to start than a motorbike. It's completely true in every way, and yet not something who uses either of them care about. Simple selects from Wikipedia are a nice demo for this, nothing more.

What about query latency? 5-8 seconds?

by alecco 14y ago

What about query latency? 5-8 seconds?

nl 14y ago

5-8 seconds isn't a problem at all. It's Analytics, not an online database for website authentication or something. It competes with datawarehouse solutions where the typical reporting models involves submitting a job, waiting an hour and then getting a report. I'd expect a lot of the most useful applications running on this will use queries that take many minutes (if not hours) to complete.

mark_integerdsv 14y ago
"...waiting an hour..." Uhm... Ke? I work in Business Intelligence and anything over 5 seconds to return a typical (eg: year to date/daily sales) report is unacceptable by my standards. Could you perhaps go I to more detail as to the definition of 'job' in your post? Are we talking about giant year end actuarial runs here or something like that?
- igrigorik 14y ago
  If you know the exact dimensions you're going to query, then you can construct a data cube to fit your exact scenario up front, and indeed, those will return fast. BigQuery allows you to explore the data without the pain of setting up your star schemas upfront. Think of it this way: you've got XXX Tb of log data, and you have a new question you want to ask it. At this point, you're heading back to Map-Reduce, or Pig/Hive, etc. BigQuery is based on Google's Dremel (check the paper, great read), and has all the operational + performance learnings from wide deployment at Google. Type in your "SQL like" query and BQ takes care of all the rest.. within an order of seconds. tl;dr: you're comparing apples and oranges.
- nl 14y ago
  No, I'm talking about ad-hoc reports over large datasets. For example, find the original source of all users who bought more than 6 different items over any 6 week period during the last 5 years, then find every web page loaded by IPs form the same subnet as those users in the same time periods.
  - mark_integerdsv 14y ago
    Got it, makes sense.
alecco 14y ago
OP ran "simple SELECT queries on the sample Wikipedia dataset", not complex queries.
- nl 14y ago
  Yeah. That's like complaining a F-22 take a lot longer to start than a motorbike. It's completely true in every way, and yet not something who uses either of them care about. Simple selects from Wikipedia are a nice demo for this, nothing more.