8 ms·

Google’s Dremel Makes Big Data Look Small

90 points by arunabh 14y ago

d99kris 14y ago
Link to the paper describing Dremel [PDF]: http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/pubs/archive/36632.pdf http://static.googleusercontent.com/external_content/untrust...
- tonfa 14y ago
  And there's already a potentially better column oriented datastore used at Google: http://vldb.org/pvldb/vol5/p1436_alexanderhall_vldb2012.pdf http://vldb.org/pvldb/vol5/p1436_alexanderhall_vldb2012.pdf
jdf 14y ago
Not sure why Cloudera is part of this article, seems like all the attention here should be on Google and the BigQuery team. Here is an open source project similar to Dremel: http://www.itworld.com/big-datahadoop/290026/new-apache-project-will-drill-big-data-near-real-time http://www.itworld.com/big-datahadoop/290026/new-apache-proj...
sbierwagen 14y ago
"Dremel" isn't trademarked by the rotary tool folks?
- jdf 14y ago
  Dremel was Google's internal name. Their public API is called BigQuery: https://developers.google.com/bigquery/ https://developers.google.com/bigquery/
- ggggit 14y ago
  I doubt they registered in a class covering data processing. How many consumers are going to confuse some data processing service from Google with Dremel tools? A trademark, unless perhaps it's famous, does not cover everything under the sun, right?
  - mysterywhiteboy 14y ago
    Yeah - a common test law for 'passing off' in trademark cases is exactly along those lines. It's referred to as "A moron in a hurry"[1] i.e. even a moron in a hurry wouldn't confuse the two brands. [1] http://en.wikipedia.org/wiki/A_moron_in_a_hurry http://en.wikipedia.org/wiki/A_moron_in_a_hurry
    - kator 14y ago
      LOL I read that as "A mormon in a hurry" and briefly thought to myself "What does religion have to do with trademark law"..
- yaroslavvb 14y ago
  It's like "Delta" that makes faucets vs "Delta" that flies planes
- duaneb 14y ago
  What are they going to do, sue google for an internal code name?
  - rachelbythebay 14y ago
    http://en.wikipedia.org/wiki/Power_Macintosh_7100#Codename_Lawsuit http://en.wikipedia.org/wiki/Power_Macintosh_7100#Codename_L...
    - jrockway 14y ago
      ``[While] Sagan lost the suit, Apple engineers complied with his demands anyway, renaming the project "BHA" (for Butt-Head Astronomer). Sagan promptly sued Apple for libel over the new name, claiming that it subjected him to contempt and ridicule, but lost this lawsuit as well.''
      - salmanapk 14y ago
        ... "Apple's third and final code name for the project was "LaW", short for "Lawyers are Wimps" haha Apple has changed so much over the years.
- archangel_one 14y ago
  I suspect the trademark may be genericised; I know what kind of tool a "dremel" is referring to, but didn't realise it was actually a brand name until just now.
- nodata 14y ago
  Trademarks apply to domains. Dremel have a trademark on the Dremel drill, not on anything in the whole world called Dremel.
  - sbierwagen 14y ago
    Ah, that is true, but if I recall, a "fanciful mark" that's entirely made up, like Kodak, Google, or Dremel; has to pass a higher bar. If Dremel produced and sold a power tool named "the Google", then there might be grounds.
    - nodata 14y ago
      Like Coca-Cola.
    - aneth4 14y ago
      It's an internal name not used in commerce so it is not subject to the same liability. Not an expert beyond that, but you can nickname your sister Kleenex without violating a trademark.
    - maxerickson 14y ago
      Wikipedia says that Dremel is named after the founder of the company.
egillie 14y ago
There's an apache version of this in the works: http://www.itworld.com/big-datahadoop/290026/new-apache-project-will-drill-big-data-near-real-time http://www.itworld.com/big-datahadoop/290026/new-apache-proj...
- moondowner 14y ago
  Nice, I see that MapR are initiators of this project, which can be seen as a good thing. Here's the direct link to the project. https://wiki.apache.org/incubator/DrillProposal https://wiki.apache.org/incubator/DrillProposal "Drill is a distributed system for interactive analysis of large-scale datasets, inspired by Google's Dremel."
iskander 14y ago
If I remember correctly, BigQuery only lets you import data via local csv files, uploaded one at a time. That makes importing data sets of relevant size quite a pain.
- bockris 14y ago
  Oddly enough, I found out about this restriction just yesterday. I wanted to play around with Google's ngram dataset. Amazon hosts it for free in S3 making it a no brainer to run in EC2 or EMR. Google doesn't have it available in Google Cloud Storage and I would burn a ton of bandwidth and time just to get it there. http://stackoverflow.com/questions/11990839/is-there-a-way-to-upload-to-cloudstorage-from-a-url http://stackoverflow.com/questions/11990839/is-there-a-way-t... This makes it too difficult to kick the tires on BigQuery IMO. A file that will upload in a reasonable amount of time is more or less a toy dataset and if it's big enough to be a valid test, you will probably spend days trying to get it uploaded.
  - proppy 14y ago
    There is publicdata.samples.trigrams in the public datasets.
    - bockris 14y ago
      Yeah, I left a comment on the SO post noting that and my need for bigrams rather than trigrams.
- sp332 14y ago
  Does Google charge for bandwidth from Google App Engine to this service? It might be cheaper to upload the data to GAE in a smaller form, then transform it to CSV and upload from there.
  - alooPotato 14y ago
    Sadly you must still go through cloud storage to get data from appengine to bigquery. No savings there except maybe time.
kamaal 14y ago
A small note: Its great to see so many great tools coming up to solve the kind of problems which were earlier difficult/impossible to solve. But however please check your bid data use cases many times before using big data tools. Because frankly 'big data' is becoming a just cool must use tool regardless of use cases people have these days. I've even seen data sizes as small as 10 MB being considered for bid data use cases. Often this gets subjected to a monstrously complex architecture for no good reason. Generally most of these cases can be addressed and solved with as simple a tool like sqlite! And all you generally need is something like Perl with sqlite and ability to write simple SQL queries. People get deceived very easily, When they look at GB scale XML files they think that is what big data is. Yet most of that generally and easily goes into a traditional RDBMS. And the performance is generally is in pretty acceptable limits. Mark up eats a lot of space and data size. When converted to flat file structures like csv's, tsv's and then imported to a RDBMS the data sizes are way smaller. I've some times seen an order of 10x difference. Another annoying thing is abuse of NoSQL databases. Perfectly relational data is being de normalized, force fed in NoSQL databases and access data interfaces are generally bad buggy sub implementations of SQL. This is almost like, people who don't understand SQL are condemned to implement it badly.
- fkdjs 14y ago
  "Woops you want to query that column? Better wait 12 hours while we add an index to petabytes of data! Oh shoot, and that column too? Another 12 hours, and you have to use both columns in the where to use the index. " If you are storing xml files for your column-oriented DB then you should be shot. I imagine protocol buffers or something like that might be smarter. IMO. Your other stuff is way off base as well, but I'm lazy.
  - garysieling 14y ago
    He's talking about people mis-applying big data tools for things which are clearly not big data - there are a lot of applications that don't have, or will ever have petabytes of data. The point about XML is comparing the storage size on disk of the same data in an XML files vs the equivalent data in a database, not stored as XML in a database.
majorturd 14y ago
From TFA "We discuss the core ideas in the context of a read-only system, for simplicity. Many Dremel queries are one-pass aggregations; there-fore, we focus on explaining those and use them for experiments in the next section. We defer the discussion of joins, indexing, up-dates, etc. to future work." Really, it takes Dremel multiple SECONDS to complete trivial massively parallelized read queries? It must take hours for an UPDATE or JOIN then. Wake me up when you move past the trivial, until then, enjoy your hair.
- jrockway 14y ago
  Dremel is a query tool, not a database.
  - virmundi 14y ago
    That's true, BUT to query you need to have the ability to perform joins. That is what makes raw MapReduce such a pain and even higher level abstractions slow. I like the idea that Dremel is showing, I even downloaded the Google paper to read tonight, but the Apache implementation needs to have joins otherwise it's not a "query tool".
    - ag3mo 14y ago
      You can join on top of BigQuery with small join tables. https://developers.google.com/bigquery/docs/query-reference#joins https://developers.google.com/bigquery/docs/query-reference#...
      - majorturd 14y ago
        cough Here small means less than 8MB of compressed data cough
- akldfgj 14y ago
  What is the performance of your JOIN system?
peterwwillis 14y ago
Every time I see a paper with Web-Scale in the title I throw up in my mouth a little. So they're using large amounts of nodes for parallel processing of complex queries with specific data segregated to individual nodes. The fuck does that have to do with the world-wide web or scaling the performance of an application on the web?
- akldfgj 14y ago
  Because the data is the traffic of one of the most heavily used sites on the web, and the application is an index of the web. So there's two.
  - peterwwillis 14y ago
    So I should call my dinner utensils Cow-Scale because they can be used on roast beef? Hey, I could call my socket wrenches Navy-Destroyer-Class-Scale because they can be used on boats! Dremel does not have anything to do with the web at all. It's just data processing. You can use it for anything.
    - akldfgj 14y ago
      Dremel was specifically invented to solve the problem of analyzing website logs for a web search engine. They didn't build it to study particle accelerator traces and then throw it at weblogs later.

90 points by arunabh 14y ago

Link to the paper describing Dremel [PDF]: http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/pubs/archive/36632.pdf http://static.googleusercontent.com/external_content/untrust...

tonfa 14y ago
And there's already a potentially better column oriented datastore used at Google: http://vldb.org/pvldb/vol5/p1436_alexanderhall_vldb2012.pdf http://vldb.org/pvldb/vol5/p1436_alexanderhall_vldb2012.pdf

jdf 14y ago

Not sure why Cloudera is part of this article, seems like all the attention here should be on Google and the BigQuery team. Here is an open source project similar to Dremel: http://www.itworld.com/big-datahadoop/290026/new-apache-project-will-drill-big-data-near-real-time http://www.itworld.com/big-datahadoop/290026/new-apache-proj...

sbierwagen 14y ago

"Dremel" isn't trademarked by the rotary tool folks?

jdf 14y ago
Dremel was Google's internal name. Their public API is called BigQuery: https://developers.google.com/bigquery/ https://developers.google.com/bigquery/
ggggit 14y ago
I doubt they registered in a class covering data processing. How many consumers are going to confuse some data processing service from Google with Dremel tools? A trademark, unless perhaps it's famous, does not cover everything under the sun, right?
- mysterywhiteboy 14y ago
  Yeah - a common test law for 'passing off' in trademark cases is exactly along those lines. It's referred to as "A moron in a hurry"[1] i.e. even a moron in a hurry wouldn't confuse the two brands. [1] http://en.wikipedia.org/wiki/A_moron_in_a_hurry http://en.wikipedia.org/wiki/A_moron_in_a_hurry
  - kator 14y ago
    LOL I read that as "A mormon in a hurry" and briefly thought to myself "What does religion have to do with trademark law"..
yaroslavvb 14y ago
It's like "Delta" that makes faucets vs "Delta" that flies planes
duaneb 14y ago
What are they going to do, sue google for an internal code name?
- rachelbythebay 14y ago
  http://en.wikipedia.org/wiki/Power_Macintosh_7100#Codename_Lawsuit http://en.wikipedia.org/wiki/Power_Macintosh_7100#Codename_L...
  - jrockway 14y ago
    ``[While] Sagan lost the suit, Apple engineers complied with his demands anyway, renaming the project "BHA" (for Butt-Head Astronomer). Sagan promptly sued Apple for libel over the new name, claiming that it subjected him to contempt and ridicule, but lost this lawsuit as well.''
    - salmanapk 14y ago
      ... "Apple's third and final code name for the project was "LaW", short for "Lawyers are Wimps" haha Apple has changed so much over the years.
archangel_one 14y ago
I suspect the trademark may be genericised; I know what kind of tool a "dremel" is referring to, but didn't realise it was actually a brand name until just now.
nodata 14y ago
Trademarks apply to domains. Dremel have a trademark on the Dremel drill, not on anything in the whole world called Dremel.
- sbierwagen 14y ago
  Ah, that is true, but if I recall, a "fanciful mark" that's entirely made up, like Kodak, Google, or Dremel; has to pass a higher bar. If Dremel produced and sold a power tool named "the Google", then there might be grounds.
  - nodata 14y ago
    Like Coca-Cola.
  - aneth4 14y ago
    It's an internal name not used in commerce so it is not subject to the same liability. Not an expert beyond that, but you can nickname your sister Kleenex without violating a trademark.
  - maxerickson 14y ago
    Wikipedia says that Dremel is named after the founder of the company.

egillie 14y ago

There's an apache version of this in the works: http://www.itworld.com/big-datahadoop/290026/new-apache-project-will-drill-big-data-near-real-time http://www.itworld.com/big-datahadoop/290026/new-apache-proj...

moondowner 14y ago
Nice, I see that MapR are initiators of this project, which can be seen as a good thing. Here's the direct link to the project. https://wiki.apache.org/incubator/DrillProposal https://wiki.apache.org/incubator/DrillProposal "Drill is a distributed system for interactive analysis of large-scale datasets, inspired by Google's Dremel."

iskander 14y ago

If I remember correctly, BigQuery only lets you import data via local csv files, uploaded one at a time. That makes importing data sets of relevant size quite a pain.

bockris 14y ago
Oddly enough, I found out about this restriction just yesterday. I wanted to play around with Google's ngram dataset. Amazon hosts it for free in S3 making it a no brainer to run in EC2 or EMR. Google doesn't have it available in Google Cloud Storage and I would burn a ton of bandwidth and time just to get it there. http://stackoverflow.com/questions/11990839/is-there-a-way-to-upload-to-cloudstorage-from-a-url http://stackoverflow.com/questions/11990839/is-there-a-way-t... This makes it too difficult to kick the tires on BigQuery IMO. A file that will upload in a reasonable amount of time is more or less a toy dataset and if it's big enough to be a valid test, you will probably spend days trying to get it uploaded.
- proppy 14y ago
  There is publicdata.samples.trigrams in the public datasets.
  - bockris 14y ago
    Yeah, I left a comment on the SO post noting that and my need for bigrams rather than trigrams.
sp332 14y ago
Does Google charge for bandwidth from Google App Engine to this service? It might be cheaper to upload the data to GAE in a smaller form, then transform it to CSV and upload from there.
- alooPotato 14y ago
  Sadly you must still go through cloud storage to get data from appengine to bigquery. No savings there except maybe time.

kamaal 14y ago

A small note: Its great to see so many great tools coming up to solve the kind of problems which were earlier difficult/impossible to solve. But however please check your bid data use cases many times before using big data tools. Because frankly 'big data' is becoming a just cool must use tool regardless of use cases people have these days. I've even seen data sizes as small as 10 MB being considered for bid data use cases. Often this gets subjected to a monstrously complex architecture for no good reason. Generally most of these cases can be addressed and solved with as simple a tool like sqlite! And all you generally need is something like Perl with sqlite and ability to write simple SQL queries. People get deceived very easily, When they look at GB scale XML files they think that is what big data is. Yet most of that generally and easily goes into a traditional RDBMS. And the performance is generally is in pretty acceptable limits. Mark up eats a lot of space and data size. When converted to flat file structures like csv's, tsv's and then imported to a RDBMS the data sizes are way smaller. I've some times seen an order of 10x difference. Another annoying thing is abuse of NoSQL databases. Perfectly relational data is being de normalized, force fed in NoSQL databases and access data interfaces are generally bad buggy sub implementations of SQL. This is almost like, people who don't understand SQL are condemned to implement it badly.

fkdjs 14y ago
"Woops you want to query that column? Better wait 12 hours while we add an index to petabytes of data! Oh shoot, and that column too? Another 12 hours, and you have to use both columns in the where to use the index. " If you are storing xml files for your column-oriented DB then you should be shot. I imagine protocol buffers or something like that might be smarter. IMO. Your other stuff is way off base as well, but I'm lazy.
- garysieling 14y ago
  He's talking about people mis-applying big data tools for things which are clearly not big data - there are a lot of applications that don't have, or will ever have petabytes of data. The point about XML is comparing the storage size on disk of the same data in an XML files vs the equivalent data in a database, not stored as XML in a database.

majorturd 14y ago

From TFA "We discuss the core ideas in the context of a read-only system, for simplicity. Many Dremel queries are one-pass aggregations; there-fore, we focus on explaining those and use them for experiments in the next section. We defer the discussion of joins, indexing, up-dates, etc. to future work." Really, it takes Dremel multiple SECONDS to complete trivial massively parallelized read queries? It must take hours for an UPDATE or JOIN then. Wake me up when you move past the trivial, until then, enjoy your hair.

jrockway 14y ago
Dremel is a query tool, not a database.
- virmundi 14y ago
  That's true, BUT to query you need to have the ability to perform joins. That is what makes raw MapReduce such a pain and even higher level abstractions slow. I like the idea that Dremel is showing, I even downloaded the Google paper to read tonight, but the Apache implementation needs to have joins otherwise it's not a "query tool".
  - ag3mo 14y ago
    You can join on top of BigQuery with small join tables. https://developers.google.com/bigquery/docs/query-reference#joins https://developers.google.com/bigquery/docs/query-reference#...
    - majorturd 14y ago
      cough Here small means less than 8MB of compressed data cough
akldfgj 14y ago
What is the performance of your JOIN system?

peterwwillis 14y ago

Every time I see a paper with Web-Scale in the title I throw up in my mouth a little. So they're using large amounts of nodes for parallel processing of complex queries with specific data segregated to individual nodes. The fuck does that have to do with the world-wide web or scaling the performance of an application on the web?

akldfgj 14y ago
Because the data is the traffic of one of the most heavily used sites on the web, and the application is an index of the web. So there's two.
- peterwwillis 14y ago
  So I should call my dinner utensils Cow-Scale because they can be used on roast beef? Hey, I could call my socket wrenches Navy-Destroyer-Class-Scale because they can be used on boats! Dremel does not have anything to do with the web at all. It's just data processing. You can use it for anything.
  - akldfgj 14y ago
    Dremel was specifically invented to solve the problem of analyzing website logs for a web search engine. They didn't build it to study particle accelerator traces and then throw it at weblogs later.