5 ms·

Cloudera announces real-time query engine for Hadoop

56 points by TallGuyShort 14y ago

hobbyist 14y ago
So, what is the key innovation that changes hadoop from batch processing to real time. Is it similar to google's dremel?
- TallGuyShort 14y ago
  This doesn't change Hadoop from batch processing to real time - it's a new query engine that uses the same data sources/formats as Hadoop and the same interfaces as Hive. So yeah, similar idea to Google Dremel / F1.
  - monstrado 14y ago
    You're right that the data is still stored in HDFS, and you can access the data that was originally accessed in Hive. However, because of the way Impala works, it does in fact provide a real-time query interface for data that is sitting in Hadoop (HDFS). The back-end is written in C++ and accesses/processes data much differently than MapReduce which allows for some pretty awesome performance numbers :-)
  - ryanpers 14y ago
    It isn't like F1 at all... F1 is a multi-datacenter, transactional datastore and SQL, that google uses to replace mysql. This aint nothing like that.
- monstrado 14y ago
  Correct, the technology is based on Google Dremel.
  - jbellis 14y ago
    My reading was that columnar storage was a critical piece of Dremel. "Accessing the same data as Hive" doesn't sound like Impala is doing that. Is that optional then or is this just a query engine?
    - monstrado 14y ago
      I can't say for sure, but I believe Doug Cutting's Trevni format is already in the beta but it's a slow version. Impala is initially focusing on architecture, I'm sure columnar format will have attention eventually. https://github.com/cutting/trevni https://github.com/cutting/trevni
  - hobbyist 14y ago
    But google's dremel actually doesnt use Map-Reduce at all. So is this using just the HDFS and a new layer along side map-r that is using the HDFS?
    - qwertymaniac 14y ago
      Impala does not seem to be using MR at all, see note: https://ccp.cloudera.com/display/IMPALA10BETADOC/Impala+Frequently+Asked+Questions#ImpalaFrequentlyAskedQuestions-HowdoesImpalacomparetoHive%2FPig%3FCanIdotransformsoraddnewfunctionality%3F https://ccp.cloudera.com/display/IMPALA10BETADOC/Impala+Freq...
jgrahamc 14y ago
This looks very interesting. I've got multiple PB of log files that I'd like to be able to do real-time and batch processing on. Log files are being generated at a small number of Gbps. Anyone else got that much data and have a good solution?
- untitledwiz 14y ago
  I don't think anyone has managed to get interactive speeds (~seconds to max couple of minutes) at the PB scale.
- benbjohnson 14y ago
  What kind of analysis are you trying to do on that dataset? Log files can be verbose as well. Do you need all the data in the logs or can you cut down the data in each row to only what you need?
pella 14y ago
"Cloudera Impala: Real-Time Queries in Apache Hadoop, For Real" http://blog.cloudera.com/blog/2012/10/cloudera-impala-real-time-queries-in-apache-hadoop-for-real/ http://blog.cloudera.com/blog/2012/10/cloudera-impala-real-t... Impala source code: https://github.com/cloudera/impala https://github.com/cloudera/impala Impala documentation can be found here: https://ccp.cloudera.com/display/IMPALA10BETADOC/Cloudera+Impala+1.0+Beta+Documentation https://ccp.cloudera.com/display/IMPALA10BETADOC/Cloudera+Im...
benbjohnson 14y ago
Metamarkets also open sourced their data store today. High performance, distributed query database. http://metamarkets.com/category/technology/druid/ http://metamarkets.com/category/technology/druid/
nphase 14y ago
Interesting, I wonder how this will be different than Apache Drill, championed by Ted Dunning (MapR)
- rmnoon 14y ago
  They're both Dremel clones, but this one seems to be a lot more complete (as Drill is just getting started).
turingbook 14y ago
I noted that Marcel Kornacker, the architect of Impala, was the lead developer for the query engine of Google’s F1 project. In the end of the article: http://blog.cloudera.com/blog/2012/10/cloudera-impala-real-time-queries-in-apache-hadoop-for-real/ http://blog.cloudera.com/blog/2012/10/cloudera-impala-real-t...

56 points by TallGuyShort 14y ago

So, what is the key innovation that changes hadoop from batch processing to real time. Is it similar to google's dremel?

TallGuyShort 14y ago
This doesn't change Hadoop from batch processing to real time - it's a new query engine that uses the same data sources/formats as Hadoop and the same interfaces as Hive. So yeah, similar idea to Google Dremel / F1.
- monstrado 14y ago
  You're right that the data is still stored in HDFS, and you can access the data that was originally accessed in Hive. However, because of the way Impala works, it does in fact provide a real-time query interface for data that is sitting in Hadoop (HDFS). The back-end is written in C++ and accesses/processes data much differently than MapReduce which allows for some pretty awesome performance numbers :-)
- ryanpers 14y ago
  It isn't like F1 at all... F1 is a multi-datacenter, transactional datastore and SQL, that google uses to replace mysql. This aint nothing like that.
monstrado 14y ago
Correct, the technology is based on Google Dremel.
- jbellis 14y ago
  My reading was that columnar storage was a critical piece of Dremel. "Accessing the same data as Hive" doesn't sound like Impala is doing that. Is that optional then or is this just a query engine?
  - monstrado 14y ago
    I can't say for sure, but I believe Doug Cutting's Trevni format is already in the beta but it's a slow version. Impala is initially focusing on architecture, I'm sure columnar format will have attention eventually. https://github.com/cutting/trevni https://github.com/cutting/trevni
- hobbyist 14y ago
  But google's dremel actually doesnt use Map-Reduce at all. So is this using just the HDFS and a new layer along side map-r that is using the HDFS?
  - qwertymaniac 14y ago
    Impala does not seem to be using MR at all, see note: https://ccp.cloudera.com/display/IMPALA10BETADOC/Impala+Frequently+Asked+Questions#ImpalaFrequentlyAskedQuestions-HowdoesImpalacomparetoHive%2FPig%3FCanIdotransformsoraddnewfunctionality%3F https://ccp.cloudera.com/display/IMPALA10BETADOC/Impala+Freq...

jgrahamc 14y ago

This looks very interesting. I've got multiple PB of log files that I'd like to be able to do real-time and batch processing on. Log files are being generated at a small number of Gbps. Anyone else got that much data and have a good solution?

untitledwiz 14y ago
I don't think anyone has managed to get interactive speeds (~seconds to max couple of minutes) at the PB scale.
benbjohnson 14y ago
What kind of analysis are you trying to do on that dataset? Log files can be verbose as well. Do you need all the data in the logs or can you cut down the data in each row to only what you need?

pella 14y ago

"Cloudera Impala: Real-Time Queries in Apache Hadoop, For Real" http://blog.cloudera.com/blog/2012/10/cloudera-impala-real-time-queries-in-apache-hadoop-for-real/ http://blog.cloudera.com/blog/2012/10/cloudera-impala-real-t... Impala source code: https://github.com/cloudera/impala https://github.com/cloudera/impala Impala documentation can be found here: https://ccp.cloudera.com/display/IMPALA10BETADOC/Cloudera+Impala+1.0+Beta+Documentation https://ccp.cloudera.com/display/IMPALA10BETADOC/Cloudera+Im...

benbjohnson 14y ago

Metamarkets also open sourced their data store today. High performance, distributed query database. http://metamarkets.com/category/technology/druid/ http://metamarkets.com/category/technology/druid/

nphase 14y ago

Interesting, I wonder how this will be different than Apache Drill, championed by Ted Dunning (MapR)

rmnoon 14y ago
They're both Dremel clones, but this one seems to be a lot more complete (as Drill is just getting started).

turingbook 14y ago

I noted that Marcel Kornacker, the architect of Impala, was the lead developer for the query engine of Google’s F1 project. In the end of the article: http://blog.cloudera.com/blog/2012/10/cloudera-impala-real-time-queries-in-apache-hadoop-for-real/ http://blog.cloudera.com/blog/2012/10/cloudera-impala-real-t...