4 ms·

So, what is the key innovation that changes hadoop from batch processing to real time. Is it similar to google's dremel?

TallGuyShort 14y ago
This doesn't change Hadoop from batch processing to real time - it's a new query engine that uses the same data sources/formats as Hadoop and the same interfaces as Hive. So yeah, similar idea to Google Dremel / F1.
- monstrado 14y ago
  You're right that the data is still stored in HDFS, and you can access the data that was originally accessed in Hive. However, because of the way Impala works, it does in fact provide a real-time query interface for data that is sitting in Hadoop (HDFS). The back-end is written in C++ and accesses/processes data much differently than MapReduce which allows for some pretty awesome performance numbers :-)
- ryanpers 14y ago
  It isn't like F1 at all... F1 is a multi-datacenter, transactional datastore and SQL, that google uses to replace mysql. This aint nothing like that.
monstrado 14y ago
Correct, the technology is based on Google Dremel.
- jbellis 14y ago
  My reading was that columnar storage was a critical piece of Dremel. "Accessing the same data as Hive" doesn't sound like Impala is doing that. Is that optional then or is this just a query engine?
  - monstrado 14y ago
    I can't say for sure, but I believe Doug Cutting's Trevni format is already in the beta but it's a slow version. Impala is initially focusing on architecture, I'm sure columnar format will have attention eventually. https://github.com/cutting/trevni https://github.com/cutting/trevni
- hobbyist 14y ago
  But google's dremel actually doesnt use Map-Reduce at all. So is this using just the HDFS and a new layer along side map-r that is using the HDFS?
  - qwertymaniac 14y ago
    Impala does not seem to be using MR at all, see note: https://ccp.cloudera.com/display/IMPALA10BETADOC/Impala+Frequently+Asked+Questions#ImpalaFrequentlyAskedQuestions-HowdoesImpalacomparetoHive%2FPig%3FCanIdotransformsoraddnewfunctionality%3F https://ccp.cloudera.com/display/IMPALA10BETADOC/Impala+Freq...

TallGuyShort 14y ago

This doesn't change Hadoop from batch processing to real time - it's a new query engine that uses the same data sources/formats as Hadoop and the same interfaces as Hive. So yeah, similar idea to Google Dremel / F1.

monstrado 14y ago
You're right that the data is still stored in HDFS, and you can access the data that was originally accessed in Hive. However, because of the way Impala works, it does in fact provide a real-time query interface for data that is sitting in Hadoop (HDFS). The back-end is written in C++ and accesses/processes data much differently than MapReduce which allows for some pretty awesome performance numbers :-)
ryanpers 14y ago
It isn't like F1 at all... F1 is a multi-datacenter, transactional datastore and SQL, that google uses to replace mysql. This aint nothing like that.

monstrado 14y ago

Correct, the technology is based on Google Dremel.

jbellis 14y ago
My reading was that columnar storage was a critical piece of Dremel. "Accessing the same data as Hive" doesn't sound like Impala is doing that. Is that optional then or is this just a query engine?
- monstrado 14y ago
  I can't say for sure, but I believe Doug Cutting's Trevni format is already in the beta but it's a slow version. Impala is initially focusing on architecture, I'm sure columnar format will have attention eventually. https://github.com/cutting/trevni https://github.com/cutting/trevni
hobbyist 14y ago
But google's dremel actually doesnt use Map-Reduce at all. So is this using just the HDFS and a new layer along side map-r that is using the HDFS?
- qwertymaniac 14y ago
  Impala does not seem to be using MR at all, see note: https://ccp.cloudera.com/display/IMPALA10BETADOC/Impala+Frequently+Asked+Questions#ImpalaFrequentlyAskedQuestions-HowdoesImpalacomparetoHive%2FPig%3FCanIdotransformsoraddnewfunctionality%3F https://ccp.cloudera.com/display/IMPALA10BETADOC/Impala+Freq...