4 ms·

This is awesome, thanks for sharing! Redshift looks to be order-of-magnitude faster than Impala or Shark in all the test. Does this mean that once RedShift su

This is awesome, thanks for sharing! Redshift looks to be order-of-magnitude faster than Impala or Shark in all the test. Does this mean that once RedShift supports user-defined functions, there is no competing solution that is any match? (Unless you want to avoid using the cloud)

flyovercountry 13y ago
The file format for hadoop tools is a sequence file. For most queries this is the second slowest format, after text. Rcfile or parquet would be a more interesting benchmark.
monstrado 13y ago
The benchmark above is testing Impala with SequenceFiles compressed with GZIP, against RedShift, which is not a fair comparison. In the "What's next?" section, they say they want to re-do the Impala tests using Parquet, which is a columnar format based on the Dremel whitepaper (http://parquet.io/ http://parquet.io/).
- dude_abides 13y ago
  Ah that makes sense! Looking at their results and how RedShift was so much faster in every scenario, it looked like something was amiss. Is Parquet Cloudera-only like Impala or is it available with vanilla Hadoop?
  - monstrado 13y ago
    Impala isn't technically Cloudera only, it's open source (https://github.com/cloudera/impala https://github.com/cloudera/impala), and other people have gotten it to run on their Hadoop distribution, but since it's developed by Cloudera, it was developed to run on the CDH platform (Hadoop). Parquet was a joint effort between Cloudera and Twitter, and now it's being developed by many other companies. You can use it with Hive, Pig, MapReduce, Cascading, Crunch and I think Apache Drill's first milestone has adopted it as a columnar format as well. Parquet also allows you to use your Avro or Thrift schema (soon Protobuffs) to write Parquet data, too. It's a separate project in the ecosystem and has its own roadmap (https://github.com/Parquet/parquet-mr https://github.com/Parquet/parquet-mr).
- justinerickson 13y ago
  Note that the suggested benchmark (https://amplab.cs.berkeley.edu/benchmark/ https://amplab.cs.berkeley.edu/benchmark/) is a slightly modified version of the Hive Benchmark. Both of these are just 3-4 tables total and 4 very basic queries. I recommend looking at something more realistic (e.g. TPC-DS, TPC-H, etc).

flyovercountry 13y ago

The file format for hadoop tools is a sequence file. For most queries this is the second slowest format, after text. Rcfile or parquet would be a more interesting benchmark.

monstrado 13y ago

The benchmark above is testing Impala with SequenceFiles compressed with GZIP, against RedShift, which is not a fair comparison. In the "What's next?" section, they say they want to re-do the Impala tests using Parquet, which is a columnar format based on the Dremel whitepaper (http://parquet.io/ http://parquet.io/).

dude_abides 13y ago
Ah that makes sense! Looking at their results and how RedShift was so much faster in every scenario, it looked like something was amiss. Is Parquet Cloudera-only like Impala or is it available with vanilla Hadoop?
- monstrado 13y ago
  Impala isn't technically Cloudera only, it's open source (https://github.com/cloudera/impala https://github.com/cloudera/impala), and other people have gotten it to run on their Hadoop distribution, but since it's developed by Cloudera, it was developed to run on the CDH platform (Hadoop). Parquet was a joint effort between Cloudera and Twitter, and now it's being developed by many other companies. You can use it with Hive, Pig, MapReduce, Cascading, Crunch and I think Apache Drill's first milestone has adopted it as a columnar format as well. Parquet also allows you to use your Avro or Thrift schema (soon Protobuffs) to write Parquet data, too. It's a separate project in the ecosystem and has its own roadmap (https://github.com/Parquet/parquet-mr https://github.com/Parquet/parquet-mr).
justinerickson 13y ago
Note that the suggested benchmark (https://amplab.cs.berkeley.edu/benchmark/ https://amplab.cs.berkeley.edu/benchmark/) is a slightly modified version of the Hive Benchmark. Both of these are just 3-4 tables total and 4 very basic queries. I recommend looking at something more realistic (e.g. TPC-DS, TPC-H, etc).