4 ms·

I can least say that I've used both a large Hadoop+Hive farm and a moderately sized RedShift cluster on the same large data set, and holy smoke, RedShift is ord

by snorkel 13y ago

I can least say that I've used both a large Hadoop+Hive farm and a moderately sized RedShift cluster on the same large data set, and holy smoke, RedShift is orders of magnitude faster. Results vary by how big the nodes are you allocate to Redshift, and it's not cheap at all, but very impressive.

dman 13y ago
I am working on a related product. Is there some way I could contact you to pick your brain about some things?
spikels 13y ago
This is exactly what you should expect in almost all cases because you are kinda comparing apples and oranges. Hadoop/Hive was not focused on speed but proving it is even possible to do queries reliably on such large datasets. Once this was achieved it immediately became clear the long waits for Hadoop/Hive batch jobs to finish made it impractical for many uses. Presto, Impala, Drill, RedShift, etc were all designed Primarily to address this problem and be much, much faster than Hadoop/Hive so that the data could be queried interactively. All these new projects/products are in a very active competition to find the best way or ways to do this. You should compare RedShift to these other projects rather than Hadoop if speed is an issue for you. Each has it's pluses and minuses depending on the situation.
- aquadrop 13y ago
  Wait, so RedShift (Drill, Impala, Presto) do the same job as Hadoop\Hive, only faster? You only mentioned they addressed the speed, but what's downside? At what cost they achieved their velocity?
  - necubi 13y ago
    Basically, Hive is incredibly inefficient. Hive works by taking apart the HiveQL query and turning it into a series of Hadoop MR steps. A complex hive query may have many such steps, and for each one data must be read from HDFS, processed, and written back to HDFS. For most queries, the vast majority of your time will be spent just doing IO. Hadoop is also very, very slow to start up tasks (> 1 minute), so when you have a lot of them that can come to dominate your total run time. Impala, Drill, etc. avoid all those unnecessary reads and writes by implementing the querying logic directly, rather than by compiling to Map Reduce. Shark [0] is an interesting counterpoint. It takes essentially the same approach as Hive but on Spark instead of Hadoop, and achieves similar or better performance than the more "direct" implementations. [0] http://shark.cs.berkeley.edu http://shark.cs.berkeley.edu
  - spikels 13y ago
    The tradeoffs are both complex and evolving and are different for each project/product. In general Hadoop is more scalable and more flexible but much slower. For example as of today RedShift can hold a maximum of 256 terrabytes of compressed data while Facebook's Hadoop cluster was over 200 Petabytes in late 2012. RedShift only supports limited query and data types and a single index while Hadoop can theoretically handle arbitrary data processing. But if these constraints are acceptable then RedShift will likely be orders of magnitude faster in most cases. Other projects/products will have different tradeoffs but they are almost always faster as this was almost always the primary goal.
    - cjg_ 13y ago
      Where do you get the limit of 256TB compressed from? Amazon Redshift enables you to start with as little as a single 2TB XL node and scale up all the way to a hundred 16TB 8XL nodes for 1.6PB of compressed user data. from http://aws.amazon.com/redshift/features-and-benefits/ http://aws.amazon.com/redshift/features-and-benefits/
      - spikels 13y ago
        My mistake, thank for the correction. The 16 node limit I was familiar with is not a hard limit. You can request more nodes. http://aws.amazon.com/redshift/faqs/#0080 http://aws.amazon.com/redshift/faqs/#0080 It would be interesting to see performance comparisons on these huge datasets. I would expect to see new and interesting problems at that scale.

by snorkel 13y ago

dman 13y ago

I am working on a related product. Is there some way I could contact you to pick your brain about some things?

spikels 13y ago

This is exactly what you should expect in almost all cases because you are kinda comparing apples and oranges. Hadoop/Hive was not focused on speed but proving it is even possible to do queries reliably on such large datasets. Once this was achieved it immediately became clear the long waits for Hadoop/Hive batch jobs to finish made it impractical for many uses. Presto, Impala, Drill, RedShift, etc were all designed Primarily to address this problem and be much, much faster than Hadoop/Hive so that the data could be queried interactively. All these new projects/products are in a very active competition to find the best way or ways to do this. You should compare RedShift to these other projects rather than Hadoop if speed is an issue for you. Each has it's pluses and minuses depending on the situation.

aquadrop 13y ago
Wait, so RedShift (Drill, Impala, Presto) do the same job as Hadoop\Hive, only faster? You only mentioned they addressed the speed, but what's downside? At what cost they achieved their velocity?
- necubi 13y ago
  Basically, Hive is incredibly inefficient. Hive works by taking apart the HiveQL query and turning it into a series of Hadoop MR steps. A complex hive query may have many such steps, and for each one data must be read from HDFS, processed, and written back to HDFS. For most queries, the vast majority of your time will be spent just doing IO. Hadoop is also very, very slow to start up tasks (> 1 minute), so when you have a lot of them that can come to dominate your total run time. Impala, Drill, etc. avoid all those unnecessary reads and writes by implementing the querying logic directly, rather than by compiling to Map Reduce. Shark [0] is an interesting counterpoint. It takes essentially the same approach as Hive but on Spark instead of Hadoop, and achieves similar or better performance than the more "direct" implementations. [0] http://shark.cs.berkeley.edu http://shark.cs.berkeley.edu
- spikels 13y ago
  The tradeoffs are both complex and evolving and are different for each project/product. In general Hadoop is more scalable and more flexible but much slower. For example as of today RedShift can hold a maximum of 256 terrabytes of compressed data while Facebook's Hadoop cluster was over 200 Petabytes in late 2012. RedShift only supports limited query and data types and a single index while Hadoop can theoretically handle arbitrary data processing. But if these constraints are acceptable then RedShift will likely be orders of magnitude faster in most cases. Other projects/products will have different tradeoffs but they are almost always faster as this was almost always the primary goal.
  - cjg_ 13y ago
    Where do you get the limit of 256TB compressed from? Amazon Redshift enables you to start with as little as a single 2TB XL node and scale up all the way to a hundred 16TB 8XL nodes for 1.6PB of compressed user data. from http://aws.amazon.com/redshift/features-and-benefits/ http://aws.amazon.com/redshift/features-and-benefits/
    - spikels 13y ago
      My mistake, thank for the correction. The 16 node limit I was familiar with is not a hard limit. You can request more nodes. http://aws.amazon.com/redshift/faqs/#0080 http://aws.amazon.com/redshift/faqs/#0080 It would be interesting to see performance comparisons on these huge datasets. I would expect to see new and interesting problems at that scale.