35 ms·

> Reasons to move to ClickHouse > the [VACUUM] process requires an outrageous amount of time As of Dec 19 2018, Amazon Redshift now runs VACUUM DELETE automat

> Reasons to move to ClickHouse > the [VACUUM] process requires an outrageous amount of time As of Dec 19 2018, Amazon Redshift now runs VACUUM DELETE automatically, and has been made drastically more resource-efficient [1]. > There are no queries in Redshift that take less than a couple of seconds. This is likely due to incorrectly-tuned Workload Management Queues. In addition, as of Aug 8 2018, Redshift automatically enables short query acceleration [2], which should speed up short queries by default without additional tuning. > According to our calculations, deploying ClickHouse on AWS instances with the same resources was exactly half as expensive. You also need to factor in the cost of migration (two full-time specialists, three months effort), and ongoing support effort needed to maintain the now-custom system. Plus, it sounds like some (possibly similar?) cost savings could have been achieved simply optimizing the existing Redshift cluster: > We can’t make a straight comparison about query speed because the data schema changed so much. But many queries sped up simply because less data are read from the disk. Truth be told, we should have made this change in Redshift, but we decided to combine it with our migration to ClickHouse. In short, continue testing assumptions and tracking updated feature-sets on a regular basis- valid reasons to move to ClickHouse yesterday may no longer be valid today, or at some point in the future. Also, don't double down on sunk costs- it might make sense to migrate back to Redshift in the future, if your issues with the system are eventually improved or resolved. Or maybe not, depending on your use-case. [1] https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-redshift-automatic-vacuum/ https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-re... [2] https://aws.amazon.com/about-aws/whats-new/2018/08/amazon-redshift-automatically-enables-short-query-acceleration/ https://aws.amazon.com/about-aws/whats-new/2018/08/amazon-re...

hodgesrm 8y ago
I think the authors were up-front that some of the issues they encounters (for example table optimization) could have been done in Redshift. What I found intriguing about the cost savings was that it was achieved on Amazon as opposed to going on-prem where you can push costs down further especially if the workload is reasonably stable. It's therefore likely that there are still substantial savings that can be squeezed out. That gap gets bigger if you scale to large cluster sizes. In thinking about costs it's also important to remember you can run a data warehouse perfectly well without a cloud management plane. It really depends on the data warehouse complexity, how many other things the team has to manage, and their comfort level with the technology. Redshift--to its credit--has so changed the conversation about data warehouse operation that we tend to forget this. Disclaimer: I work for Altinity and the article came from our blog. We did not write it and the authors are not customers.

hodgesrm 8y ago

I think the authors were up-front that some of the issues they encounters (for example table optimization) could have been done in Redshift. What I found intriguing about the cost savings was that it was achieved on Amazon as opposed to going on-prem where you can push costs down further especially if the workload is reasonably stable. It's therefore likely that there are still substantial savings that can be squeezed out. That gap gets bigger if you scale to large cluster sizes. In thinking about costs it's also important to remember you can run a data warehouse perfectly well without a cloud management plane. It really depends on the data warehouse complexity, how many other things the team has to manage, and their comfort level with the technology. Redshift--to its credit--has so changed the conversation about data warehouse operation that we tend to forget this. Disclaimer: I work for Altinity and the article came from our blog. We did not write it and the authors are not customers.