3 ms·
It would be great if AWS would publish the code necessary to reproduce this benchmark. They mentioned applying sort and dist keys; it matters A LOT how these ar
by georgewfraser 10y ago
It would be great if AWS would publish the code necessary to reproduce this benchmark. They mentioned applying sort and dist keys; it matters A LOT how these are chosen. For example, if you know in advance how you're going to join, you can distribute both tables on the foreign key column so the join is a local operation. This isn't always an option in real data warehouses, where you do lots of different joins on the same tables. So it represents an opportunity to "cheat" the benchmark. It's also important whether they used BigQuery partitioned tables, which are the equivalent of SORTKEY in some ways.
Having said all this, my company is a fully-managed data pipeline that supports both data warehouses, and we find they both work extremely well for our customers real datasets.
- maslam 10y agoGeorge - fan of Fivetran. What, in your opinion, are common cases where customers choose BQ over Redshift? Or is it really a function of which cloud you're on?
- georgewfraser 10y agoIf you are already familiar with one, that is actually a really good reason to use it. If you store a lot of data but rarely query it, BQ will be WAY cheaper because you pay for compute and storage separately. On the other hand, you will never see sub-second queries in BQ. Another option absolutely worth checking out is Snowflake, which compromises between the approach of BQ and Redshift in a very interesting way.