2 ms·
Your comment is a little bit perplexing to me because I don't really recognize much difference between composing relational operations and map / reduce operatio
by barrkel 9y ago
Your comment is a little bit perplexing to me because I don't really recognize much difference between composing relational operations and map / reduce operations. SQL `select` is map, `group by` is reduce, `where` is a flatMap to zero or one items, and `join` is a kind of specialized flatMap to zero or more items. You have to consider IO quantity and throughput (both disk and memory) to get decent performance either way.
For example, if you're writing straight-up SQL, the fewer columns you project out before you start sorting the better off you are, it's better to join stuff after you've done your sort than the reverse because it means physically less data shuffling.
- nothrabannosir 9y agoIn redshift, I control the size of nodes. That means I can brute force my way through an expensive operation by increasing the memory capacity of the nodes. In BigQuery, you play by the rules or you’re out. It forces you to think in map reduce, because there really is nothing else under the hood. Also, you get immediate parallelisation across O(n) nodes. Again; not in redshift. These are crucially different to regular DBs. They’re both semantically sql, nobody denies that, but they describe different underlying models. E.g.: in BigQuery, you can’t sort your entire column, even if you do other stuff afterwards. That makes sense in a map reduce system, but not in a “normal” DB.
- barrkel 9y agoThanks for the clarification.