2 ms·
You can (and often should) have indices in a map-reduce situation as well - you just build them in an explicit, visible way. But in most of the relevant use cas
by lmm 5y ago
You can (and often should) have indices in a map-reduce situation as well - you just build them in an explicit, visible way. But in most of the relevant use cases you're doing some kind of aggregation over the whole table, so indices don't help any.
- tremon 5y agoAnd if your primary use-case is column-wise aggregation over the whole table, in SQL you'd use a (compressed) column store rather than a row store as your table storage method.
- 988747 5y agoTo be fair, Parquet, which is commonly used in Big Data solutions is a column store format. So, once you normalize your data and save it as Parquet you can have efficient column-wise aggregation - but that assumes some preprocessing step.