3 ms·
This looks amazing! Hadley, Wes, what are your thoughts on how to implement compression? I recall some open source columnar datastores (e.g. infobright) that
by zzleeper 11y ago
This looks amazing!
Hadley, Wes, what are your thoughts on how to implement compression?
I recall some open source columnar datastores (e.g. infobright) that achieved very VERY fast compression rates with just a few tricks:
https://news.ycombinator.com/item?id=8354416 https://news.ycombinator.com/item?id=8354416
In particular, compression is extremely fast for columnar datastores (its the same type one after the other). Since a lot of times the data is sorted by some ID (date, individual, etc.), you should see large improvements in both speed and disk space.
- hadley 11y agoCompression is on the long term to do list - Wes knows a lot more about it than me.