Compression and out-of-core are not nearly discussed enough in the "big data" circles. The key challenge is efficiently using bandwidth-limited channels, wheth

Compression and out-of-core are not nearly discussed enough in the "big data" circles. The key challenge is efficiently using bandwidth-limited channels, whether it's a network link between nodes, the SATA bus between disk and RAM, or the memory channel between RAM and L3. It's also why we are both building compression into the native storage format for Blaze, and why Blaze is designed to run out-of-core from the start. https://github.com/ContinuumIO/blaze/blob/master/docs/source/persistence.rst https://github.com/ContinuumIO/blaze/blob/master/docs/source...