3 ms·
Exactly - encoding is the key. I just gave a presentation last week at the SF Python Meetup about how AdRoll uses a single server to query terabytes of compres
by vtuulos 13y ago
Exactly - encoding is the key.
I just gave a presentation last week at the SF Python Meetup about how AdRoll uses a single server to query terabytes of compressed data with sub-minute latencies, using a system that is implemented in Python:
http://tuulos.github.io/sf-python-meetup-sep-2013 http://tuulos.github.io/sf-python-meetup-sep-2013
This is probably a thousand times less resource intensive than using Hadoop for the same queries.
- pwang 13y agoCompression and out-of-core are not nearly discussed enough in the "big data" circles. The key challenge is efficiently using bandwidth-limited channels, whether it's a network link between nodes, the SATA bus between disk and RAM, or the memory channel between RAM and L3. It's also why we are both building compression into the native storage format for Blaze, and why Blaze is designed to run out-of-core from the start. https://github.com/ContinuumIO/blaze/blob/master/docs/source/persistence.rst https://github.com/ContinuumIO/blaze/blob/master/docs/source...