4 ms·

99% of the cases I have seen where people have been working with tables that are in the 5+ TB range for analysis, there is some obvious way to compress the data

99% of the cases I have seen where people have been working with tables that are in the 5+ TB range for analysis, there is some obvious way to compress the data that they have overlooked. Most analysts find some way to aggregate a dataset once, then do actual work on that aggregated dataset, rather than the raw data. In geospatial analytics, for example, a trillion records can be aggregated down to census blocks/block groups so you only have a few million records to deal with. The initial aggregation often takes several days, but after that you can calculate most things in a few seconds with reasonable hardware.

pge 13y ago
In addition to compression, let's not forget sampling. For a lot of problems a random subset of the data will give you a statistically meaningful answer with sufficient precision. It seems like the rise of "big data" has led to the assumption that all queries have to be run against the entire dataset.
- disgruntledphd2 13y ago
  True. The issue with "Big Data" is that sometimes, especially when you need to produce personalised recommendations, the sampling doesn't cut it (or at least produces sub-optimal results).
vtuulos 13y ago
Exactly - encoding is the key. I just gave a presentation last week at the SF Python Meetup about how AdRoll uses a single server to query terabytes of compressed data with sub-minute latencies, using a system that is implemented in Python: http://tuulos.github.io/sf-python-meetup-sep-2013 http://tuulos.github.io/sf-python-meetup-sep-2013 This is probably a thousand times less resource intensive than using Hadoop for the same queries.
- pwang 13y ago
  Compression and out-of-core are not nearly discussed enough in the "big data" circles. The key challenge is efficiently using bandwidth-limited channels, whether it's a network link between nodes, the SATA bus between disk and RAM, or the memory channel between RAM and L3. It's also why we are both building compression into the native storage format for Blaze, and why Blaze is designed to run out-of-core from the start. https://github.com/ContinuumIO/blaze/blob/master/docs/source/persistence.rst https://github.com/ContinuumIO/blaze/blob/master/docs/source...
MidsizeBlowfish 13y ago
In the cases where I've had to use Hadoop, this was exactly what happened. We received a huge volume of data daily, which was aggregated by a nightly Hadoop job into a much more manageable amount of data for us to analyze with python/pandas.
sgt101 13y ago
I've found compression to be one of the big benefits of Hive / Impala; we are able to load and manipulate compressed data very quickly compared with our friends using a RDBMS who have to uncompress and then import the data. This shortens the cycle time on some analytics challenges by days (typically we get data in the 3-8TB range) where we can answer simple questions in a few hours (and senior management loose focus immediately afterwards).

pge 13y ago

In addition to compression, let's not forget sampling. For a lot of problems a random subset of the data will give you a statistically meaningful answer with sufficient precision. It seems like the rise of "big data" has led to the assumption that all queries have to be run against the entire dataset.

disgruntledphd2 13y ago
True. The issue with "Big Data" is that sometimes, especially when you need to produce personalised recommendations, the sampling doesn't cut it (or at least produces sub-optimal results).

vtuulos 13y ago

Exactly - encoding is the key. I just gave a presentation last week at the SF Python Meetup about how AdRoll uses a single server to query terabytes of compressed data with sub-minute latencies, using a system that is implemented in Python: http://tuulos.github.io/sf-python-meetup-sep-2013 http://tuulos.github.io/sf-python-meetup-sep-2013 This is probably a thousand times less resource intensive than using Hadoop for the same queries.

pwang 13y ago
Compression and out-of-core are not nearly discussed enough in the "big data" circles. The key challenge is efficiently using bandwidth-limited channels, whether it's a network link between nodes, the SATA bus between disk and RAM, or the memory channel between RAM and L3. It's also why we are both building compression into the native storage format for Blaze, and why Blaze is designed to run out-of-core from the start. https://github.com/ContinuumIO/blaze/blob/master/docs/source/persistence.rst https://github.com/ContinuumIO/blaze/blob/master/docs/source...

MidsizeBlowfish 13y ago

In the cases where I've had to use Hadoop, this was exactly what happened. We received a huge volume of data daily, which was aggregated by a nightly Hadoop job into a much more manageable amount of data for us to analyze with python/pandas.

sgt101 13y ago

I've found compression to be one of the big benefits of Hive / Impala; we are able to load and manipulate compressed data very quickly compared with our friends using a RDBMS who have to uncompress and then import the data. This shortens the cycle time on some analytics challenges by days (typically we get data in the 3-8TB range) where we can answer simple questions in a few hours (and senior management loose focus immediately afterwards).