2 ms·

How is that different from, say, opening up a gzipped file with a reader object in python?

nijave 6y ago
It's not but implementing intelligent data access each time can become pretty tedious (sure you can write libraries and tools, but I'm basically asking if those already exist) For something like AWS CloudTrail logs, 5GB is 40k 100-130kb gzipped json files so hit single-core CPU bounds almost immediately (just reading/decompressing/json parsing off an SSD). CPU scale out model in Python is processes so now you're copying data between processes if you want to parallelize it so now you hit IPC bottlenecks just using the standard library multiprocessing/concurrent futures stuff 5GB compressed /probably/ won't fit in memory so now you have to deal with that, too unless you have a way to keep it compressed (which would come at the cost of additional CPU usage) tldr; it's non-trivial to actually fully use the hardware
- sgtnoodle 6y ago
  Stepping back a bit, if you're CPU bound in a single thread decompressing data off an SSD, does copying the compressed data into memory first actually buy you anything? If you truly need the dataset fully loaded into memory for performance reasons, then it's presumably because of the need to do lots of random accesses, where the read latency would otherwise harm you. The tricky bit is the fact that it's generally hard to randomly access compressed streams of data. You need to compress the data in a way that makes random access possible, likely to the detriment of compression ratio. Unless you also use the same compression format to store the data on disk, then you're back to having to decompress (and recompress) the whole file sequentially anyway in order to build the data structure in RAM. I've seen purpose-made compressed log formats that support efficient seeking. I've never seen them loaded into RAM in their raw compressed form, though. Generally they do have a corresponding library to make accessing the log data easy.

It's not but implementing intelligent data access each time can become pretty tedious (sure you can write libraries and tools, but I'm basically asking if those already exist) For something like AWS CloudTrail logs, 5GB is 40k 100-130kb gzipped json files so hit single-core CPU bounds almost immediately (just reading/decompressing/json parsing off an SSD). CPU scale out model in Python is processes so now you're copying data between processes if you want to parallelize it so now you hit IPC bottlenecks just using the standard library multiprocessing/concurrent futures stuff 5GB compressed /probably/ won't fit in memory so now you have to deal with that, too unless you have a way to keep it compressed (which would come at the cost of additional CPU usage) tldr; it's non-trivial to actually fully use the hardware

sgtnoodle 6y ago
Stepping back a bit, if you're CPU bound in a single thread decompressing data off an SSD, does copying the compressed data into memory first actually buy you anything? If you truly need the dataset fully loaded into memory for performance reasons, then it's presumably because of the need to do lots of random accesses, where the read latency would otherwise harm you. The tricky bit is the fact that it's generally hard to randomly access compressed streams of data. You need to compress the data in a way that makes random access possible, likely to the detriment of compression ratio. Unless you also use the same compression format to store the data on disk, then you're back to having to decompress (and recompress) the whole file sequentially anyway in order to build the data structure in RAM. I've seen purpose-made compressed log formats that support efficient seeking. I've never seen them loaded into RAM in their raw compressed form, though. Generally they do have a corresponding library to make accessing the log data easy.