5 ms·

Not sure if anything exists but I wish something would do in memory compression + smart disk spillover. Sometimes I want to work with 5-10GB compressed data set

by nijave 6y ago

Not sure if anything exists but I wish something would do in memory compression + smart disk spillover. Sometimes I want to work with 5-10GB compressed data sets (usually log files) and decompressed that ends up being 10x (plus add data structure overhead). There's stuff like Apache Drill but it's more optimized for multi node than running locally

Keyframe 6y ago
gzip, zcat, parallel... That sort of thing?
- nijave 6y ago
  Yeah but starts to fall apart for anything beyond basic search/filtering (like grouping, computed fields)
cinntaile 6y ago
What do you mean with smart disk spillover?
- FridgeSeal 6y ago
  When the library attempts to load something from disk that doesn’t fit into memory, it’s transparently, and (usually) without extra intervention from the user, swaps to memory-mapping and chunking through the file(s). Particularly useful for when you’ve got a bunch of data that doesn’t fit in memory, but setting up a whole cluster is not worth the overhead (operationally or otherwise) and/or if you’ve already got a processing pipeline written in a language/framework and you can’t/don’t-want to go through rewriting it for something distributed.
  - hermitcrab 6y ago
    How is that different to virtual memory?
    - dTal 6y ago
      Memory mapping lazily loads the file, completes immediately, and scales arbitrarily; using disk-backed virtual memory would require the entire file to be read from disk, written back out to disk, and then read in from disk again on access; it would also require swap to be set up at the OS level, and the amount of swap set up puts a hard limit on the size of the file.
      - wtallis 6y ago
        You've described the difference between using mmap vs relying on the operating system's swap mechanism. But neither of those is quite the same as having an application that's aware of its memory usage and explicitly manages what it keeps in RAM. Using mmap may be useful for achieving that, but mmap on its own still leaves most of the management up to the OS.
      - hermitcrab 6y ago
        Thanks for clearing that up!
    - nijave 6y ago
      In memory representation also tends to have some data structure overhead so 5GB compressed -> 50GB uncompressed -> 100-250GB as a data structure (seems 2-5x is pretty normal). It starts out seeming pretty innocuous but quickly explodes. In addition, some things like Drill can do some automatic indexing/metadata recording which can reduce the amount of data it needs to access
rscho 6y ago
If you're not afraid of exotic languages, I encourage you to have a look at the APL ecosystem, and especially J and it's integrated columnar store Jd. I have just embarked on an adventure to do just what you describe in... Racket. But it's nowhere to be seen yet. I'm an epidemiologist and I've been wanting to make my own tools for a while, now. It'll be interesting to see how far I can go with Racket, which already includes many pieces of the puzzle.
- cfourpio 6y ago
  Would you mind pointing me at a resource that explains how J/Jd handle it by comparison?
  - rscho 6y ago
    https://code.jsoftware.com/wiki/Jd/Overview https://code.jsoftware.com/wiki/Jd/Overview Jd only packs int vectors, though. So if you're hoping for string compression then I don't know of any free solution. Jd heavily leverages SIMD and mmap. Larger-than RAM columns can be easily processed by ftable. I use Jd for data wrangling before making models in R. Of course, the J language is not for the faint of heart but it's really well-suited to the task.
Enginerrrd 6y ago
I feel like the HDF5 libraries could be helpful here if you could figure out how to get compatible compression.
amatsukawa 6y ago
Dask is popular for this purpose. No memory compression, but supports disk spillage as well as distributed dataframes.
kthejoker2 6y ago
If youre doing OLAP style queries you should look at DuckDB, it's hella fast and supports out-of-memory compute (it's not exactly "smart" but it handles spillover)
sgtnoodle 6y ago
How is that different from, say, opening up a gzipped file with a reader object in python?
- nijave 6y ago
  It's not but implementing intelligent data access each time can become pretty tedious (sure you can write libraries and tools, but I'm basically asking if those already exist) For something like AWS CloudTrail logs, 5GB is 40k 100-130kb gzipped json files so hit single-core CPU bounds almost immediately (just reading/decompressing/json parsing off an SSD). CPU scale out model in Python is processes so now you're copying data between processes if you want to parallelize it so now you hit IPC bottlenecks just using the standard library multiprocessing/concurrent futures stuff 5GB compressed /probably/ won't fit in memory so now you have to deal with that, too unless you have a way to keep it compressed (which would come at the cost of additional CPU usage) tldr; it's non-trivial to actually fully use the hardware
  - sgtnoodle 6y ago
    Stepping back a bit, if you're CPU bound in a single thread decompressing data off an SSD, does copying the compressed data into memory first actually buy you anything? If you truly need the dataset fully loaded into memory for performance reasons, then it's presumably because of the need to do lots of random accesses, where the read latency would otherwise harm you. The tricky bit is the fact that it's generally hard to randomly access compressed streams of data. You need to compress the data in a way that makes random access possible, likely to the detriment of compression ratio. Unless you also use the same compression format to store the data on disk, then you're back to having to decompress (and recompress) the whole file sequentially anyway in order to build the data structure in RAM. I've seen purpose-made compressed log formats that support efficient seeking. I've never seen them loaded into RAM in their raw compressed form, though. Generally they do have a corresponding library to make accessing the log data easy.
bobbylarrybobby 6y ago
I wonder if Arrow (https://arrow.apache.org/faq/ https://arrow.apache.org/faq/) would work for this: "The Arrow IPC mechanism is based on the Arrow in-memory format, such that there is no translation necessary between the on-disk representation and the in-memory representation. Therefore, performing analytics on an Arrow IPC file can use memory-mapping, avoiding any deserialization cost and extra copies."
- deleted 6y ago
  [deleted]
nighthawk454 6y ago
maybe give Vaex a try. it's along those lines – out-of-core data frame library, but also works on a single machine https://github.com/vaexio/vaex https://github.com/vaexio/vaex
f311a 6y ago
I think ClickHouse does that.
- nijave 6y ago
  I actually started looking at Clickhouse a couple weeks ago but got a bit side tracked trying to grok how distributed tables work. It looks promising but there's a bit of a learning curve (seems some of the performance also comes from its use of arrays but best I can tell my use case should just use regular tables)
  - hodgesrm 6y ago
    ClickHouse performance is principally due to column storage, compression, and ability to parallelize processing. Arrays can improve performance in some specific cases but are more commonly used to help deal with semi-structured data or perform custom processing on values within groups. If your data maps cleanly to tables, that's in fact the best case with the easiest options for performance enhancement.
superdimwit 6y ago
You can memory map Arrow

by nijave 6y ago

Keyframe 6y ago

gzip, zcat, parallel... That sort of thing?

nijave 6y ago
Yeah but starts to fall apart for anything beyond basic search/filtering (like grouping, computed fields)

cinntaile 6y ago

What do you mean with smart disk spillover?

FridgeSeal 6y ago
When the library attempts to load something from disk that doesn’t fit into memory, it’s transparently, and (usually) without extra intervention from the user, swaps to memory-mapping and chunking through the file(s). Particularly useful for when you’ve got a bunch of data that doesn’t fit in memory, but setting up a whole cluster is not worth the overhead (operationally or otherwise) and/or if you’ve already got a processing pipeline written in a language/framework and you can’t/don’t-want to go through rewriting it for something distributed.
- hermitcrab 6y ago
  How is that different to virtual memory?
  - dTal 6y ago
    Memory mapping lazily loads the file, completes immediately, and scales arbitrarily; using disk-backed virtual memory would require the entire file to be read from disk, written back out to disk, and then read in from disk again on access; it would also require swap to be set up at the OS level, and the amount of swap set up puts a hard limit on the size of the file.
    - wtallis 6y ago
      You've described the difference between using mmap vs relying on the operating system's swap mechanism. But neither of those is quite the same as having an application that's aware of its memory usage and explicitly manages what it keeps in RAM. Using mmap may be useful for achieving that, but mmap on its own still leaves most of the management up to the OS.
    - hermitcrab 6y ago
      Thanks for clearing that up!
  - nijave 6y ago
    In memory representation also tends to have some data structure overhead so 5GB compressed -> 50GB uncompressed -> 100-250GB as a data structure (seems 2-5x is pretty normal). It starts out seeming pretty innocuous but quickly explodes. In addition, some things like Drill can do some automatic indexing/metadata recording which can reduce the amount of data it needs to access

rscho 6y ago

If you're not afraid of exotic languages, I encourage you to have a look at the APL ecosystem, and especially J and it's integrated columnar store Jd. I have just embarked on an adventure to do just what you describe in... Racket. But it's nowhere to be seen yet. I'm an epidemiologist and I've been wanting to make my own tools for a while, now. It'll be interesting to see how far I can go with Racket, which already includes many pieces of the puzzle.

cfourpio 6y ago
Would you mind pointing me at a resource that explains how J/Jd handle it by comparison?
- rscho 6y ago
  https://code.jsoftware.com/wiki/Jd/Overview https://code.jsoftware.com/wiki/Jd/Overview Jd only packs int vectors, though. So if you're hoping for string compression then I don't know of any free solution. Jd heavily leverages SIMD and mmap. Larger-than RAM columns can be easily processed by ftable. I use Jd for data wrangling before making models in R. Of course, the J language is not for the faint of heart but it's really well-suited to the task.

Enginerrrd 6y ago

I feel like the HDF5 libraries could be helpful here if you could figure out how to get compatible compression.

amatsukawa 6y ago

Dask is popular for this purpose. No memory compression, but supports disk spillage as well as distributed dataframes.

kthejoker2 6y ago

If youre doing OLAP style queries you should look at DuckDB, it's hella fast and supports out-of-memory compute (it's not exactly "smart" but it handles spillover)

sgtnoodle 6y ago

How is that different from, say, opening up a gzipped file with a reader object in python?

nijave 6y ago
It's not but implementing intelligent data access each time can become pretty tedious (sure you can write libraries and tools, but I'm basically asking if those already exist) For something like AWS CloudTrail logs, 5GB is 40k 100-130kb gzipped json files so hit single-core CPU bounds almost immediately (just reading/decompressing/json parsing off an SSD). CPU scale out model in Python is processes so now you're copying data between processes if you want to parallelize it so now you hit IPC bottlenecks just using the standard library multiprocessing/concurrent futures stuff 5GB compressed /probably/ won't fit in memory so now you have to deal with that, too unless you have a way to keep it compressed (which would come at the cost of additional CPU usage) tldr; it's non-trivial to actually fully use the hardware
- sgtnoodle 6y ago
  Stepping back a bit, if you're CPU bound in a single thread decompressing data off an SSD, does copying the compressed data into memory first actually buy you anything? If you truly need the dataset fully loaded into memory for performance reasons, then it's presumably because of the need to do lots of random accesses, where the read latency would otherwise harm you. The tricky bit is the fact that it's generally hard to randomly access compressed streams of data. You need to compress the data in a way that makes random access possible, likely to the detriment of compression ratio. Unless you also use the same compression format to store the data on disk, then you're back to having to decompress (and recompress) the whole file sequentially anyway in order to build the data structure in RAM. I've seen purpose-made compressed log formats that support efficient seeking. I've never seen them loaded into RAM in their raw compressed form, though. Generally they do have a corresponding library to make accessing the log data easy.

bobbylarrybobby 6y ago

I wonder if Arrow (https://arrow.apache.org/faq/ https://arrow.apache.org/faq/) would work for this: "The Arrow IPC mechanism is based on the Arrow in-memory format, such that there is no translation necessary between the on-disk representation and the in-memory representation. Therefore, performing analytics on an Arrow IPC file can use memory-mapping, avoiding any deserialization cost and extra copies."

deleted 6y ago
[deleted]

nighthawk454 6y ago

maybe give Vaex a try. it's along those lines – out-of-core data frame library, but also works on a single machine https://github.com/vaexio/vaex https://github.com/vaexio/vaex

f311a 6y ago

I think ClickHouse does that.

nijave 6y ago
I actually started looking at Clickhouse a couple weeks ago but got a bit side tracked trying to grok how distributed tables work. It looks promising but there's a bit of a learning curve (seems some of the performance also comes from its use of arrays but best I can tell my use case should just use regular tables)
- hodgesrm 6y ago
  ClickHouse performance is principally due to column storage, compression, and ability to parallelize processing. Arrays can improve performance in some specific cases but are more commonly used to help deal with semi-structured data or perform custom processing on values within groups. If your data maps cleanly to tables, that's in fact the best case with the easiest options for performance enhancement.

superdimwit 6y ago

You can memory map Arrow