4 ms·
Yes it's apparently so fast that in some scenarios it's even usable for compressing RAM. A framework I'm using does that to be able to process much bigger data
by w0utert 6y ago
Yes it's apparently so fast that in some scenarios it's even usable for compressing RAM. A framework I'm using does that to be able to process much bigger data sets than what would fit in RAM otherwise.
- ddorian43 6y agoCan you be more specific around the framework and data type and access patterns ?
- w0utert 6y agoIt's a framework called OpenVDB [1], which we use to represent and manipulate volumetric data (level sets). It stores the data as a sparse hierarchical grid with (from a practical perspective) infinite dimensions, and allows very efficient iteration and local manipulations of the grid. I'm not an expert on how it is implemted exactly, but I believe the way it uses Blosc is by saving leaves of the VDB grids in blosc-compressed chunks, which are loaded into memory directly and only decompressed on-demand when the data is accessed, then re-compressed when the leaves are processed. [1] https://www.openvdb.org https://www.openvdb.org