3 ms·
Blosc is an excellent choice if speed is what you are after. Give or take 5 years ago I had to use a compression to transport a lot of data over zmq and blosc r
by axegon_ 6y ago
Blosc is an excellent choice if speed is what you are after. Give or take 5 years ago I had to use a compression to transport a lot of data over zmq and blosc ran in circles over all other compressions.
- w0utert 6y agoYes it's apparently so fast that in some scenarios it's even usable for compressing RAM. A framework I'm using does that to be able to process much bigger data sets than what would fit in RAM otherwise.
- ddorian43 6y agoCan you be more specific around the framework and data type and access patterns ?
- w0utert 6y agoIt's a framework called OpenVDB [1], which we use to represent and manipulate volumetric data (level sets). It stores the data as a sparse hierarchical grid with (from a practical perspective) infinite dimensions, and allows very efficient iteration and local manipulations of the grid. I'm not an expert on how it is implemted exactly, but I believe the way it uses Blosc is by saving leaves of the VDB grids in blosc-compressed chunks, which are loaded into memory directly and only decompressed on-demand when the data is accessed, then re-compressed when the leaves are processed. [1] https://www.openvdb.org https://www.openvdb.org