2 ms·
In “faster than memcpy” series we have also Blosch for modern CPUs https://www.blosc.org/pages/blosc-in-depth/ https://www.blosc.org/pages/blosc-in-depth/ I h
by miohtama 3y ago
In “faster than memcpy” series we have also Blosch for modern CPUs
https://www.blosc.org/pages/blosc-in-depth/ https://www.blosc.org/pages/blosc-in-depth/
I have never been able to use Blosch myself. But it sounds really interesting, outperforming RAM. Not sure what are the applications - columnar data processing, Parquet files, etc?
- thechao 3y agoWhy doesn't BLOSC have a little chart comparing itself to LZ4, Zstd, etc? Kinda like this: https://stackoverflow.com/questions/37614410/comparison-between-lz4-vs-lz4-hc-vs-blosc-vs-snappy-vs-fastlz https://stackoverflow.com/questions/37614410/comparison-betw... Because it seems like such a trivial chart to make.
- jeramey 3y agoIt gets used a fair amount in the weather data space. Forecasting and climate reanalysis grids are typically large (gigabytes) N-dimensional arrays of float32 values and Blosc provides enough tunable knobs that it's fairly easy to find a combination that performs acceptably without writing a bunch of custom handling code to keep track of which underlying compression schemes and settings were used. Additionally, it supports byte- and bit-shuffling filters which can really help boost the compressibility of certain data sets.