3 ms·

Excellent blog post serie about floating point compression. All those (lossless+lossy) floating point compression algos including 1D/2D/3D SIMD transpose with

Excellent blog post serie about floating point compression. All those (lossless+lossy) floating point compression algos including 1D/2D/3D SIMD transpose with integrated xor and zigzag delta are available in the TurboPFor [1] Integer Compession package. There is a benchmark with the same dataset available in [2] with a comparison to zfp and blosc with the new bytedelta. TurboPFor functions compress/decompress faster with better compression. The bytedelta described in the blog is suboptimal and might not work with other datasets. Download icapp from [3] and make your own tests with your data. [1] https://github.com/powturbo/TurboPFor-Integer-Compression https://github.com/powturbo/TurboPFor-Integer-Compression [2] https://github.com/powturbo/TurboPFor-Integer-Compression/issues/92 https://github.com/powturbo/TurboPFor-Integer-Compression/is... [3] https://github.com/powturbo/TurboPFor-Integer-Compression/releases https://github.com/powturbo/TurboPFor-Integer-Compression/re...

aras_p 4y ago
Yeah I was considering whether to include TurboPFor at some point or not. It looks really good, but for my particular use case is not applicable due to GPL license.
- powturbo 3y ago
  Well, this doesn't prevent you to consider it in your survey. Again, thank you for making this great floating point compression survey.
aras_p 3y ago
I can't comment on the linked github issues, but looking at the turbopfor icapp.c sources, it looks like the blosc2 bytedelta is not used correctly there: the expected order is "shuffle, then bytedelta", but icapp.c does "bytedelta, then shuffle". That might explain the poor result you get :)
- powturbo 3y ago
  Thanks for your response. Unfortunatelly blosc benchmark data is not easy accessible and you're not publishing the exact numbers in your survey. This makes it difficult to compare the results. The data in my benchmarks are directly downloadable and is a proposal from academia covering diverse distrutions. I've now reverted the order to "shuffle, then bytedelta". It looks better but still inferior to TurboPFor floating point compression and is also 35% slower in decompression. TurboPFor FP with lz4 instead of zstd,15 has similar ratio than blosc+bytedelta+zstd,15, but again it is a lot faster. Please, publish the exact numbers of your survey at least those with zstd,15. The zfp results in my benchmark don't look good. Are the numbers similar to yours?

aras_p 4y ago

Yeah I was considering whether to include TurboPFor at some point or not. It looks really good, but for my particular use case is not applicable due to GPL license.

powturbo 3y ago
Well, this doesn't prevent you to consider it in your survey. Again, thank you for making this great floating point compression survey.

aras_p 3y ago

I can't comment on the linked github issues, but looking at the turbopfor icapp.c sources, it looks like the blosc2 bytedelta is not used correctly there: the expected order is "shuffle, then bytedelta", but icapp.c does "bytedelta, then shuffle". That might explain the poor result you get :)

powturbo 3y ago
Thanks for your response. Unfortunatelly blosc benchmark data is not easy accessible and you're not publishing the exact numbers in your survey. This makes it difficult to compare the results. The data in my benchmarks are directly downloadable and is a proposal from academia covering diverse distrutions. I've now reverted the order to "shuffle, then bytedelta". It looks better but still inferior to TurboPFor floating point compression and is also 35% slower in decompression. TurboPFor FP with lz4 instead of zstd,15 has similar ratio than blosc+bytedelta+zstd,15, but again it is a lot faster. Please, publish the exact numbers of your survey at least those with zstd,15. The zfp results in my benchmark don't look good. Are the numbers similar to yours?