powturbo

7 ms·

31.
by powturbo 3y ago
You'll get significantly better compression by using general purpose compressors. Compressesing your sample file, we get 823 bytes with brotli Download TurboBench and make your own tests: [1] - https://github.com/powtur
32.
by powturbo 3y ago
I've made a benchmark with real html data including memory usage in [2] and a recent benchmark in [3] with TurboBench [1]. In the speedup plots you can see the best compressors for content providers: - brotli 11 is best for static cont
33.
by powturbo 3y ago
Most time-series / analytical databases,... are already using or switching to integer-compression [1] where you can compress/decompress several times (>100GB/s see TurboBitByte in [2]) faster than general purpose compresso
34.
by powturbo 3y ago
Ideally, dynamic detecting cpu isa (sse/avx2/avx512) and respectively dispatching, used in most of my projects for ex. [1] and [2] - [1] https://github.com/powturbo/Turbo-Range-Coder - [2] https://
35.
by powturbo 3y ago
I understand downgrading as thermal throtlling.
36.
by powturbo 3y ago
> and even the performance cores have to be downgraded for it. Old rumours, it's no longer true.
37.
by powturbo 3y ago
sorry, Ice-Lake not skylake
38.
by powturbo 3y ago
Not only AVX-512, but AVX-512 + VBMI2
39.
by powturbo 3y ago
As general purpose compressor iguana is decompressing a lot slower than advertised, when tested with a typical data compression corpus. It requires: avx512-vbmi2, available only on ice-lake/Tiger-lake/AMD zen4 - Benchmark from enc
40.
by powturbo 3y ago
Build or download TurboBench [1] executables for linux and windows from releases [2] ans make your own tests comparing oodle,zstd and other compressors. [1] https://github.com/powturbo/TurboBench [2] https:/&#x2F
41.
by powturbo 3y ago
Well, this doesn't prevent you to consider it in your survey. Again, thank you for making this great floating point compression survey.
42.
by powturbo 3y ago
Thanks for your response. Unfortunatelly blosc benchmark data is not easy accessible and you're not publishing the exact numbers in your survey. This makes it difficult to compare the results. The data in my benchmarks are directly dow
43.
by powturbo 3y ago
I've some remarks regarding the tests and the test data in this survey. - The data used in the benchmark seems very compressible. However floating point data (also time series) is in general not well losslessly compressible and the Gor
44.
Show HN:The Gorilla in the Room:Exploring RedisTimeSeries Optimizations(community.intel.com)
1 points by powturbo 3y ago|1 comments
45.
by powturbo 4y ago
Excellent blog post serie about floating point compression. All those (lossless+lossy) floating point compression algos including 1D/2D/3D SIMD transpose with integrated xor and zigzag delta are available in the TurboPFor [1] Inte
46.
by powturbo 4y ago
- 1D/2D/3d Lossy/Lossless Benchmark comparing TurboPFor Floating Point Compression to zfp & blosc. - Several datasets with different distributions and from diverse fields used in the benchmark - TurboPFor with 1D/2D&
47.
Show HN: 1D/2D/3D Lossless/Lossy Floating Point Compression with TurboPFor(github.com)
3 points by powturbo 4y ago|1 comments
48.
by powturbo 4y ago
Try a bwt compressor like BwtSatan : https://github.com/powturbo/Turbo-Range-Coder/releases/tag/2... bwtsatan -20 inputfile outputfile
49.
Show HN: Turbo Base64 the fastest base64 now more faster(github.com)
1 points by powturbo 5y ago|0 comments
50.
by powturbo 5y ago
100% C (C++ headers). OS/Arch: Linux amd/intel, arm, PowerPC, s390x, MacOs+Apple M1. Windows: Mingw, visual c++ No other Range Coder / Arithmetic Coder encode or decode faster with better compression Up to 3 times faster than
51.
Show HN:Fastest Range Coder for byte, integer, floating point data and BWT(github.com)
2 points by powturbo 5y ago|1 comments
52.
by powturbo 7y ago
There are nice plots [1] to see the tranfer+decompression speedup depening on the network bandwidth. This is for html web compression, but the results are similar for other datasets. For internet transfer more compression is better than mor
53.
JPEG XL Reference Software(gitlab.com)
5 points by powturbo 7y ago|0 comments
54.
by powturbo 7y ago
JPEG-XL: Next Generation Image Compression Codec Open source / Royality free NEW public software repostory https://gitlab.com/wg1/jpeg-xl
55.
by powturbo 7y ago
You might be interested on "Turbo Range Coder" [1] a bitwise and multisymbol range coder, that can be used as a library. A command line compressor is also included 1 - https://github.com/powturbo/Turbo-Range-C
56.
by powturbo 7y ago
Fastest Range Coder/Arithmetic Coder 100% C (C++ headers). OS:Linux amd64, arm64, Power9, MacOs. Windows: Mingw, visual c++ No other Range Coder / Arithmetic Coder encode or decode faster Can work as bitwise or/and as multisy
57.
Show HN: HN: Turbo Range Coder / Arithmetic Coding(github.com)
2 points by powturbo 7y ago|1 comments
58.
by powturbo 7y ago
It might be possible to use integer compression with block offsets like in the TurboPFor [1] demo. TurboPFor offers also direct access to single values in a block. 1 - https://github.com/powturbo/TurboPFor
59.
by powturbo 7y ago
Elias-fano are overestimated without any proof or comparison benchmarks. Like other integer compression methods, you still need offsets to small blocks for efficient random access. Getting a single value in a block is done with slow branch
60.
by powturbo 7y ago
L1 cache is 64k for this cpu. A benchmark with large files is more realistic, because it's unlikely that the data already be in the L1 cache.

More ›