4 ms·
As general purpose compressor iguana is decompressing a lot slower than advertised, when tested with a typical data compression corpus. It requires: avx512-vbm
by powturbo 3y ago
As general purpose compressor iguana is decompressing a lot slower than advertised, when tested with a typical data compression corpus.
It requires: avx512-vbmi2, available only on ice-lake/Tiger-lake/AMD zen4
- Benchmark from encode.su experts: https://encode.su/threads/4041-Iguana-a-fast-vectorized-compressor?p=79634&viewfull=1#post79634 https://encode.su/threads/4041-Iguana-a-fast-vectorized-comp...
- benchmark from the iguana developers here: https://github.com/SnellerInc/sneller/tree/master/cmd/iguanabench https://github.com/SnellerInc/sneller/tree/master/cmd/iguana...
Silesia corpus / cpu Xeon Gold 5320
zstd -b3 3.186 943.9 MB/s
zstd -b9 3.574 1015.8 MB/s
zstd -b18 3.967 910.6 MB/s
lz4 -b1 2.101 3493.8 MB/s
lz4 -b5 2.687 3323.5 MB/s
lz4 -b9 2.721 3381.5 MB/s
iguana -t=0 2.58 4450 MB/s
iguana -t=1 3.11 2260 MB/s
As you can see, iguana with entropy coding enabled (-t 1) has a similar compression ratio to zstd -3, but it decompresses more than twice as quickly. With entropy coding disabled (-t 0), iguana has a compression ratio roughly equivalent to lz4 -5 and decompresses about 33% faster.
- thomasahle 3y ago> It requires: avx512-vbmi2, available only on skylake-X/Tiger-lake/AMD zen4 Isn't Avx 512 also available on Xeon which is most hosted servers, like aws.
- powturbo 3y agoNot only AVX-512, but AVX-512 + VBMI2
- Aardwolf 3y agoNo modern intel consumer CPU has avx512 anymore, they only support at most the 10 year old avx2 vector instruction set, because they want to give every single consumer chip efficiency cores, those can't handle it, and even the performance cores have this feature removed due to it. The modern AMD Zen 4 CPU's for consumers do support avx512 (including avx512_vbmi2).
- powturbo 3y ago> and even the performance cores have to be downgraded for it. Old rumours, it's no longer true.
- skavi 3y agonot really rumors, but also not necessarily true. the performance cores on consumer intel parts no longer have AVX-512 enabled. This is because it’s tough to implement on the smaller efficiency cores, and scheduling is more annoying with heterogeneous ISAs.
- powturbo 3y agoI understand downgrading as thermal throtlling.
- Aardwolf 3y agoI did mean have the feature removed, I edited the comment (replaced "downgraded" with "have this feature removed". I actually originally typed "crippled" but wanted to phrase it more mildly). What I mean is to avoid the heterogeneous cores they chose to disable it in the performance cores rather than e.g. emulate or double pump it in the efficiency cores. And they chose to give you efficiency cores + performance cores without avx512 on all consumer CPU's, rather than, what would be my preference, sell some with only performance cores with avx512 like their previous gen CPUs. I'd be ok with efficiency cores on a laptop, sure, but not on a desktop machine. But AMD already gives plenty fast cores with avx512 now so I'm satisfied, I'm simply not an intel user anymore. Of course intel's choice still affects me anyway since the less consumers have avx512, the less software will make use of it.
- pjc50 3y agoSo .. what's the vector set that people should be targeting for wide compatibility these days?
- 3y ago
- bugboy73 3y agoAll major cloud providers (AWS, Azure and GCP) provide instance-types that support AVX512-VBMI2. Iguana targets large amounts of that are decompressed often and speed is of the essence. Those are typical server scenarios, so the AVX512-VBMI2 requirement isn't really an issue in common-case scenarios. DISCLAIMER: I work for Sneller
- moonchild 3y agovbmi2 is not available on skylakex
- powturbo 3y agosorry, Ice-Lake not skylake