4 ms·

I bet switching to LZMA would have saved even more. LZMA beats Brotli nearly every time. zStandard would likely have worked better as well. Brotli is very sl

I bet switching to LZMA would have saved even more. LZMA beats Brotli nearly every time. zStandard would likely have worked better as well. Brotli is very slow to compress.

gsylvie 10y ago
LZMA is much slower at decompress compared to Brotli. https://www.opencpu.org/posts/brotli-benchmarks/ https://www.opencpu.org/posts/brotli-benchmarks/
- adtac 10y ago
  Apps are usually in the order of 40-50 MB. From a back of the hand calculation, the a 50 MB file when compressed with Brotli is 14.5 MB and LZMA is 17.5 MB. The difference of 3 MB translates to an additional 8 seconds of data transfer in a 3.1 Mbps 3G connection (an average one). This [1] states that a LZMA has a decompression speed of 70 MB/s, which is about 0.7 seconds. The 334 MB/s speed of Brotli does the same in about 0.15 seconds. So the additional overhead of LZMA (compared to Brotli) in decompression is just 0.65 seconds. Given these orders of magnitudes, I think optimizing for compression ratio is a much better option than decompression speed. [1] https://cran.r-project.org/web/packages/brotli/vignettes/brotli-2015-09-22.pdf https://cran.r-project.org/web/packages/brotli/vignettes/bro...
  - bhouston 10y ago
    In my tests and others LZMA seems to beat Brotli by quite a bit on non-text cases (DB, 3D meshes): https://news.ycombinator.com/item?id=13589048 https://news.ycombinator.com/item?id=13589048 I am concerned that the Brotli v LZMA v GZip paper you cite is not fully representative as it is written by the the Brotli team.
  - bhouston 10y ago
    I just installed brotli on my ubuntu machine and compared it compressing a executable. Brotli: 4.77MB -> 1.21MB (8.5s) xz: 4.77MB -> 1.129MB (1.4s) LZMA does better than Brotli in the case of binaries by a fair bit every time.
mturmon 10y ago
More on the (speed, compression) trade space here: http://www.gstatic.com/b/brotlidocs/brotli-2015-09-22.pdf http://www.gstatic.com/b/brotlidocs/brotli-2015-09-22.pdf
- bhouston 10y ago
  This is not correct for the case of binaries.
Someone1234 10y ago
That doesn't appear to be true: https://cran.r-project.org/web/packages/brotli/vignettes/brotli-2015-09-22.pdf https://cran.r-project.org/web/packages/brotli/vignettes/bro... I'm sure you can use those results to argue that LZMA is superior in some way (e.g. compression speed) but it definitely isn't clear cut superior in other important ways (compressed size and decompression speed are inferior). I can see why, given those results, that they would use Brotli over LZMA.
- bhouston 10y ago
  The independent tests I did here: https://github.com/google/brotli/issues/165 https://github.com/google/brotli/issues/165 And also those here: https://www.percona.com/blog/2016/03/09/evaluating-database-compression-methods/ https://www.percona.com/blog/2016/03/09/evaluating-database-... Suggest that LZMA compresses better than Brotli except in the case of text documents.
gribbly 10y ago
I agree that LZMA beats Brotli in compression in the majority of cases (not by much), zStandard does not however. The thing that makes Brotli attractive though, is that it has high compression (again, very close to LZMA, sometimes even better) while decompressing MUCH faster than LZMA. The big downside is that it is very slow in compressing, which makes it mainly suitable for 'compress once, decompress MANY times' type data.

I bet switching to LZMA would have saved even more. LZMA beats Brotli nearly every time. zStandard would likely have worked better as well. Brotli is very slow to compress.

gsylvie 10y ago

LZMA is much slower at decompress compared to Brotli. https://www.opencpu.org/posts/brotli-benchmarks/ https://www.opencpu.org/posts/brotli-benchmarks/

adtac 10y ago
Apps are usually in the order of 40-50 MB. From a back of the hand calculation, the a 50 MB file when compressed with Brotli is 14.5 MB and LZMA is 17.5 MB. The difference of 3 MB translates to an additional 8 seconds of data transfer in a 3.1 Mbps 3G connection (an average one). This [1] states that a LZMA has a decompression speed of 70 MB/s, which is about 0.7 seconds. The 334 MB/s speed of Brotli does the same in about 0.15 seconds. So the additional overhead of LZMA (compared to Brotli) in decompression is just 0.65 seconds. Given these orders of magnitudes, I think optimizing for compression ratio is a much better option than decompression speed. [1] https://cran.r-project.org/web/packages/brotli/vignettes/brotli-2015-09-22.pdf https://cran.r-project.org/web/packages/brotli/vignettes/bro...
- bhouston 10y ago
  In my tests and others LZMA seems to beat Brotli by quite a bit on non-text cases (DB, 3D meshes): https://news.ycombinator.com/item?id=13589048 https://news.ycombinator.com/item?id=13589048 I am concerned that the Brotli v LZMA v GZip paper you cite is not fully representative as it is written by the the Brotli team.
- bhouston 10y ago
  I just installed brotli on my ubuntu machine and compared it compressing a executable. Brotli: 4.77MB -> 1.21MB (8.5s) xz: 4.77MB -> 1.129MB (1.4s) LZMA does better than Brotli in the case of binaries by a fair bit every time.

mturmon 10y ago

More on the (speed, compression) trade space here: http://www.gstatic.com/b/brotlidocs/brotli-2015-09-22.pdf http://www.gstatic.com/b/brotlidocs/brotli-2015-09-22.pdf

bhouston 10y ago
This is not correct for the case of binaries.

Someone1234 10y ago

That doesn't appear to be true: https://cran.r-project.org/web/packages/brotli/vignettes/brotli-2015-09-22.pdf https://cran.r-project.org/web/packages/brotli/vignettes/bro... I'm sure you can use those results to argue that LZMA is superior in some way (e.g. compression speed) but it definitely isn't clear cut superior in other important ways (compressed size and decompression speed are inferior). I can see why, given those results, that they would use Brotli over LZMA.

bhouston 10y ago
The independent tests I did here: https://github.com/google/brotli/issues/165 https://github.com/google/brotli/issues/165 And also those here: https://www.percona.com/blog/2016/03/09/evaluating-database-compression-methods/ https://www.percona.com/blog/2016/03/09/evaluating-database-... Suggest that LZMA compresses better than Brotli except in the case of text documents.

gribbly 10y ago

I agree that LZMA beats Brotli in compression in the majority of cases (not by much), zStandard does not however. The thing that makes Brotli attractive though, is that it has high compression (again, very close to LZMA, sometimes even better) while decompressing MUCH faster than LZMA. The big downside is that it is very slow in compressing, which makes it mainly suitable for 'compress once, decompress MANY times' type data.