4 ms·

The benchmark for Zstandard against Brotli seems to miss a key information---the compression levels used for both algorithms, because both the compression ratio

The benchmark for Zstandard against Brotli seems to miss a key information---the compression levels used for both algorithms, because both the compression ratio and compression time will depend on them. In fact this had been my long suspicion about introducing Zstandard to the web standard, because lower compression levels for Brotli are not that slow and it was never publicly mentioned whether improving lower Brotli levels deemed infeasible or not. Given Zstandard Content-Encoding was initially proposed by Meta, I'm not even sure they have at least tried. Given we now have two strictly better algorithms than gzip, I also wonder about a hybrid scheme that starts with Zstandard but switches to Brotli when the compression time is no longer significant for given request. We might even be able to cheaply convert the existing Zstandard stream into Brotli with some restrictions, as they are really LZSS behind the scene?

terrelln 2y ago
Meta drove the Zstandard content encoding, but Google drove the adoption of Zstandard in Chrome. The faster Brotli levels could probably be made to match Zstandard’s compression speed. But we’ve invested a lot in optimizing these levels, so it would likely take significant investment to match. Google is also contributing to improving the speed of Zstandard compression. A cheaper conversion from Zstandard to Brotli is possible, but I wouldn’t really expect an improvement to compressed size. The encoding scheme impacts how efficient a LZ coding is, so for Brotli to beat Zstandard, it would likely want a different LZ than Zstandard uses. The same applies for a conversion from Brotli to Zstandard.
- lifthrasiir 2y ago
  The conversion is meant to avoid the expensive backreference optimization when it has been already done once by Zstandard, because you can't prepend a Zstandard bitstream with a Brotli bitstream without turning one to another. But well, I think such hybrid scheme is hard to make when the latency is at stake.

terrelln 2y ago

Meta drove the Zstandard content encoding, but Google drove the adoption of Zstandard in Chrome. The faster Brotli levels could probably be made to match Zstandard’s compression speed. But we’ve invested a lot in optimizing these levels, so it would likely take significant investment to match. Google is also contributing to improving the speed of Zstandard compression. A cheaper conversion from Zstandard to Brotli is possible, but I wouldn’t really expect an improvement to compressed size. The encoding scheme impacts how efficient a LZ coding is, so for Brotli to beat Zstandard, it would likely want a different LZ than Zstandard uses. The same applies for a conversion from Brotli to Zstandard.

lifthrasiir 2y ago
The conversion is meant to avoid the expensive backreference optimization when it has been already done once by Zstandard, because you can't prepend a Zstandard bitstream with a Brotli bitstream without turning one to another. But well, I think such hybrid scheme is hard to make when the latency is at stake.