4 ms·

They were designed more or less independently from each other, and your statement that Zstandard predates Brotli is also wrong (both around 2015). They are both

They were designed more or less independently from each other, and your statement that Zstandard predates Brotli is also wrong (both around 2015). They are both designed to be much more efficient than DEFLATE in both time and size axes, but otherwise have little similarities. Zstandard is notable in its use of ANS, while Brotli is probably the first mainstream format that prominently uses a 2nd-order context model which was previously thought to be too slow. And Brotli is designed for web contents in mind, which made it excellent for small inputs with no prior. Zstandard has no such treatments but have a better performance when I/O can handle it, which makes it excellent for server jobs.

jacobgorm 3y ago
ZStd started out as ZHuff, which was first announced in 2008; see https://encode.su/threads/521-Zhuff-fast-compression https://encode.su/threads/521-Zhuff-fast-compression and http://fastcompression.blogspot.com/p/zhuff.html http://fastcompression.blogspot.com/p/zhuff.html I've been using LZ4 in commercial software since 2011, so unless I am some kind of luminary Google's compression experts must have been aware of both LZ4 and ZHuff when they started working on Brotli.
- lifthrasiir 3y ago
  By that reckoning I can claim that the origin of Brotli also dates back to 2012 or earlier. :-) Indeed, Brotli authors were also reponsible for the VP8 Lossless codec (2012), Pik (2017) and eventually JPEG XL (2019, in part). The entropy coding scheme is substantially similar among those codecs for this reason. But really, the large amount of differences between two formats rather suggest that they were designed for different uses in mind, which I believe relate to the maximum possible I/O. Formats designed for the web don't have to have an extremely fast decoder because I/O is the bottleneck: 100 MB/s should be enough, and the earned time can be invested to higher compression ratio which affects the decoder performance.
  - adgjlsfhk1 3y ago
    otoh, they achieve pretty similar compression ratios and faster decode saves battery life and will let you get on to running the JavaScript earlier
miohtama 3y ago
The performance depends on the data. I have found that for English text and source code Brotli overperforms Zstd. Not sure if this because of the static dictionary or some other tweaks, though. However none of this is static, as both technologies are actively developed.
- lifthrasiir 3y ago
  Better decoding time performance, I should have said. The use of 2nd-order context model indeed makes Brotli more efficient in compression ratio for many kinds of data.
pmarreck 3y ago
Yes, but hasn't "separate compression dictionary" been a core feature of Zstd since its inception, at least under that name?
- lifthrasiir 3y ago
  Not even close, zlib had one. (Search for "FDICT" from RFC 1950.) What Zstandard did was a CLI to automatically generate a good enough dictionary from sample inputs.
  - pmarreck 3y ago
    FDICT (Preset dictionary) If FDICT is set, a DICT dictionary identifier is present immediately after the FLG byte. The dictionary is a sequence of bytes which are initially fed to the compressor without producing any compressed output. DICT is the Adler-32 checksum of this sequence of bytes (see the definition of ADLER32 below). The decompressor can use this identifier to determine which dictionary has been used by the compressor. Well, wow. I have to wonder why this wasn't more utilized, then. There are a ton of contexts (columnar data in databases, for example) where shared-dictionary-based compression might have helped a ton before now.

jacobgorm 3y ago

ZStd started out as ZHuff, which was first announced in 2008; see https://encode.su/threads/521-Zhuff-fast-compression https://encode.su/threads/521-Zhuff-fast-compression and http://fastcompression.blogspot.com/p/zhuff.html http://fastcompression.blogspot.com/p/zhuff.html I've been using LZ4 in commercial software since 2011, so unless I am some kind of luminary Google's compression experts must have been aware of both LZ4 and ZHuff when they started working on Brotli.

lifthrasiir 3y ago
By that reckoning I can claim that the origin of Brotli also dates back to 2012 or earlier. :-) Indeed, Brotli authors were also reponsible for the VP8 Lossless codec (2012), Pik (2017) and eventually JPEG XL (2019, in part). The entropy coding scheme is substantially similar among those codecs for this reason. But really, the large amount of differences between two formats rather suggest that they were designed for different uses in mind, which I believe relate to the maximum possible I/O. Formats designed for the web don't have to have an extremely fast decoder because I/O is the bottleneck: 100 MB/s should be enough, and the earned time can be invested to higher compression ratio which affects the decoder performance.
- adgjlsfhk1 3y ago
  otoh, they achieve pretty similar compression ratios and faster decode saves battery life and will let you get on to running the JavaScript earlier

miohtama 3y ago

The performance depends on the data. I have found that for English text and source code Brotli overperforms Zstd. Not sure if this because of the static dictionary or some other tweaks, though. However none of this is static, as both technologies are actively developed.

lifthrasiir 3y ago
Better decoding time performance, I should have said. The use of 2nd-order context model indeed makes Brotli more efficient in compression ratio for many kinds of data.

pmarreck 3y ago

Yes, but hasn't "separate compression dictionary" been a core feature of Zstd since its inception, at least under that name?

lifthrasiir 3y ago
Not even close, zlib had one. (Search for "FDICT" from RFC 1950.) What Zstandard did was a CLI to automatically generate a good enough dictionary from sample inputs.
- pmarreck 3y ago
  FDICT (Preset dictionary) If FDICT is set, a DICT dictionary identifier is present immediately after the FLG byte. The dictionary is a sequence of bytes which are initially fed to the compressor without producing any compressed output. DICT is the Adler-32 checksum of this sequence of bytes (see the definition of ADLER32 below). The decompressor can use this identifier to determine which dictionary has been used by the compressor. Well, wow. I have to wonder why this wasn't more utilized, then. There are a ton of contexts (columnar data in databases, for example) where shared-dictionary-based compression might have helped a ton before now.