3 ms·

Porting is generally very easy. * If you already have the compression algorithm tagged, through a file extension, or a field, then you can use that to dispatch

Porting is generally very easy. * If you already have the compression algorithm tagged, through a file extension, or a field, then you can use that to dispatch to the right decompression algorithm. * Zlib, gzip, xz, zstd, ... all have headers. If you are using zlib, and switching to zstd, you simply have to check the first 4 bytes for the zstd header using ZSTD_isFrame() [0], or attempting to decompress with zstd and if it fails fall back to the previous decompression algorithm. * The zstd CLI can decompress both zstd and zlib/gzip if compiled with zlib support. * Zstd provides a wrapper around the zlib API so you could transparently switch to zstd. [1] [0] https://github.com/facebook/zstd/blob/dev/lib/zstd.h#L1409 https://github.com/facebook/zstd/blob/dev/lib/zstd.h#L1409 [1] https://github.com/facebook/zstd/tree/dev/zlibWrapper https://github.com/facebook/zstd/tree/dev/zlibWrapper

jclay 8y ago
Great, thanks! Last question. On the blog you mention you are underway porting the internal code to replace Zlib with Zstd. Is there a reason you decided not to use the wrapper as a first pass to migrate all uses of Zlib to Zstd across the entire codebase?
- terrelln 8y ago
  There are a few reasons we haven't used the wrapper. * The larger services require tuning to get the best performance out of zstd, and we use some advanced options. * We have a "Managed Compression" library which does zstd dictionary compression, which doesn't work with the wrapper. * We have our own automatic decompression framework that handles many algorithms [0]. * A lot of use cases switched over from other algorithms than zlib. * A lot of use cases switched over to zstd organically, without our involvement, since it was such a clear win. [0] https://github.com/facebook/folly/blob/master/folly/compression/Compression.h#L522 https://github.com/facebook/folly/blob/master/folly/compress...

jclay 8y ago

Great, thanks! Last question. On the blog you mention you are underway porting the internal code to replace Zlib with Zstd. Is there a reason you decided not to use the wrapper as a first pass to migrate all uses of Zlib to Zstd across the entire codebase?

terrelln 8y ago
There are a few reasons we haven't used the wrapper. * The larger services require tuning to get the best performance out of zstd, and we use some advanced options. * We have a "Managed Compression" library which does zstd dictionary compression, which doesn't work with the wrapper. * We have our own automatic decompression framework that handles many algorithms [0]. * A lot of use cases switched over from other algorithms than zlib. * A lot of use cases switched over to zstd organically, without our involvement, since it was such a clear win. [0] https://github.com/facebook/folly/blob/master/folly/compression/Compression.h#L522 https://github.com/facebook/folly/blob/master/folly/compress...