4 ms·

That sounds fantastic. What is the porting process generally like? Are there any possibilities to create an API compatible wrapper to make it a drop in replacem

• Cyan4973 8y ago
  There is a zlib wrapper included in the project : https://github.com/facebook/zstd/tree/master/zlibWrapper https://github.com/facebook/zstd/tree/master/zlibWrapper
• megous 8y ago
  Rather simple. For example here's my port of qemu to use zstd compression algorithm for QCOW2 images, instead of zlib: https://megous.com/git/qemu-zstd/commit/?id=45df9c0510e737e6b2538401ec41f0f19f881b06 https://megous.com/git/qemu-zstd/commit/?id=45df9c0510e737e6...
  • kstrauser 8y ago
    Awesome! What kind of metrics differences have you seen from the change?
    • megous 8y ago
      Much faster and/or better compression/decompression of images (tunable of course, depending on what level you set in the code), and also faster startup of VMs. I haven't measured exactly. I'd guess ~30% in my case. But all this depends on where the bottleneck is in any particular case. My VMs are on HDD, so increased decompression speed doesn't matter that much, but reduced size helps reading the necessary data faster. Linux VMs seem to be more compressible than Windows ones. It's certainly better than zlib in any case.
• terrelln 8y ago
  Porting is generally very easy. * If you already have the compression algorithm tagged, through a file extension, or a field, then you can use that to dispatch to the right decompression algorithm. * Zlib, gzip, xz, zstd, ... all have headers. If you are using zlib, and switching to zstd, you simply have to check the first 4 bytes for the zstd header using ZSTD_isFrame() [0], or attempting to decompress with zstd and if it fails fall back to the previous decompression algorithm. * The zstd CLI can decompress both zstd and zlib/gzip if compiled with zlib support. * Zstd provides a wrapper around the zlib API so you could transparently switch to zstd. [1] [0] https://github.com/facebook/zstd/blob/dev/lib/zstd.h#L1409 https://github.com/facebook/zstd/blob/dev/lib/zstd.h#L1409 [1] https://github.com/facebook/zstd/tree/dev/zlibWrapper https://github.com/facebook/zstd/tree/dev/zlibWrapper
  • jclay 8y ago
    Great, thanks! Last question. On the blog you mention you are underway porting the internal code to replace Zlib with Zstd. Is there a reason you decided not to use the wrapper as a first pass to migrate all uses of Zlib to Zstd across the entire codebase?
    • terrelln 8y ago
      There are a few reasons we haven't used the wrapper. * The larger services require tuning to get the best performance out of zstd, and we use some advanced options. * We have a "Managed Compression" library which does zstd dictionary compression, which doesn't work with the wrapper. * We have our own automatic decompression framework that handles many algorithms [0]. * A lot of use cases switched over from other algorithms than zlib. * A lot of use cases switched over to zstd organically, without our involvement, since it was such a clear win. [0] https://github.com/facebook/folly/blob/master/folly/compression/Compression.h#L522 https://github.com/facebook/folly/blob/master/folly/compress...