13 ms·

Consider using Zstandard and/or LZ4 instead of Deflate

191 points by marklit 1y ago

privatelypublic 1y ago
Does deflate lead the pack in any metric at all anymore? Only one I can think of is extreme low spec compression (microcontrollers).
- adgjlsfhk1 1y ago
  Even there, LZ4 is probably better.
  - hinkley 1y ago
    You think LZ4 is more portable than zlib? I'm gonna need some citations on that. zlib is 30 years old, according to Wikipedia. And that's technically wrong since 'zlib' was factored out of gzip (nearly 33 years old) for use in libpng, which is also 30 years old.
    - adgjlsfhk1 1y ago
      not more portable, but probably faster in resource constrained environments
    - duskwuff 1y ago
      A basic LZ4 decompressor is on the order of a few dozen lines of code. It's exceptionally easy to implement.
    - moonshadow565 1y ago
      Just because its old doesn't mean it's more portable. If anything it makes me think it's even less portable.
- JoshTriplett 1y ago
  The only metric deflate leads on is widespread support. By any other metric, it has been superseded.
  - atiedebee 1y ago
    I'd assume memory usage as well, because it has a tiny context window compared to zstd
    - JoshTriplett 1y ago
      You can change the context window of zstd if you want. But yes, the default context window size for zstd is 8MB, versus 32k.
zX41ZdbW 1y ago
Very reasonable. I've recently experimented with the methods of serving bitmaps out of the database in my project[1]. One option was to generate PNG on the fly, but simply outputting an array of pixel color values over HTTP with Content-Encoding: zstd has won over PNG. Combined with the 2D-delta-encoding as in PNG, it will be even better. [1] https://adsb.exposed/ https://adsb.exposed/
arp242 1y ago
Comparison of "zpng" (PNG wth zstd) and WebP lossless, with current PNG. From https://github.com/WangXuan95/Image-Compression-Benchmark https://github.com/WangXuan95/Image-Compression-Benchmark : Compressed format Compressed size (bytes) Compress Time Decompress Time WEBP (lossless m5) 1,475,908,700 1,112 49 WEBP (lossless m1) 1,496,478,650 720 37 ZPNG (-19) 1,703,197,687 1,529 20 ZPNG 1,755,786,378 26 24 PNG (optipng -o5) 1,899,273,578 27,680 26 PNG (optipng -o2) 1,905,215,734 4,395 27 PNG (optimize=True) 1,935,713,540 1,120 29 PNG (optimize=False) 2,003,016,524 335 34 Doesn't really seem worth it? It doesn't compress better, and only slightly faster in decompression time.
- bobmcnamara 1y ago
  Am I reading those numbers right? That's like 25x faster compression than WEBP-M1, there's probably a use case for that.
  - arp242 1y ago
    The numbers seem small enough that it will rarely matter, but I suppose there might be a use case somewhere? But lets be real here: this is basically just a new image format. With more code to maintain, fresh new exciting zero-days, and all of that. You need a strong use case to justify that, and "already fast encode is now faster" is probably not it.
    - scott_w 1y ago
      I don’t think it’s quite as bad, though? It’s using a known compression library that (from reading other comments) has seen use and testing. The rest of PNG would remain unchanged, as the decompression format is a plugin. I know it needs to be battle tested as a single entity but it’s not the same as writing a new image format from scratch.
    - realityking 1y ago
      Considering both zstandard and PNG are already web facing technologies, would the combination of both really increase the attack surface?
e-topy 1y ago
Instead of using a new PNG standard, I'd still rather use JPEG XL just because it has progressive decoding. And you know, whilst looking like png, being as small as webp, supporting HDR and animations, and having even faster decoding speed. https://dennisforbes.ca/articles/jpegxl_just_won_the_image_wars.html https://dennisforbes.ca/articles/jpegxl_just_won_the_image_w...
- jchw 1y ago
  JPEG XL definitely has advantages over PNG but there is one serious seemingly insurmountable obstacle: https://caniuse.com/jpegxl https://caniuse.com/jpegxl Nothing really supports it. Latest Safari at least has support for it not feature-flagged or anything, but it doesn't support JPEG XL animations. To be fair, nothing supports a theoretical PNG with Zstandard compression either. While that would be an obstacle to using PNG with Zstandard for a while, I kinda suspect it wouldn't be that long of a wait because many things that support PNG today also support Zstandard anyways, so it's not a huge leap for them to add Zstandard support to their PNG codecs. Adding JPEG-XL support is a relatively bigger ticket that has struggled to cross the finish line. The thing I'm really surprised about is that you still can't use arithmetic coding with JPEG. I think the original reason is due to patents, but I don't think there have been active patents around that in years now.
  - bawolff 1y ago
    > The thing I'm really surprised about is that you still can't use arithmetic coding with JPEG. I was under the impression libjpeg added support in 2009 (in v7). I'd assume most things support it by now.
    - jchw 1y ago
      Believe it or not, last I checked, many browsers and some other software (file managers, etc.) still couldn't do anything with JPEG files that have arithmetic coding. Apparently, although I haven't tried this myself, Adobe Photoshop also specifically doesn't support it.
bawolff 1y ago
I think there is a benefit to knowing that if you have a png file it works everywhere that supports png. Better to make the back compat breaks be entirely new formats.
encom 1y ago
(2021) In my opinion PNG doesn't need fixing. Being ancient is a feature. Everything supports it. As much as I appreciate the nerdy exercise, PNG is fine as it is. My only gripe is that some software writes needlessly bloated files (like adding a useless alpha channel, when it's not needed). I wish we didn't need tools like OptiPNG etc.
- heinrich5991 1y ago
  Most of the comments on that issue are from this year.
- ori_b 1y ago
  Yes. One of the best features of png is that I don't have to wonder if it's going to work somewhere. Throwing that away in favor of a bit of premature optimization seems like a big loss. Especially as this wouldn't be the only modernized image compression format out there. Why use this over, eg, lossless webp? I don't think I have ever noticed the decode time of a png.
  - encom 1y ago
    >I don't think I have ever noticed the decode time of a png. When it was developed, 200 Mhz Pentium was the current tech. Back of the envelope numbers (ie. chatgpt) says my current desktop CPU (i7-14700K) decodes 7000x faster.
    - atiedebee 1y ago
      PNG is fully serial, no? So 7000x seems like a big over exaggeration
      - encom 1y ago
        I assumed decoding multiple files in parallel. I think very few websites have just one PNG file.
willvarfar 1y ago
We ought consider using QOI instead. QOI is often equivalent or better compression than PNG, _before_ you even compress it with something like LZ4 etc. Compressing QOI with something like LZ4 would generally outperform PNG.
- adgjlsfhk1 1y ago
  QOI has some pretty major downsides. it only supports 8 bit SRGB, and is optimized for images with 8 bit transparency. Also, the handsome it uses seems to harm compression when entropy compression is used. Also, the focus on streaming means that the algorithm can't take advantage of 2d locality. QOI is really cool, but I think the author cut the final version of the spec too early, and intentionally closed it off to a future version with more improvements. With another year or 2 of development, I think it probably works have become ~10% more efficient and suitable for more usecases.
- nigeltao 1y ago
  > Compressing QOI with something like LZ4 would generally outperform PNG. https://github.com/nigeltao/qoir https://github.com/nigeltao/qoir has some numbers comparing QOIR (which is QOI-inspired-with-LZ4) vs PNG. QOIR has better decode speed and comparable compression ratio (depending on which PNG encoder you use). QOIR's numbers are also roughly similar to ZPNG.
HocusLocus 1y ago
The reason we have a world full of .gif today is that the .png committee rejected animation back when everyone was saying PNG would be the "GIF killer". Just sayin'. Don't hold your breath.
- edoceo 1y ago
  Remember this: https://burnallgifs.org/ https://burnallgifs.org/
hughw 1y ago
Related: what's the status of content negotiation? Any browsers use it seriously, and has it been successful? If so, then why not zpng.
jasonthorsness 1y ago
One of the interesting features of ZStandard is the support for external dictionaries. It supports "training" a dictionary on a set of samples, of whatever size (16KiB, 64 KiB, etc.), then applying that dictionary as a separate input file for compression and decompression. This lets you compress short content much more effectively. I doubt it would apply to PNG because of the length and content doesn't seem to be dictionary-friendly, but it would be interesting to try from some giant collection of scraped PNGs. This approach was important enough for Brotli to include a "built-in" dictionary covering HTML.
- DefineOutside 1y ago
  This has been applied to minecraft region files in a fork of paper, which is a type of minecraft server. https://github.com/UltraVanilla/paper-zstd/blob/main/patches/server/8002-Zstd-compression.patch https://github.com/UltraVanilla/paper-zstd/blob/main/patches... from the author of this patch on discord - the level 9 for compression isn't practical and is too slow for a real production server but it does show the effectiveness of zstd with a shared dictionary. So you start off with a 755.2 MiB world (in this test, it is a section of an existing DEFLATE-compressed world that has been lived in for a while). If you recreate its regions it will compact it down to 695.1 MiB You set region-file-compression=lz4 and run --recreateRegionFiles and it turns into a 998.9 MiB world. Makes sense, worse compression ratios but less CPU is what mojang documented in the changelog. Neat, but I'm confused as to what the benefits are as I/O increasingly becomes the more constrained thing nowadays. This is just a brief detour from what I'm really trying to test You set region-file-compression=none and it turns into a 3583.0 MiB world. The largest region file in this sample was 57 MiB Now, you take this world, and compress each of the region files individually using zstd -9, so that the region files are now .mca.zst files. And you get a world that is 390.2 MiB
  - immibis 1y ago
    Note that each region file contains 1024 chunks that are designed to be (but probably aren't) accessed at random, so compressing a region file is like a solid archive with a solid block size of 1024 files.
citrin_ru 1y ago
ZSTD is a great compression algorithm but an important PNG (v1.2) advantage is that implementations are available in almost all actively used operating systems and in most popular languages. The same cannot be said about ZSTD with very few implementations except https://github.com/facebook/zstd https://github.com/facebook/zstd I'm not even sure there is a good pure Java (no JNI) and Go (without Cgo) implementations for ZSTD. And it definitely would require more powerful hardware - some micro-controllers which can use PNG are too small for ZSTD.
- jonathanoliver 1y ago
  For Go, we've been using this library which supports ZSTD. https://github.com/klauspost/compress https://github.com/klauspost/compress
pornel 1y ago
The developer who asked for the faster compression formats has later solved the problem himself: https://github.com/richgel999/fpng https://github.com/richgel999/fpng It turns out that deflate can be much faster when implemented specifically for PNG data, instead general-purpose compression (while still remaining 100%-standard-compatible).
- hyperman1 1y ago
  Note he also expects a worse compression as tradeoff. I think he implements RLE in terms of zlib: [...]Deflate compressor which was optimized for simplicity over high ratios. The "parser" only supports RLE matches using a match distance of 3/4 bytes, [...]
  - deleted 1y ago
    [deleted]
- spider-mario 1y ago
  In a similar vein: https://github.com/veluca93/fpnge https://github.com/veluca93/fpnge https://x.com/LucaVersari3/status/1485971553892323333 https://x.com/LucaVersari3/status/1485971553892323333
physicles 1y ago
Years ago I built a slippy map (google maps-style) tile server for non-image data. One of the use cases was to be able to quickly sample elevation data at an arbitrary lat/lng in a few milliseconds. The data set is so large that you obviously want to delay decompression as long as possible. I turned to 16-bit grayscale PNGs, because PNG is a widely-used a standard. These were fine, but I wasn't close to my target latency. After some experimentation, I was surprised to discover two things: 1. Deflate, this widely used standard, is just super slow compared to other algorithms (at least, in Go's native PNG decoder) 2. Tool and library support for formats other than ARGB32 is pretty lacking So I turned to some bespoke integer compression algorithms like Snappy and Simple8b, and got a 20x decompression speedup, with maybe 20% worse compression ratios. This, along with some other tricks, got me where I needed to go. Maybe there are some niche file formats out there that would've solved this. But in total we're not even talking about that much code, so it was easier to just invent my own.
- anitil 1y ago
  I was impressed at how much Jart squeezed out of simple run-length-encoding of their binaries, and decoding only took 14 bytes of code [0] [0] https://justine.lol/sizetricks/#rle https://justine.lol/sizetricks/#rle
  - hyperman1 1y ago
    When looking at that code: aa 1: stosb e2 fd loop 1b Why doesnt jart simply use rep stosb? It would take 1 less byte and even be slightly more idiomatic.
    - anitil 1y ago
      Ok I'm not an expert here, but it seems they do [0] and in the pr [1] the comment says it's now 13 bytes (I _believe_ previously it was incorrectly stating 17 which it should have been 14) [0] https://github.com/jart/cosmopolitan/blob/master/libc/nexgen32e/rldecode.S#L36 https://github.com/jart/cosmopolitan/blob/master/libc/nexgen... [1] https://github.com/jart/cosmopolitan/commit/e96aceae4112163009bca459acfce68824806118 https://github.com/jart/cosmopolitan/commit/e96aceae41121630...
electroly 1y ago
Zstandard gets a lot of attention but I love LZ4 for speed. At least in the .NET world we have a fast LZ4 compressor/decompressor that is barely slower than memcpy. If you're already copying the data, you might as well LZ4 it. It's great over the wire when I control both the server and the client.

191 points by marklit 1y ago

Does deflate lead the pack in any metric at all anymore? Only one I can think of is extreme low spec compression (microcontrollers).

adgjlsfhk1 1y ago
Even there, LZ4 is probably better.
- hinkley 1y ago
  You think LZ4 is more portable than zlib? I'm gonna need some citations on that. zlib is 30 years old, according to Wikipedia. And that's technically wrong since 'zlib' was factored out of gzip (nearly 33 years old) for use in libpng, which is also 30 years old.
  - adgjlsfhk1 1y ago
    not more portable, but probably faster in resource constrained environments
  - duskwuff 1y ago
    A basic LZ4 decompressor is on the order of a few dozen lines of code. It's exceptionally easy to implement.
  - moonshadow565 1y ago
    Just because its old doesn't mean it's more portable. If anything it makes me think it's even less portable.
JoshTriplett 1y ago
The only metric deflate leads on is widespread support. By any other metric, it has been superseded.
- atiedebee 1y ago
  I'd assume memory usage as well, because it has a tiny context window compared to zstd
  - JoshTriplett 1y ago
    You can change the context window of zstd if you want. But yes, the default context window size for zstd is 8MB, versus 32k.

zX41ZdbW 1y ago

Very reasonable. I've recently experimented with the methods of serving bitmaps out of the database in my project[1]. One option was to generate PNG on the fly, but simply outputting an array of pixel color values over HTTP with Content-Encoding: zstd has won over PNG. Combined with the 2D-delta-encoding as in PNG, it will be even better. [1] https://adsb.exposed/ https://adsb.exposed/

arp242 1y ago

Comparison of "zpng" (PNG wth zstd) and WebP lossless, with current PNG. From https://github.com/WangXuan95/Image-Compression-Benchmark https://github.com/WangXuan95/Image-Compression-Benchmark : Compressed format Compressed size (bytes) Compress Time Decompress Time WEBP (lossless m5) 1,475,908,700 1,112 49 WEBP (lossless m1) 1,496,478,650 720 37 ZPNG (-19) 1,703,197,687 1,529 20 ZPNG 1,755,786,378 26 24 PNG (optipng -o5) 1,899,273,578 27,680 26 PNG (optipng -o2) 1,905,215,734 4,395 27 PNG (optimize=True) 1,935,713,540 1,120 29 PNG (optimize=False) 2,003,016,524 335 34 Doesn't really seem worth it? It doesn't compress better, and only slightly faster in decompression time.

bobmcnamara 1y ago
Am I reading those numbers right? That's like 25x faster compression than WEBP-M1, there's probably a use case for that.
- arp242 1y ago
  The numbers seem small enough that it will rarely matter, but I suppose there might be a use case somewhere? But lets be real here: this is basically just a new image format. With more code to maintain, fresh new exciting zero-days, and all of that. You need a strong use case to justify that, and "already fast encode is now faster" is probably not it.
  - scott_w 1y ago
    I don’t think it’s quite as bad, though? It’s using a known compression library that (from reading other comments) has seen use and testing. The rest of PNG would remain unchanged, as the decompression format is a plugin. I know it needs to be battle tested as a single entity but it’s not the same as writing a new image format from scratch.
  - realityking 1y ago
    Considering both zstandard and PNG are already web facing technologies, would the combination of both really increase the attack surface?

e-topy 1y ago

Instead of using a new PNG standard, I'd still rather use JPEG XL just because it has progressive decoding. And you know, whilst looking like png, being as small as webp, supporting HDR and animations, and having even faster decoding speed. https://dennisforbes.ca/articles/jpegxl_just_won_the_image_wars.html https://dennisforbes.ca/articles/jpegxl_just_won_the_image_w...

jchw 1y ago
JPEG XL definitely has advantages over PNG but there is one serious seemingly insurmountable obstacle: https://caniuse.com/jpegxl https://caniuse.com/jpegxl Nothing really supports it. Latest Safari at least has support for it not feature-flagged or anything, but it doesn't support JPEG XL animations. To be fair, nothing supports a theoretical PNG with Zstandard compression either. While that would be an obstacle to using PNG with Zstandard for a while, I kinda suspect it wouldn't be that long of a wait because many things that support PNG today also support Zstandard anyways, so it's not a huge leap for them to add Zstandard support to their PNG codecs. Adding JPEG-XL support is a relatively bigger ticket that has struggled to cross the finish line. The thing I'm really surprised about is that you still can't use arithmetic coding with JPEG. I think the original reason is due to patents, but I don't think there have been active patents around that in years now.
- bawolff 1y ago
  > The thing I'm really surprised about is that you still can't use arithmetic coding with JPEG. I was under the impression libjpeg added support in 2009 (in v7). I'd assume most things support it by now.
  - jchw 1y ago
    Believe it or not, last I checked, many browsers and some other software (file managers, etc.) still couldn't do anything with JPEG files that have arithmetic coding. Apparently, although I haven't tried this myself, Adobe Photoshop also specifically doesn't support it.

bawolff 1y ago

I think there is a benefit to knowing that if you have a png file it works everywhere that supports png. Better to make the back compat breaks be entirely new formats.

encom 1y ago

(2021) In my opinion PNG doesn't need fixing. Being ancient is a feature. Everything supports it. As much as I appreciate the nerdy exercise, PNG is fine as it is. My only gripe is that some software writes needlessly bloated files (like adding a useless alpha channel, when it's not needed). I wish we didn't need tools like OptiPNG etc.

heinrich5991 1y ago
Most of the comments on that issue are from this year.
ori_b 1y ago
Yes. One of the best features of png is that I don't have to wonder if it's going to work somewhere. Throwing that away in favor of a bit of premature optimization seems like a big loss. Especially as this wouldn't be the only modernized image compression format out there. Why use this over, eg, lossless webp? I don't think I have ever noticed the decode time of a png.
- encom 1y ago
  >I don't think I have ever noticed the decode time of a png. When it was developed, 200 Mhz Pentium was the current tech. Back of the envelope numbers (ie. chatgpt) says my current desktop CPU (i7-14700K) decodes 7000x faster.
  - atiedebee 1y ago
    PNG is fully serial, no? So 7000x seems like a big over exaggeration
    - encom 1y ago
      I assumed decoding multiple files in parallel. I think very few websites have just one PNG file.

willvarfar 1y ago

We ought consider using QOI instead. QOI is often equivalent or better compression than PNG, _before_ you even compress it with something like LZ4 etc. Compressing QOI with something like LZ4 would generally outperform PNG.

adgjlsfhk1 1y ago
QOI has some pretty major downsides. it only supports 8 bit SRGB, and is optimized for images with 8 bit transparency. Also, the handsome it uses seems to harm compression when entropy compression is used. Also, the focus on streaming means that the algorithm can't take advantage of 2d locality. QOI is really cool, but I think the author cut the final version of the spec too early, and intentionally closed it off to a future version with more improvements. With another year or 2 of development, I think it probably works have become ~10% more efficient and suitable for more usecases.
nigeltao 1y ago
> Compressing QOI with something like LZ4 would generally outperform PNG. https://github.com/nigeltao/qoir https://github.com/nigeltao/qoir has some numbers comparing QOIR (which is QOI-inspired-with-LZ4) vs PNG. QOIR has better decode speed and comparable compression ratio (depending on which PNG encoder you use). QOIR's numbers are also roughly similar to ZPNG.

HocusLocus 1y ago

The reason we have a world full of .gif today is that the .png committee rejected animation back when everyone was saying PNG would be the "GIF killer". Just sayin'. Don't hold your breath.

edoceo 1y ago
Remember this: https://burnallgifs.org/ https://burnallgifs.org/

hughw 1y ago

Related: what's the status of content negotiation? Any browsers use it seriously, and has it been successful? If so, then why not zpng.

jasonthorsness 1y ago

One of the interesting features of ZStandard is the support for external dictionaries. It supports "training" a dictionary on a set of samples, of whatever size (16KiB, 64 KiB, etc.), then applying that dictionary as a separate input file for compression and decompression. This lets you compress short content much more effectively. I doubt it would apply to PNG because of the length and content doesn't seem to be dictionary-friendly, but it would be interesting to try from some giant collection of scraped PNGs. This approach was important enough for Brotli to include a "built-in" dictionary covering HTML.

DefineOutside 1y ago
This has been applied to minecraft region files in a fork of paper, which is a type of minecraft server. https://github.com/UltraVanilla/paper-zstd/blob/main/patches/server/8002-Zstd-compression.patch https://github.com/UltraVanilla/paper-zstd/blob/main/patches... from the author of this patch on discord - the level 9 for compression isn't practical and is too slow for a real production server but it does show the effectiveness of zstd with a shared dictionary. So you start off with a 755.2 MiB world (in this test, it is a section of an existing DEFLATE-compressed world that has been lived in for a while). If you recreate its regions it will compact it down to 695.1 MiB You set region-file-compression=lz4 and run --recreateRegionFiles and it turns into a 998.9 MiB world. Makes sense, worse compression ratios but less CPU is what mojang documented in the changelog. Neat, but I'm confused as to what the benefits are as I/O increasingly becomes the more constrained thing nowadays. This is just a brief detour from what I'm really trying to test You set region-file-compression=none and it turns into a 3583.0 MiB world. The largest region file in this sample was 57 MiB Now, you take this world, and compress each of the region files individually using zstd -9, so that the region files are now .mca.zst files. And you get a world that is 390.2 MiB
- immibis 1y ago
  Note that each region file contains 1024 chunks that are designed to be (but probably aren't) accessed at random, so compressing a region file is like a solid archive with a solid block size of 1024 files.

citrin_ru 1y ago

ZSTD is a great compression algorithm but an important PNG (v1.2) advantage is that implementations are available in almost all actively used operating systems and in most popular languages. The same cannot be said about ZSTD with very few implementations except https://github.com/facebook/zstd https://github.com/facebook/zstd I'm not even sure there is a good pure Java (no JNI) and Go (without Cgo) implementations for ZSTD. And it definitely would require more powerful hardware - some micro-controllers which can use PNG are too small for ZSTD.

jonathanoliver 1y ago
For Go, we've been using this library which supports ZSTD. https://github.com/klauspost/compress https://github.com/klauspost/compress

pornel 1y ago

The developer who asked for the faster compression formats has later solved the problem himself: https://github.com/richgel999/fpng https://github.com/richgel999/fpng It turns out that deflate can be much faster when implemented specifically for PNG data, instead general-purpose compression (while still remaining 100%-standard-compatible).

hyperman1 1y ago
Note he also expects a worse compression as tradeoff. I think he implements RLE in terms of zlib: [...]Deflate compressor which was optimized for simplicity over high ratios. The "parser" only supports RLE matches using a match distance of 3/4 bytes, [...]
- deleted 1y ago
  [deleted]
spider-mario 1y ago
In a similar vein: https://github.com/veluca93/fpnge https://github.com/veluca93/fpnge https://x.com/LucaVersari3/status/1485971553892323333 https://x.com/LucaVersari3/status/1485971553892323333

physicles 1y ago

Years ago I built a slippy map (google maps-style) tile server for non-image data. One of the use cases was to be able to quickly sample elevation data at an arbitrary lat/lng in a few milliseconds. The data set is so large that you obviously want to delay decompression as long as possible. I turned to 16-bit grayscale PNGs, because PNG is a widely-used a standard. These were fine, but I wasn't close to my target latency. After some experimentation, I was surprised to discover two things: 1. Deflate, this widely used standard, is just super slow compared to other algorithms (at least, in Go's native PNG decoder) 2. Tool and library support for formats other than ARGB32 is pretty lacking So I turned to some bespoke integer compression algorithms like Snappy and Simple8b, and got a 20x decompression speedup, with maybe 20% worse compression ratios. This, along with some other tricks, got me where I needed to go. Maybe there are some niche file formats out there that would've solved this. But in total we're not even talking about that much code, so it was easier to just invent my own.

anitil 1y ago
I was impressed at how much Jart squeezed out of simple run-length-encoding of their binaries, and decoding only took 14 bytes of code [0] [0] https://justine.lol/sizetricks/#rle https://justine.lol/sizetricks/#rle
- hyperman1 1y ago
  When looking at that code: aa 1: stosb e2 fd loop 1b Why doesnt jart simply use rep stosb? It would take 1 less byte and even be slightly more idiomatic.
  - anitil 1y ago
    Ok I'm not an expert here, but it seems they do [0] and in the pr [1] the comment says it's now 13 bytes (I _believe_ previously it was incorrectly stating 17 which it should have been 14) [0] https://github.com/jart/cosmopolitan/blob/master/libc/nexgen32e/rldecode.S#L36 https://github.com/jart/cosmopolitan/blob/master/libc/nexgen... [1] https://github.com/jart/cosmopolitan/commit/e96aceae4112163009bca459acfce68824806118 https://github.com/jart/cosmopolitan/commit/e96aceae41121630...

electroly 1y ago

Zstandard gets a lot of attention but I love LZ4 for speed. At least in the .NET world we have a fast LZ4 compressor/decompressor that is barely slower than memcpy. If you're already copying the data, you might as well LZ4 it. It's great over the wire when I control both the server and the client.