3 ms·

if you throw lz4 or zstd on top of it, of it, it actually can beat png

oefrha 4y ago
So, if you throw another >100KB decoder on top of the 1KB decoder, you can beat png (haven’t verified). What’s the point.
- adgjlsfhk1 4y ago
  lz4 is also pretty small https://nigeltao.github.io/blog/2022/qoir.html https://nigeltao.github.io/blog/2022/qoir.html is an implementation of QOI with some tweaks+lz4 and it's still pretty tiny
- masklinn 4y ago
  Also I could see it with zstd (which is a lot more than 100k, though very fast indeed) but I’m a bit doubtful about lz4, it’s generally worse than deflate (which png uses) at compression and qoi lacks filtering.
  - duskwuff 4y ago
    QOI's QOI_OP_DIFF and QOI_OP_LUMA opcodes can act like horizontal filtering once the results are fed into a stream compressor. There's nothing to provide vertical filtering, though, which is a pretty huge limitation. I wonder if you'd get better compression using a zigzag scan?
    - adgjlsfhk1 4y ago
      there was some work done to find better scan arrangements. unfortunately, is pretty hard to do so without storing the previous row in memory which has a pretty large effect on speed
- robinsonb5 4y ago
  I once ported an LZ4 decoder from 68k to my own toy CPU design - the lz4_depack routine weighs in at a whopping 72 bytes. https://github.com/robinsonb5/EightThirtyTwoDemos/blob/master/LZ4/Firmware/lz4.asm https://github.com/robinsonb5/EightThirtyTwoDemos/blob/maste...
duskwuff 4y ago
> if you throw lz4 or zstd on top of it, of it, it actually can beat png PNG is literally just "perform some simple filtering on the image data, then run that through zlib and wrap it with some metadata". It shouldn't come as any great surprise that you can outperform PNG with a newer stream compressor. You'd probably get even better results by using PNG's filters instead of QOI.
- nh2 4y ago
  > You'd probably get even better results by using PNG's filters instead of QOI Did anybody ever try this? It would be quite interesting and does not seem too difficult. The only thing I could find is https://github.com/catid/Zpng https://github.com/catid/Zpng which does not use the normal PNG filtering.
- mananaysiempre 4y ago
  LZ4[1] intentionally compresses worse than Deflate—where Deflate is LZ77 with Huffman in a bitstream, LZ4 is just LZ77 with a good choice of octet-aligned encoding. It’s “we’re saturating Gigabit Ethernet and have some compute left over” compression, not “we need to squeeze our backups as much as possible” compression. If filtering+LZ4 does better than Deflate, then you’ve chosen the filter well. Curiously, for some kinds of serialized binary data varints or even zero-packing[2] can be a good enough filter[3]. [1] https://lz4.github.io/lz4/ https://lz4.github.io/lz4/ [2] https://capnproto.org/encoding.html#packing https://capnproto.org/encoding.html#packing [3] https://news.ycombinator.com/item?id=24073752 https://news.ycombinator.com/item?id=24073752

So, if you throw another >100KB decoder on top of the 1KB decoder, you can beat png (haven’t verified). What’s the point.

adgjlsfhk1 4y ago
lz4 is also pretty small https://nigeltao.github.io/blog/2022/qoir.html https://nigeltao.github.io/blog/2022/qoir.html is an implementation of QOI with some tweaks+lz4 and it's still pretty tiny
masklinn 4y ago
Also I could see it with zstd (which is a lot more than 100k, though very fast indeed) but I’m a bit doubtful about lz4, it’s generally worse than deflate (which png uses) at compression and qoi lacks filtering.
- duskwuff 4y ago
  QOI's QOI_OP_DIFF and QOI_OP_LUMA opcodes can act like horizontal filtering once the results are fed into a stream compressor. There's nothing to provide vertical filtering, though, which is a pretty huge limitation. I wonder if you'd get better compression using a zigzag scan?
  - adgjlsfhk1 4y ago
    there was some work done to find better scan arrangements. unfortunately, is pretty hard to do so without storing the previous row in memory which has a pretty large effect on speed
robinsonb5 4y ago
I once ported an LZ4 decoder from 68k to my own toy CPU design - the lz4_depack routine weighs in at a whopping 72 bytes. https://github.com/robinsonb5/EightThirtyTwoDemos/blob/master/LZ4/Firmware/lz4.asm https://github.com/robinsonb5/EightThirtyTwoDemos/blob/maste...

duskwuff 4y ago

> if you throw lz4 or zstd on top of it, of it, it actually can beat png PNG is literally just "perform some simple filtering on the image data, then run that through zlib and wrap it with some metadata". It shouldn't come as any great surprise that you can outperform PNG with a newer stream compressor. You'd probably get even better results by using PNG's filters instead of QOI.

nh2 4y ago
> You'd probably get even better results by using PNG's filters instead of QOI Did anybody ever try this? It would be quite interesting and does not seem too difficult. The only thing I could find is https://github.com/catid/Zpng https://github.com/catid/Zpng which does not use the normal PNG filtering.
mananaysiempre 4y ago
LZ4[1] intentionally compresses worse than Deflate—where Deflate is LZ77 with Huffman in a bitstream, LZ4 is just LZ77 with a good choice of octet-aligned encoding. It’s “we’re saturating Gigabit Ethernet and have some compute left over” compression, not “we need to squeeze our backups as much as possible” compression. If filtering+LZ4 does better than Deflate, then you’ve chosen the filter well. Curiously, for some kinds of serialized binary data varints or even zero-packing[2] can be a good enough filter[3]. [1] https://lz4.github.io/lz4/ https://lz4.github.io/lz4/ [2] https://capnproto.org/encoding.html#packing https://capnproto.org/encoding.html#packing [3] https://news.ycombinator.com/item?id=24073752 https://news.ycombinator.com/item?id=24073752