4 ms·

This is why for compression tests, they incorporate the size of everything needed to decompress the file. You can compress down to 4.97KB all you want, just in

by Dwedit 4y ago

This is why for compression tests, they incorporate the size of everything needed to decompress the file. You can compress down to 4.97KB all you want, just include the 4GB trained model.

Gigachad 4y ago
Do you also include the library to render a jpeg? And maybe the whole OS required to display it on your screen? There are very many uses where any fixed overhead is meaningless. Imagine archiving billions of images for long term storage. The 4GB model quickly becomes meaningless.
- stavros 4y ago
  > Do you also include the library to render a jpeg? And maybe the whole OS required to display it on your screen? No, what does that have to do with reconstructing the original data? If the fixed overhead works for you, that's fine, but including it is not meaningless.
- 112233 4y ago
  Fixed overheads are never meaningless. O(n^2) algorithm that processes your data in 5s is faster on your data than O(log n) that takes 20 hours. Long term storage of billions of images is meaningless, if it takes billions of years to archive these images.
  - Gigachad 4y ago
    It’s a one time cost rather than per image. You need the 4GB model only once and then you can uncompress unlimited images.
    - 112233 4y ago
      Yes, but each image needs access to this 4GB (actually, I have no idea how much RAM it takes up), plus whatever the working set size is. It is a non-trivial overhead that really limits throughput of your system, so you can process less images in parallel, so compressing billion of images in reasonable time suddenly may cost much more than the amount of storage it would save, compared to other methods.
janekm 4y ago
Is that true? I have never seen this done for any image compression comparisons that I have seen (i.e. only data that is specific to the image that is being compressed is included, not standard tables that are always used by the algorithm like the quantisation tables used in JPG compression)
- fsiefken 4y ago
  For text compression benchmarks it's done http://mattmahoney.net/dc/text.html http://mattmahoney.net/dc/text.html Matt doesn't do this on the Silesia corpus compression benchmark, even though it would make sense there as well: http://mattmahoney.net/dc/silesia.html http://mattmahoney.net/dc/silesia.html So a compressor of a few gigabyte would make sense if you would have a set of pictures of more then a few gigabyte. It's a bit similar to preprocessing text compression with a dictionary and adding the dictionary to the extractor to squeeze a bit more bytes.
  - goombacloud 4y ago
    By the way, the leading nncp in the LTCB (text.html) "is a free, experimental file compressor by Fabrice Bellard, released May 8, 2019" :)
- jerf 4y ago
  Yes, it is done all the time. However, several people here are conflating "best compression as determined for a competition" and "best compression for use in the real world". There is an important relationship between them, absolutely, but in the real world we do not download custom decoders for every bit of compressed content. Just because there is a competition that quite correctly measures the entire size of the decompressor and encoded content does not mean that is now the only valid metric to measure decompression performance. The competitions use that metric for good and valid reasons, but those good and valid reasons are only vaguely correlated to the issues faced in the normal world. (Among the reasons why competitions must include the size of the decoder is that without that the answer is trivial; I define all your test inputs as a simple enumeration of them and my decoder hard-codes the output as the test values. This is trivially the optimal algorithm, making competition useless. If you could have a real-world encoder that worked this well, and had the storage to implement it, it would be optimal, but you can't possibly store all possible messages. For a humorous demonstration of this encoding method, see the classic joke: https://onemansblog.com/2010/05/18/prison-joke/ https://onemansblog.com/2010/05/18/prison-joke/ )

by Dwedit 4y ago

This is why for compression tests, they incorporate the size of everything needed to decompress the file. You can compress down to 4.97KB all you want, just include the 4GB trained model.

Gigachad 4y ago

Do you also include the library to render a jpeg? And maybe the whole OS required to display it on your screen? There are very many uses where any fixed overhead is meaningless. Imagine archiving billions of images for long term storage. The 4GB model quickly becomes meaningless.

stavros 4y ago
> Do you also include the library to render a jpeg? And maybe the whole OS required to display it on your screen? No, what does that have to do with reconstructing the original data? If the fixed overhead works for you, that's fine, but including it is not meaningless.
112233 4y ago
Fixed overheads are never meaningless. O(n^2) algorithm that processes your data in 5s is faster on your data than O(log n) that takes 20 hours. Long term storage of billions of images is meaningless, if it takes billions of years to archive these images.
- Gigachad 4y ago
  It’s a one time cost rather than per image. You need the 4GB model only once and then you can uncompress unlimited images.
  - 112233 4y ago
    Yes, but each image needs access to this 4GB (actually, I have no idea how much RAM it takes up), plus whatever the working set size is. It is a non-trivial overhead that really limits throughput of your system, so you can process less images in parallel, so compressing billion of images in reasonable time suddenly may cost much more than the amount of storage it would save, compared to other methods.

janekm 4y ago

Is that true? I have never seen this done for any image compression comparisons that I have seen (i.e. only data that is specific to the image that is being compressed is included, not standard tables that are always used by the algorithm like the quantisation tables used in JPG compression)

fsiefken 4y ago
For text compression benchmarks it's done http://mattmahoney.net/dc/text.html http://mattmahoney.net/dc/text.html Matt doesn't do this on the Silesia corpus compression benchmark, even though it would make sense there as well: http://mattmahoney.net/dc/silesia.html http://mattmahoney.net/dc/silesia.html So a compressor of a few gigabyte would make sense if you would have a set of pictures of more then a few gigabyte. It's a bit similar to preprocessing text compression with a dictionary and adding the dictionary to the extractor to squeeze a bit more bytes.
- goombacloud 4y ago
  By the way, the leading nncp in the LTCB (text.html) "is a free, experimental file compressor by Fabrice Bellard, released May 8, 2019" :)
jerf 4y ago
Yes, it is done all the time. However, several people here are conflating "best compression as determined for a competition" and "best compression for use in the real world". There is an important relationship between them, absolutely, but in the real world we do not download custom decoders for every bit of compressed content. Just because there is a competition that quite correctly measures the entire size of the decompressor and encoded content does not mean that is now the only valid metric to measure decompression performance. The competitions use that metric for good and valid reasons, but those good and valid reasons are only vaguely correlated to the issues faced in the normal world. (Among the reasons why competitions must include the size of the decoder is that without that the answer is trivial; I define all your test inputs as a simple enumeration of them and my decoder hard-codes the output as the test values. This is trivially the optimal algorithm, making competition useless. If you could have a real-world encoder that worked this well, and had the storage to implement it, it would be optimal, but you can't possibly store all possible messages. For a humorous demonstration of this encoding method, see the classic joke: https://onemansblog.com/2010/05/18/prison-joke/ https://onemansblog.com/2010/05/18/prison-joke/ )