4 ms·

Is that true? I have never seen this done for any image compression comparisons that I have seen (i.e. only data that is specific to the image that is being com

by janekm 4y ago

Is that true? I have never seen this done for any image compression comparisons that I have seen (i.e. only data that is specific to the image that is being compressed is included, not standard tables that are always used by the algorithm like the quantisation tables used in JPG compression)

fsiefken 4y ago
For text compression benchmarks it's done http://mattmahoney.net/dc/text.html http://mattmahoney.net/dc/text.html Matt doesn't do this on the Silesia corpus compression benchmark, even though it would make sense there as well: http://mattmahoney.net/dc/silesia.html http://mattmahoney.net/dc/silesia.html So a compressor of a few gigabyte would make sense if you would have a set of pictures of more then a few gigabyte. It's a bit similar to preprocessing text compression with a dictionary and adding the dictionary to the extractor to squeeze a bit more bytes.
- goombacloud 4y ago
  By the way, the leading nncp in the LTCB (text.html) "is a free, experimental file compressor by Fabrice Bellard, released May 8, 2019" :)
jerf 4y ago
Yes, it is done all the time. However, several people here are conflating "best compression as determined for a competition" and "best compression for use in the real world". There is an important relationship between them, absolutely, but in the real world we do not download custom decoders for every bit of compressed content. Just because there is a competition that quite correctly measures the entire size of the decompressor and encoded content does not mean that is now the only valid metric to measure decompression performance. The competitions use that metric for good and valid reasons, but those good and valid reasons are only vaguely correlated to the issues faced in the normal world. (Among the reasons why competitions must include the size of the decoder is that without that the answer is trivial; I define all your test inputs as a simple enumeration of them and my decoder hard-codes the output as the test values. This is trivially the optimal algorithm, making competition useless. If you could have a real-world encoder that worked this well, and had the storage to implement it, it would be optimal, but you can't possibly store all possible messages. For a humorous demonstration of this encoding method, see the classic joke: https://onemansblog.com/2010/05/18/prison-joke/ https://onemansblog.com/2010/05/18/prison-joke/ )

by janekm 4y ago

fsiefken 4y ago

For text compression benchmarks it's done http://mattmahoney.net/dc/text.html http://mattmahoney.net/dc/text.html Matt doesn't do this on the Silesia corpus compression benchmark, even though it would make sense there as well: http://mattmahoney.net/dc/silesia.html http://mattmahoney.net/dc/silesia.html So a compressor of a few gigabyte would make sense if you would have a set of pictures of more then a few gigabyte. It's a bit similar to preprocessing text compression with a dictionary and adding the dictionary to the extractor to squeeze a bit more bytes.

goombacloud 4y ago
By the way, the leading nncp in the LTCB (text.html) "is a free, experimental file compressor by Fabrice Bellard, released May 8, 2019" :)

jerf 4y ago

Yes, it is done all the time. However, several people here are conflating "best compression as determined for a competition" and "best compression for use in the real world". There is an important relationship between them, absolutely, but in the real world we do not download custom decoders for every bit of compressed content. Just because there is a competition that quite correctly measures the entire size of the decompressor and encoded content does not mean that is now the only valid metric to measure decompression performance. The competitions use that metric for good and valid reasons, but those good and valid reasons are only vaguely correlated to the issues faced in the normal world. (Among the reasons why competitions must include the size of the decoder is that without that the answer is trivial; I define all your test inputs as a simple enumeration of them and my decoder hard-codes the output as the test values. This is trivially the optimal algorithm, making competition useless. If you could have a real-world encoder that worked this well, and had the storage to implement it, it would be optimal, but you can't possibly store all possible messages. For a humorous demonstration of this encoding method, see the classic joke: https://onemansblog.com/2010/05/18/prison-joke/ https://onemansblog.com/2010/05/18/prison-joke/ )