3 ms·
> I had a hack back in the previous millenium that improved on .tar.gz, compatibly. FWIW, this is known as the 'sort key trick'. It's been in the folklore, as
by gwern 11y ago
> I had a hack back in the previous millenium that improved on .tar.gz, compatibly.
FWIW, this is known as the 'sort key trick'. It's been in the folklore, as you say, for quite a while. I give some examples in http://www.gwern.net/Archiving%20URLs#sort---key-compression-trick http://www.gwern.net/Archiving%20URLs#sort---key-compression... for WWW archives.
> Reordering a tar file doesn't change the meaning, and IMO the big, big gap in compression research is to identify such expression transformations.
This hasn't been totally ignored. The problem is that you need to know and be able to specify in advance a useful ordering of input files to make up a better stream or a tool (like 'lrzip', designed to look for long-range correlations) must infer it which seems hard to do without requiring multiple passes; and what's the incentive? In a world in which RAR and ZIP and Gzip are still considered the standards and people aren't even using xz, no one wants to use such a niche tool for anything serious.
- Arnt 11y ago1. Well, there are people who serve web fonts and CSS sprite assets to about 10¹²³ people per hour and would go to some lengths to save a few bytes on those, even if it involves niche tools that many others don't bother with. 2. "What's the incentive" is IMNSHO a terrible question for a research area.
- gwern 11y ago> 1. Well, there are people who serve web fonts and CSS sprite assets to about 10¹²³ people per hour Those people have the problem that the delivery mechanisms are extremely limited and hardwired and a lot of the possible advances cannot be shoehorned into a DEFLATE-compatible bytestream or whatever it is IE6 supports... > 2. "What's the incentive" is IMNSHO a terrible question for a research area. Researchers are human too. It's demotivating to try to research fancier data compression techniques if you're certain that no one will ever use it.