3 ms·

> Fortunately, we can fix the fact that tar.gz files are unindexed and unseekable, while still making the file a valid tar.gz file by taking advantage of the fa

by koolba 8y ago

> Fortunately, we can fix the fact that tar.gz files are unindexed and unseekable, while still making the file a valid tar.gz file by taking advantage of the fact that two gzip streams can be concatenated and still be a valid gzip stream. So you can just make a tar file where each tar entry is its own gzip stream. I'm surprised nobody came up with this idea till now. It's brilliantly simple.

xyzzy_plugh 8y ago
Well, for one, it's not obviously useful in traditional applications: > This makes images a few percent larger (due to more gzip headers and loss of compression context between files), but it's plenty acceptable. Compressing an entire image is generally great. Compressing all of the individual files in an image, is generally not great.
- bradfitz 8y ago
  Maybe not great, but like I said: acceptable. About 7.6% bigger: https://github.com/golang/build/commit/8a5a4d227f08eb1d889fa0c0f67dc71020c222f1 https://github.com/golang/build/commit/8a5a4d227f08eb1d889fa...
  - koolba 8y ago
    That's what I meant by "brilliantly simple"; namely having something like this work with existing systems while imposing a marginal overhead.
CSDude 8y ago
Compressing individual files would probably result in a bigger file, ofc depending on the content mostly.
_wmd 8y ago
> This makes images a few percent larger Teehee. The method is not new at all, for example compressors like xzip do this out of the box, and almost the exact thing they're doing is basically how ZIP files work The trouble with discarding state on every file is that it really hurts performance with small files, or when using anything like a modern codec, which gzip/deflate is not. Gzip maintains a 32kb dictionary which is quite easy to exceed with contemporary data, but with a modern compressor (like lzma2) losing that window will absolutely devastate ratios The usual solution is so-called 'solid' compression, where the uncompressed input is partitioned into blocks spanning file boundaries. It configurably trades seek efficiency for reliably preserving compressor context -- including allowing seeking within files. Their format could be modified to support this while retaining backwards compatibility as good as the current method. What they have is already pretty much solid compression, except it only chunks large files. This is basically a weird special case of a simpler and more general design everyone else uses. Finally on the compatibility angle, the end of stream is visible at an API level, so this isn't going to be 100% perfect. I'd expect one or more obscure implementations (maybe Windows apps? Java?) to potentially break
david_ar 8y ago
dict uses something similar: https://linux.die.net/man/1/dictzip https://linux.die.net/man/1/dictzip

by koolba 8y ago

xyzzy_plugh 8y ago

Well, for one, it's not obviously useful in traditional applications: > This makes images a few percent larger (due to more gzip headers and loss of compression context between files), but it's plenty acceptable. Compressing an entire image is generally great. Compressing all of the individual files in an image, is generally not great.

bradfitz 8y ago
Maybe not great, but like I said: acceptable. About 7.6% bigger: https://github.com/golang/build/commit/8a5a4d227f08eb1d889fa0c0f67dc71020c222f1 https://github.com/golang/build/commit/8a5a4d227f08eb1d889fa...
- koolba 8y ago
  That's what I meant by "brilliantly simple"; namely having something like this work with existing systems while imposing a marginal overhead.

CSDude 8y ago

Compressing individual files would probably result in a bigger file, ofc depending on the content mostly.

_wmd 8y ago

> This makes images a few percent larger Teehee. The method is not new at all, for example compressors like xzip do this out of the box, and almost the exact thing they're doing is basically how ZIP files work The trouble with discarding state on every file is that it really hurts performance with small files, or when using anything like a modern codec, which gzip/deflate is not. Gzip maintains a 32kb dictionary which is quite easy to exceed with contemporary data, but with a modern compressor (like lzma2) losing that window will absolutely devastate ratios The usual solution is so-called 'solid' compression, where the uncompressed input is partitioned into blocks spanning file boundaries. It configurably trades seek efficiency for reliably preserving compressor context -- including allowing seeking within files. Their format could be modified to support this while retaining backwards compatibility as good as the current method. What they have is already pretty much solid compression, except it only chunks large files. This is basically a weird special case of a simpler and more general design everyone else uses. Finally on the compatibility angle, the end of stream is visible at an API level, so this isn't going to be 100% perfect. I'd expect one or more obscure implementations (maybe Windows apps? Java?) to potentially break

david_ar 8y ago

dict uses something similar: https://linux.die.net/man/1/dictzip https://linux.die.net/man/1/dictzip