4 ms·

>change from .tar.xz to .zip format. this doubles the size, but speeds up file access from 30s to 0.01s Perhaps not the best trade, than again, the choice migh

by yyyk 4y ago

>change from .tar.xz to .zip format. this doubles the size, but speeds up file access from 30s to 0.01s Perhaps not the best trade, than again, the choice might not be necessary at all: xz is rather slow on decompression. What were to happen if the format was .zstd instead of .zip (with/without .tar)? We should be able to achieve a lower size than .zip without being so slow.

VHRanger 4y ago
Yes, there's a lot better compression available these days, but not everyone has the software installed to unpack them
- yyyk 4y ago
  We practically need libgen-crystal to use this anyway, so just link zstd to libgen-crystal and now it's available. * Also, this consideration did not prevent trying .tar.xz. I don't imagine .tar.xz is that much more available than .zstd.
- unicornporn 4y ago
  > not everyone has the software installed to unpack them People downloading 2 TB of compressed books in plain text format are not exactly the definition of "everyone". > Yes, there's a lot better compression available these days Tell me more!
  - VHRanger 4y ago
    zstd: https://github.com/facebook/zstd https://github.com/facebook/zstd Or any of the ones they benchmark against in their .README (lz4, snappy, zlib, etc.)
duskwuff 4y ago
ZIP is seekable -- you can get a list of files and/or read individual files out of the archive without decompressing the whole thing. TAR+(stream compressor) isn't; the only way to get a list of files or to decompress individual files is to read through the whole thing. The 7zip file format is a nice middle ground in this regard. The file has an index, just like ZIP, and files are compressed either individually or in small groups ("solid compression"); either approach makes it practical to extract individual files.
- yyyk 4y ago
  7z's PPMd should do better than zip on compressing text files/speed (7z's default LZMA method is rather slow).
torginus 4y ago
I did something similar 3 years ago. My goals were minimal file size, as well as fast read times. I just dug up the code, and what I did was I compressed every text document with Zstd (with quite a high factor), and wrote them in a file sequentially. At the end of a file, I wrote a manifest, which was a MessagePack serialized data structure of the book descriptors and pointers to the file entries, also compressed with Zstd. The start of the files had a header with pointers to the manifest. The book stuff was wicked fast to seek, and quite a bit smaller than just a zip, but it had no error correction, and was proprietary. Edit: If there's some interest, I can try to convert the dataset to this format.
- sitkack 4y ago
  See my sibling comment, I did something similar but I used sqlite to store the manifest.
sitkack 4y ago
I built an offline wikipedia reader that heavily relied on compression while giving nearly instantaneous access to an arbitrary article (< 100ms). It uses sqlite FTS for searching article titles. I did not enable FTS over the body of the articles. Outline of the technique is here https://news.ycombinator.com/item?id=28013347 https://news.ycombinator.com/item?id=28013347

by yyyk 4y ago

VHRanger 4y ago

Yes, there's a lot better compression available these days, but not everyone has the software installed to unpack them

yyyk 4y ago
We practically need libgen-crystal to use this anyway, so just link zstd to libgen-crystal and now it's available. * Also, this consideration did not prevent trying .tar.xz. I don't imagine .tar.xz is that much more available than .zstd.
unicornporn 4y ago
> not everyone has the software installed to unpack them People downloading 2 TB of compressed books in plain text format are not exactly the definition of "everyone". > Yes, there's a lot better compression available these days Tell me more!
- VHRanger 4y ago
  zstd: https://github.com/facebook/zstd https://github.com/facebook/zstd Or any of the ones they benchmark against in their .README (lz4, snappy, zlib, etc.)

duskwuff 4y ago

ZIP is seekable -- you can get a list of files and/or read individual files out of the archive without decompressing the whole thing. TAR+(stream compressor) isn't; the only way to get a list of files or to decompress individual files is to read through the whole thing. The 7zip file format is a nice middle ground in this regard. The file has an index, just like ZIP, and files are compressed either individually or in small groups ("solid compression"); either approach makes it practical to extract individual files.

yyyk 4y ago
7z's PPMd should do better than zip on compressing text files/speed (7z's default LZMA method is rather slow).

torginus 4y ago

I did something similar 3 years ago. My goals were minimal file size, as well as fast read times. I just dug up the code, and what I did was I compressed every text document with Zstd (with quite a high factor), and wrote them in a file sequentially. At the end of a file, I wrote a manifest, which was a MessagePack serialized data structure of the book descriptors and pointers to the file entries, also compressed with Zstd. The start of the files had a header with pointers to the manifest. The book stuff was wicked fast to seek, and quite a bit smaller than just a zip, but it had no error correction, and was proprietary. Edit: If there's some interest, I can try to convert the dataset to this format.

sitkack 4y ago
See my sibling comment, I did something similar but I used sqlite to store the manifest.

sitkack 4y ago

I built an offline wikipedia reader that heavily relied on compression while giving nearly instantaneous access to an arbitrary article (< 100ms). It uses sqlite FTS for searching article titles. I did not enable FTS over the body of the articles. Outline of the technique is here https://news.ycombinator.com/item?id=28013347 https://news.ycombinator.com/item?id=28013347