3 ms·
Note that it's possible to make wikipedia substantially smaller if you're happy to use more aggressive compression algorithms. Kiwix divides the data into chun
by londons_explore 4y ago
Note that it's possible to make wikipedia substantially smaller if you're happy to use more aggressive compression algorithms.
Kiwix divides the data into chunks and adds various indexes and stuff to allow searching data and fast access, even on slow CPU devices. But if you can live with slow loading, you can probably halve the storage space required, or maybe more.
- kragen 4y agoWhat compression algorithms would help? It's already using lzma for the text (in the form of .xz).
- londons_explore 4y agoThe hutter prize is a competition for compressing Wikipedia: http://prize.hutter1.net/ http://prize.hutter1.net/ So the best algorithm to use from there is starlit, with a compression factor of 8.67, compared to lzma in 2MB chunks which can only achieve about 4:1 compression.
- londons_explore 4y agoOh, and if you are happy to wait days or weeks for your compressed data, Fabrice bellards nncp manages even higher ratios (but isn't eligible for the prize because it's too slow)
- planede 4y agoSubmissions for the Hutter price also include the size of the compressor in the "total size". So I assume that's hard to beat if you use huge neural networks on the compression size, even if decompression is fast enough.
- londons_explore 4y agonncp uses neural networks, but 'trains itself' as it goes, so there is no big binary blob involved in the compressor. The only reason it isn't eligible are compute constraints (and I don't think the hutter prize allows a GPU, which nncp needs for any reasonable performance).
- planede 4y agoAh, OK, fair enough.
- int_19h 4y agoThey embed full-text indices into the .zim file these days, but they used to be separate originally. IIRC at that time the index for English wiki took up around 12 Gb, with the actual data in the ballpark of 65 Gb