3 ms·
Seems to me that gzip or lzma with a constant, language-optimized dictionary would both... (1) compress better; and (2) be more robust/self-documenting. EG:
by gojomo 18y ago
Seems to me that gzip or lzma with a constant, language-optimized dictionary would both...
(1) compress better; and
(2) be more robust/self-documenting.
EG: "Feed this dictionary, optimized over 1 billion [English|Western language|etc] documents, to zlib deflateSetDictionary() before compressing."
(The zlib/gzip dictionary is only 32KiB, but the lzma dictionary can be up to 32MiB.)
Building a small library of standard dictionaries, keyed by hash (so they can be locally cached or fetched when needed) would offer good coverage of many string domains.
See for example the Google 'Shared Dictionary Compression over HTTP (SDCH)' proposal, apparently a feature of Google Toolbar:
http://sdch.googlegroups.com/web/Shared_Dictionary_Compression_over_HTTP.pdf http://sdch.googlegroups.com/web/Shared_Dictionary_Compressi...
http://groups.google.com/group/SDCH http://groups.google.com/group/SDCH
- antirez 18y agoSure you can do this, what you get is something that is also able to adapt dynamically after the first 100 bytes or so. My requirement was to have something smaller, simpler, possibly able to run on low memory systems, and especially, with a much faster decompression compared to zlib. I'm not exploiting an arithmetic encoding basically, and the dictionary is (intentionally) small. Add to this lib an arithmetic encoding and you get near to a zlib with an optimized dictionary, but at the cost of a lot of speed.