3 ms·
If you only ship the default collation it is already well under 1MB. DUCET [1] for example is 406 KB in a naive format [2], which gzips (-9) into 164 KB and Bro
by lifthrasiir 5y ago
If you only ship the default collation it is already well under 1MB. DUCET [1] for example is 406 KB in a naive format [2], which gzips (-9) into 164 KB and Brotli (-q 11) brings it down to 76 KB. Other Unicode tables combined, even in a directly-accessible form, do not exceed 200 KB with an appropriate encoding in my experience.
[1] https://www.unicode.org/Public/UCA/13.0.0/allkeys.txt https://www.unicode.org/Public/UCA/13.0.0/allkeys.txt
[2] Encodes `XXXX; [.AAAA.BBBB.CCCC] [*DDDD.EEEE.FFFF] ...` into three-byte code point and seven bytes per each collation element (`.` encoded as 0x01, `*` encoded as 0x02, the end of elements signalled with 0x00); everything is in big endian and sorted by code point. This is not suitable for direct access as is but close. (The only reason I've used a naive encoding here is that, I have never tried to compress DUCET in a directly-accessible form so I couldn't give an educated guess at all.)
- slimsag 5y agoAuthor here, I hinted at the end of the article that I am exploring exactly that: a compression algorithm for the DUCET specifically. My early findings suggest that I can bring it to under 50K and beat gzip/brotli compressions, but with those on top can go further https://github.com/jecolon/ziglyph/issues/3 https://github.com/jecolon/ziglyph/issues/3
- lifthrasiir 5y ago28 KB with Brotli! I'd like to see the number with LZ4 (or similarly simple compression algorithms) so that the dependency can be minimized, but that sounds way better than I imagined.