3 ms·

> While WASM languages could shell out to JavaScript browser APIs for collation, I suspect they won’t due to the lack of guarantees around those APIs. That's j

> While WASM languages could shell out to JavaScript browser APIs for collation, I suspect they won’t due to the lack of guarantees around those APIs. That's just it, right? Unicode support has always been a tradeoff between precise coverage/larger binaries and loose coverage/smaller binaries. I'd be excited to see the data size for collation+normalization get under 1MB.

lifthrasiir 5y ago
If you only ship the default collation it is already well under 1MB. DUCET [1] for example is 406 KB in a naive format [2], which gzips (-9) into 164 KB and Brotli (-q 11) brings it down to 76 KB. Other Unicode tables combined, even in a directly-accessible form, do not exceed 200 KB with an appropriate encoding in my experience. [1] https://www.unicode.org/Public/UCA/13.0.0/allkeys.txt https://www.unicode.org/Public/UCA/13.0.0/allkeys.txt [2] Encodes `XXXX; [.AAAA.BBBB.CCCC] [*DDDD.EEEE.FFFF] ...` into three-byte code point and seven bytes per each collation element (`.` encoded as 0x01, `*` encoded as 0x02, the end of elements signalled with 0x00); everything is in big endian and sorted by code point. This is not suitable for direct access as is but close. (The only reason I've used a naive encoding here is that, I have never tried to compress DUCET in a directly-accessible form so I couldn't give an educated guess at all.)
- slimsag 5y ago
  Author here, I hinted at the end of the article that I am exploring exactly that: a compression algorithm for the DUCET specifically. My early findings suggest that I can bring it to under 50K and beat gzip/brotli compressions, but with those on top can go further https://github.com/jecolon/ziglyph/issues/3 https://github.com/jecolon/ziglyph/issues/3
  - lifthrasiir 5y ago
    28 KB with Brotli! I'd like to see the number with LZ4 (or similarly simple compression algorithms) so that the dependency can be minimized, but that sounds way better than I imagined.
josephg 5y ago
Yeah its true but disappointing. I ported some email parsing code from C to wasm a couple of years ago and ran into this problem. Emails are encoded using all sorts of different character encodings. Canonicalizing them to utf-8 is important for consumers, but the libicu database for character encoding is several megabytes. So I'm left with a sad set of choices. I could: 1) Not support character encodings outside of ASCII, utf-8 and utf-16 2) Ship a huge wasm bundle or 3) Call back into javascript from my wasm module and use the javascript wrappers around libicu. This would make it harder to use the library outside of javascript. I'm not sure what the right solution is here. If the character conversion databases can be shrunk to <500kb then I could just ship them with the module. Or if wasm had a better standard library, I could use that. But right now the whole situation is unfortunately awkward.

lifthrasiir 5y ago

If you only ship the default collation it is already well under 1MB. DUCET [1] for example is 406 KB in a naive format [2], which gzips (-9) into 164 KB and Brotli (-q 11) brings it down to 76 KB. Other Unicode tables combined, even in a directly-accessible form, do not exceed 200 KB with an appropriate encoding in my experience. [1] https://www.unicode.org/Public/UCA/13.0.0/allkeys.txt https://www.unicode.org/Public/UCA/13.0.0/allkeys.txt [2] Encodes `XXXX; [.AAAA.BBBB.CCCC] [*DDDD.EEEE.FFFF] ...` into three-byte code point and seven bytes per each collation element (`.` encoded as 0x01, `*` encoded as 0x02, the end of elements signalled with 0x00); everything is in big endian and sorted by code point. This is not suitable for direct access as is but close. (The only reason I've used a naive encoding here is that, I have never tried to compress DUCET in a directly-accessible form so I couldn't give an educated guess at all.)

slimsag 5y ago
Author here, I hinted at the end of the article that I am exploring exactly that: a compression algorithm for the DUCET specifically. My early findings suggest that I can bring it to under 50K and beat gzip/brotli compressions, but with those on top can go further https://github.com/jecolon/ziglyph/issues/3 https://github.com/jecolon/ziglyph/issues/3
- lifthrasiir 5y ago
  28 KB with Brotli! I'd like to see the number with LZ4 (or similarly simple compression algorithms) so that the dependency can be minimized, but that sounds way better than I imagined.

josephg 5y ago

Yeah its true but disappointing. I ported some email parsing code from C to wasm a couple of years ago and ran into this problem. Emails are encoded using all sorts of different character encodings. Canonicalizing them to utf-8 is important for consumers, but the libicu database for character encoding is several megabytes. So I'm left with a sad set of choices. I could: 1) Not support character encodings outside of ASCII, utf-8 and utf-16 2) Ship a huge wasm bundle or 3) Call back into javascript from my wasm module and use the javascript wrappers around libicu. This would make it harder to use the library outside of javascript. I'm not sure what the right solution is here. If the character conversion databases can be shrunk to <500kb then I could just ship them with the module. Or if wasm had a better standard library, I could use that. But right now the whole situation is unfortunately awkward.