3 ms·
From what I understand, those "autogenerated 'synthetic' codepoints" aren't guaranteed to be identical across runs (and typically won't be). What they do is a
by Someone 9y ago
From what I understand, those "autogenerated 'synthetic' codepoints" aren't guaranteed to be identical across runs (and typically won't be).
What they do is a bit like LZW compression: use a set of codes that is larger than the number of simple graphemes, assign each simple grapheme its 'normal' code and, whenever one encounters a complex one, assign it the first hitherto unused 'code point'.
That works fine until it breaks. They hope it never breaks in practice. I think there's a good chance it won't, but still don't think this is a good idea.
- cygx 9y agothose "autogenerated 'synthetic' codepoints" aren't guaranteed to be identical across runs Indeed: It's not an exchange format, but the format of the internal in-memory string representation.
- kibwen 9y agoIf it's only intended as an in-memory representation, then how about a 24-bit representation, which is just as likely to work "in practice, if not in theory" as the 32-bit representation, while saving 25% of the memory? That gives you 16 million unique grapheme clusters per program instance, which, heck, is probably still more than 99% of programs use; you could easily get away with a 16-bit representation and use no more memory for strings than Java, C#, or Javascript. It's not like using a unique superset of UTF-32 is saving anyone the hassle of needing to convert between formats, since the vast majority of the world is ASCII-ish, UTF-8, and UTF-16 anyway.
- cygx 9y ago> how about a 24-bit representation Because 24-bit integers are not a thing on x86?
- kibwen 9y agoWho says you need to use integers? You're completely ignoring my question. Again, if you know of a design doc buried on a mailing list somewhere please just link me to that so that I can answer all of my questions at once.