4 ms·
Something I've wondered about is why, given Unicode is limited to 21 bits now, nobody seems to be using a 24-bit format ("UTF-24")? If you want constant-time in
by TazeTSchnitzel 11y ago
Something I've wondered about is why, given Unicode is limited to 21 bits now, nobody seems to be using a 24-bit format ("UTF-24")? If you want constant-time indexing without bit-packing tricks, that'd be the ideal.
Though, of course, constant-time indexing isn't what it's hyped up to be. That's only for codepoints. Actual characters you see on screen are often combinations of codepoints.
So I've answered my own question really. Why does nobody use it? Because there's no point in it.
http://stackoverflow.com/questions/10143836/why-is-there-no-utf-24 http://stackoverflow.com/questions/10143836/why-is-there-no-...
- protomyth 11y ago"That's only for codepoints. Actual characters you see on screen are often combinations of codepoints." So, there is no encoding that actually has 1 word (of whatever byte length) to a character on the screen?
- colanderman 11y agoYou might be interested in the Supercombiner: http://sbp.so/supercombiner http://sbp.so/supercombiner
- Aardwolf 11y agoWhat language uses that many diacritics tho? Unicode is supposed to support all written languages - supporting a u with 100 diacritics seems beyond that scope :)
- colanderman 11y agoMathematics. Not 100, but symbols are often decorated with 3 diacritics or so. That's a lot to cram into one fixed-size word.
- gsnedders 11y agoThere's an unbounded number of possible graphemes per Unicode, because they are formed of a start character and an unbounded number of combining characters following them. As such, no fixed-length encoding can actually enumerate all possible graphemes.
- protomyth 11y agoWhat is the practicality of an unbounded number of possible graphemes? Does anyone in real writing of languages and not some concocted test case need this? I understood there is some conflict in the encoding of kanji into unicode, does this relate?
- masklinn 11y ago> What is the practicality of an unbounded number of possible graphemes? That question doesn't really make sense. Unbounded cluster sizes is simply a result of unicode design because there's no reason to bound it. > Does anyone in real writing of languages and not some concocted test case need this? In the real writing of languages there are languages with grapheme clusters of size 4+ e.g. क्षि (Devanagari kshi) is a single cluster made up of क, ्, ष and ि. > I understood there is some conflict in the encoding of kanji into unicode, does this relate? It's unrelated. Han Unification compressed the number of clusters by unifying han characters across simplified Chinese, traditional Chinese, Japanese and Korean. Han characters were an issue in the original 16 bits unicode because supporting all of them was estimated as possibly topping out 100,000 codepoints (where 16 bits unicode only supported 65k codepoints). AFAIK Han characters don't use much if any composition (outside of romanisation). Brahmic scripts, Hebrew and Arabic scripts, on the other hand, pile on diacritics.
- protomyth 11y ago> That question doesn't really make sense. Unbounded cluster sizes is simply a result of unicode design because there's no reason to bound it. If I'm trying to draw a character on the screen, it makes a lot of sense. I guess I'm struck about how hard it would be to select the 4th through 8th character displayed on the screen from rows 3 through 13.
- vbezhenar 11y agoThe main problem is that a lot of characters in real world are from ASCII part. So for 24-bit 66% of memory space will be wasted. And it's bad not only because it uses RAM, HDD, bandwidth, but generally because it uses precious processor cache storage.
- derefr 11y agoGiven an efficient string processing methodology mostly consisting of 1. holding onto opaque string binaries until you need to compose them all together at the end (i.e. the way Erlang's iolists work) and 2. applying encapsulated operations over a given string (e.g. "split", "scan with regular expression") rather than trying to index into it; I've never understood why we can't just use a really inefficient-but-practical string encoding, e.g. UTF-32, with a very low-cost stream compression algorithm on top.
- vardump 11y ago> I've never understood why we can't just use a really inefficient-but-practical string encoding, e.g. UTF-32, with a very low-cost stream compression algorithm on top. That's right. We could figure out how to represent common characters with less bits. For low-cost character stream compression, byte granularity should work nicely with computers, so we should it. A byte could have a bit that indicates the range is extended, like highest bit, because it's easy to process. Same idea could be chained to represent full range of unicode characters. That way we'd have compressed UTF-32 characters!
- derefr 11y agoI meant more something like RLE or simple Huffman encoding. A "string" type like Redis's dict type or ObjC's array type, where the internal representation changes as the allocated object grows in size and therefore is worth making different trade-offs on. (So, a five-byte string, maybe UTF-8 is a fine encoding. A 3KB string? Worth prepending a compression dictionary to.)
- tormeh 11y agoWhy not have an array of pointers to an array each, where each pointed-to array holds a single character?
- vardump 11y ago24-bit and 21-bit format wouldn't have much difference in terms of performance of constant time indexing. Strlen (for storage space) would be dramatically slower than in any other format.
- TazeTSchnitzel 11y agoWould it really? You can still use SIMD and vectorise.
- vardump 11y agoI don't immediately know how to use SIMD to scan for 24-bit wide zero terminator or for any character for that matter. 128/24 is 5 1/3 or 5.333... Alignment would mean a lot of slow operations to stitch data between memory loads. How would you efficiently scan for a specific 24-bit character by using SIMD? Say you're not running on Haswell, but something older that's not as good with unaligned memory accesses?