5 ms·
I think that the BMP -- Basic Multilingual Plane or the first 16-bit of Unicode characters -- is pretty reasonable, and covers fairly well everything we may con
by pron 10y ago
I think that the BMP -- Basic Multilingual Plane or the first 16-bit of Unicode characters -- is pretty reasonable, and covers fairly well everything we may consider as text (all alphabets in current use plus mathematical symbols). Anything beyond that, from emojis and pictograms to ancient Greek musical notation is pretty... weird.
I think it would have made much more sense to have something like image tags: a special codepoint would introduce a link to a URL containing a sequence of glyphs, followed by an index into that sequence. Those glyphs would be guaranteed not to change (in any meaningful way), and devices would be free to cache them. This way, anything that isn't real text, would standardize representation, too, instead of just a vague "meaning". Another standard could relate those glyphs to one another in some way, giving them standard semantics and means of translation (i.e. "Egyptian hieroglyphics"). This would also allow each of those (emojis or hieroglyphics) to evolve their standards independent of a single universal standard that means little.
- yk 10y agoI mostly agree. In my opinion, separating code points from encodings (without providing a replacement for txt files) was the original sin of unicode. That just adds a lot of complexity with very little gain. (Suppose we would have standardized from ASCII to a 64 bit per character standard encoding, then text files would grow by a factor of 8 and nothing important would have happened.^1) ^1 I am pretty sure that some use cases actually profit a lot, but neither text file formats (since most text does not contain 2^64 different characters zip would work nicely) nor networking (since most data on the internet is either video or torrents) seem to be among them. So probably they would not be huge fields.
- skybrian 10y agoText files growing by 8x would be a non-starter. It would be nice to standardize on something but what do you have against UTF8?
- pron 10y agoIt's not bad, but it's complicated, as it requires an O(n) algorithm to jump to a specific character. Unicode should have been capped at 16 bits, and doubling text files in size is fine. An alternate representation of simplified UTF-8 would have kept compatibility with old ASCII files.
- hvidgaard 10y agoDoubling text files is a waste for the most part, but what makes it tolerable is compression. Still 16 bits would not be enough, it's only 65536 different code pages, less than half of what is currently in Unicode. 24 bit is sufficiently out of allignment with modern hardware and algorithms, so 32 bit it is for efficiency. That is now 4 times the size, and compression is now a requirement. In any case, Utf8 has a place, and if you want easy manipulation and search, convert it to Utf32 - it's fixed width.
- pron 10y ago> Still 16 bits would not be enough, it's only 65536 different code pages, less than half of what is currently in Unicode. But that's only because Unicode has significantly ventured well beyond what we consider to be text. The BMP is enough to represent all text (including math).
- hvidgaard 10y agoBMP is mostly filled with symbols from logographic languages, and currently has around 100 free code points. 16 bits simply isn't enough for the scope of capturing all written language of the history.
- pron 10y agoBut I don't think all written languages in history should have the same treatment when it comes to standardized data representation, or should all be standardized by the same body. It's OK to have alphabets no one has used for thousands of years other than specialized researchers standardized separately from Latin or Chinese alphabets.
- cbd1984 10y agoThat's nice, except the BMP doesn't encode all of Chinese, and it includes a number of weird control characters for compatibility with ASCII. Like Vertical Tab. Who uses Vertical Tab anymore? The dream of a 16-bit Unicode washed up on the rocks of CJK scripts. It's dead and it isn't going to be revived. You can argue for a simpler standard, with fewer assigned codepoints, but the original BMP isn't it and was never going to be it.
- pron 10y ago> but the original BMP isn't it and was never going to be it. I concede this point. I still don't see why the Unicode Consortium should spend effort standardizing non-text as text.