5 ms·

The fact that UTF-8 and UTF-16 are often exposed to programmers when dealing with text is a major failure of separation-of-concerns. If you had a stream of data

The fact that UTF-8 and UTF-16 are often exposed to programmers when dealing with text is a major failure of separation-of-concerns. If you had a stream of data that was gzipped, would it ever make sense to look at the bytes in the data stream before decompressing it? Variable-length text encodings are the same. Application code should only see Unicode code points. In general it was a mistake to put variable-length encodings into the Unicode standard. A much better design would have been to use UTF-32 for the application-level interface to characters, and use a separate compression standard that is optimized for fixed alphabets when transporting or storing text. This has the advantage that the compression scheme can be dynamically updated to match the letter frequencies in the real-world text, and it logically separates the ideas of encoding and compression so that the compression container is easier to swap out. And, of course, an entire class of bugs would be eliminated from application code. Edited first paragraph to clarify: Variable-length text encodings are the same.

est 13y ago
This looks like how Python3 handles Unicode. 1byte for ASCII, 2 bytes for BMP, 4 bytes for everything else.
cygx 13y ago
That makes it sound as if UTF-32 would be a silver bullet - it's not. Application code normally has to deal with user-perceived characters/glyphs/grapheme clusters, which means you'll have to treat UTF-32 as variable-length as well. If you want to get back the supposed benefits of UTF-32, you'll have to dynamically assign codepoints to grapheme clusters.
Someone 13y ago
I agree that that is the ideal end situation, but Unicode would have been dead on delivery if they had chosen that approach. Memory just was too expensive at the time to make a system that, in most of the computer-using world, wasted 75% in every text string. And no, just-in-time decompression wouldn't have worked either; CPU cycles also were too expensive at the time to do that. Unicode also would have been too incompatible with existing code that copied 8-bit character strings around. See http://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt http://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt for some rationale behind UTF-8.

est 13y ago

This looks like how Python3 handles Unicode. 1byte for ASCII, 2 bytes for BMP, 4 bytes for everything else.

cygx 13y ago

That makes it sound as if UTF-32 would be a silver bullet - it's not. Application code normally has to deal with user-perceived characters/glyphs/grapheme clusters, which means you'll have to treat UTF-32 as variable-length as well. If you want to get back the supposed benefits of UTF-32, you'll have to dynamically assign codepoints to grapheme clusters.

Someone 13y ago

I agree that that is the ideal end situation, but Unicode would have been dead on delivery if they had chosen that approach. Memory just was too expensive at the time to make a system that, in most of the computer-using world, wasted 75% in every text string. And no, just-in-time decompression wouldn't have worked either; CPU cycles also were too expensive at the time to do that. Unicode also would have been too incompatible with existing code that copied 8-bit character strings around. See http://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt http://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt for some rationale behind UTF-8.