3 ms·
UTF8 != unicode. Unicode is a standard, that maps the giant number of letters to unique numbers. UTF8 (and UTF16, UTF32) are various encodings of that. Unicod
by runn1ng 13y ago
UTF8 != unicode.
Unicode is a standard, that maps the giant number of letters to unique numbers.
UTF8 (and UTF16, UTF32) are various encodings of that. Unicode itself is not an encoding; unicode just says "letter 'č' is 269"; UTF-8 says "We will represent it as two bytes, first is C4, second is 8D".
UTF8 is indeed variable width; UTF16 and UTF32 are, AFAIK, fixed width. Both use unicode standard.
- pjscott 13y agoWhat the comment you're replying to is saying that glyphs may be represented by more than one code point. So sure, UTF32 has a fixed width for each code point, but it's not quite so clear how useful that is.
- runn1ng 13y agoOh, in that case I misunderstood parent. Sorry about that.
- Stratoscope 13y agoEven if we're talking about code points and not characters, UTF-16 is variable width. UTF-8 is variable width using 8, 16, 24, or 32 bits per code point. UCS-2 is fixed width using 16 bits per code point. UTF-16 is variable width using 16 or 32 bits per code point. It's an extension of the fixed width UCS-2 format: every valid 16-bit UCS-2 code point is also a valid UTF-16 code point, but UTF-16 also includes 32-bit code points. UTF-32 is fixed width using 32 bits per code point.
- alister 13y agoUTF-8 is variable width using 8, 16, 24, or 32 bits per code point. I wondered why you didn't write that "UTF-8 is variable width using 8, 16, 24, 32, 40, or 48 bits per code point", because that's how Prosser/Thompson/Pike's UTF-8 idea was proposed, as shown in the "most elegant hack" article. And here's the answer from Wikipedia: In November 2003 UTF-8 was restricted by RFC 3629 to end at U+10FFFF, in order to match the constraints of the UTF-16 character encoding. This removed all 5- and 6-byte sequences, and about half of the 4-byte sequences. Even though they eliminated the 5th and 6th bytes, UTF-8 can still represent all of Unicode's 1,114,112 code points. I'm mentioning this because others probably are wondering the same thing.
- option_greek 13y agoHow sad. One defective format limits another great format.
- ygra 13y agoWe don't see the code point space run out anytime soon this time, even with questionable additions like Emoji. It took us 20 years to get to the current state and character encoding rate has slowed down significantly since the early days (mostly because most things are covered already).
- gsnedders 13y agoThe Unicode standard nowadays defines UTF-8, UTF-16, and UTF-32. It speaks of "characters" which are assigned (i.e., mapped) to "code points" (i.e., U+0000 to U+10FFFF) and character encodings which encode scalar values (all code points except the surrogate ranges, therefore including all characters).
- btilly 13y agoUTF8 is indeed variable width; UTF16 and UTF32 are, AFAIK, fixed width. Both use unicode standard. Your knowledge is wrong. As of 2012, there are 110,182 assigned Unicode characters, and 137,468 more which have been reserved for private use. With 16 bits there are only 65,536 possible values, and so UTF-16 has to be variable width. (Though it was originally intended to be fixed width. A misconception unfortunately embedded in the design of languages from the time such as Java.)