4 ms·
You can't assume that any code point is a character. This is because of combining marks. If you have a string with an ASCII 'A' codepoint followed by U+0328 co
by pornel 4y ago
You can't assume that any code point is a character. This is because of combining marks.
If you have a string with an ASCII 'A' codepoint followed by U+0328 codepoint, then it's a two-codepoint 'Ą' character. The first code point wasn't a character itself, but a fragment of a different one.
Unicode is inherently a variable-width encoding, and later codepoints can change meaning of earlier ones.
- kazinator 4y agoIf I specify that the first character of a string is a type code A, B or C, followed by some field of data, then if the type code is A and the data starts with U+0328, it is just the type code A followed by an unrelated U+0328. That's an example of why it's important to be able to say text[0] to just get that A. It may be that if you send the whole datum to a text rendering device (say, for debugging) you get a funny result starting with the glyph Ą. If it bothers you, then don't be lazy and parse things out.
- pornel 4y agoI'm not sure what you're describing, but it's not Unicode (type code and field of data are not in Unicode jargon, so I don't know if you're just loose with terminology or describing some other system). Combining marks in Unicode are not some unrelated surprise. They're required to correctly interpret the code point they follow. Unicode requires that you to continue to consume them until they end or the input ends. In other words, Unicode is always a variable-length encoding. text[0] alone is never sufficient.