4 ms·
> This is because code points have no intrinsic meaning. They are not “characters”. This is simply false. The true statement is: "Not all code points are chara
by kazinator 4y ago
> This is because code points have no intrinsic meaning. They are not “characters”.
This is simply false. The true statement is: "Not all code points are characters".
- Many code points are characters. For instance, everything in the ASCII range.
- Codepoints in the ASCII range are often used for delimiters: quotes, commas, various brackets ... they have semantics.
- Generic text manipulating routines don't require code points to have semantics, but they require indexing.
We can make an analogy here to UTF-8. Let's pretend that "character" means "valid multi-byte UTF-8 code" and "code point" means "byte".
Code point (i.e. byte) access to a UTF-8 string is extremely useful.
UTF-8 strings can be processed by code that doesn't understand UTF-8 at all; for isntance you can split a UTF-8 string on commas or spaces using some function written in 1980. That function will use pointers or indices or some combination thereof into the string, using subroutines that just blindly copy ranges of bytes without caring what they mean. The UTF-8 won't be torn apart because the delimiters don't occur in the middle of a UTF-8 sequence.
- wonderbore 4y ago> Many code points are characters. For instance, everything in the ASCII range. Is 0x07 (BEL) a character?
- tialaramex 4y ago> Many code points are characters. For instance, everything in the ASCII range. It is definitely unclear in what sense NUL should be "a character" likewise for most of the ASCII control codes, are DELETE and ESCAPE characters? How about VERTICAL TAB? > Generic text manipulating routines don't require code points to have semantics, but they require indexing. While there have been programming languages which posit such routines, it's unclear what they mean. For example it's common to define some sort of sub-string function substr(string, A, B) and say that A and B are indexes into the code points. But, to what end? Typically such functions are dangerous (or even undefined) unless you got the indices A and B by some specialised means, suggesting actually indexing into the string isn't really how it works. In contrast Rust offers lots of text manipulation routines which are well defined but lack such indexing. Such as split_once and trim_end_matches We can see that UTF-8 has code units of bytes, and sure enough indexing the code units makes sense, but insisting these should be considered an analogy to code points misses why we have two different terms.
- mjw1007 4y agoFunctions that index into the string-considered-as-a-sequence-of-characters can be meaningful, and sometimes useful, in cases where you know that the string contains only characters from some simple restricted subset. (You might know that "by construction", or by checking.) In a language like Rust which doesn't directly support refinement types, it's common to end up using the standard str or String types in cases where you know such a thing. So I think there are times when that sort of indexing function would be sensible. A common example is when you're working with a very restricted subset of printable ASCII (things like stock codes, ISBNs, EANs). Or not so long ago I was working with 64-character-long strings containing only U+0020 and U+2654 ..= U+265F .
- kazinator 4y ago> in cases where you know that the string contains only characters from some simple restricted subset But I'm saying, even in cases where you don't! A string function that works with byte characters, written before Unicode existed, can do useful processing on UTF-8 data which contains characters that didn't exist when that code was written.
- mjw1007 4y agoThat's true, but in most of those cases you don't need to be able to use numeric character-count-based indexes into the string (which is what the article is arguing that you don't need). You'd typically be happy if the parsing function that you're using to find the location of (say) each comma in the string gives you an opaque token for each such location, with a way to use those tokens to get slices of the string back. So in practice we can use byte offsets into a UTF-8 string as those tokens, while the programmer doesn't really have to care that that's what they are.
- kazinator 4y agoNUL, DEL and ESC are control characters. > For example it's common to define some sort of sub-string function substr(string, A, B) and say that A and B are indexes into the code points. But, to what end? Right; this is what I'm getting at. substr(str, A, B) is a low level function: a servant. A and B have a meaning somewhere, but not to this routine. substr does not need to know what the indices mean. The indices could mean "start and end of a file suffix" or "third field in a line of CSV", or "integer part of floating-point token". Software is layered and not all the semantics can be known to every layer. Usually when we have semantic leakage/sharign across crossings, it's for some optimization purpose. For instance, the ethernet driver and hardware don't know that the frames passing through them are IP or whatever else. (Though, for instance, when you have TCP offloading to the hardware, then the hardware groks that semantics info.)
- deleted 4y ago[deleted]
- mseepgood 4y ago> We can make an analogy here to UTF-8. Let's pretend that "character" means "valid multi-byte UTF-8 code" and "code point" means "byte". That's not the usual terminology. For UTF-8 the "valid multi-byte UTF-8 code" is the "code point" and the byte is the "code unit".
- tuukkah 4y agoIt was an analogy: not all bytes in UTF-8 have a meaning but some do.
- pornel 4y agoYou can't assume that any code point is a character. This is because of combining marks. If you have a string with an ASCII 'A' codepoint followed by U+0328 codepoint, then it's a two-codepoint 'Ą' character. The first code point wasn't a character itself, but a fragment of a different one. Unicode is inherently a variable-width encoding, and later codepoints can change meaning of earlier ones.
- kazinator 4y agoIf I specify that the first character of a string is a type code A, B or C, followed by some field of data, then if the type code is A and the data starts with U+0328, it is just the type code A followed by an unrelated U+0328. That's an example of why it's important to be able to say text[0] to just get that A. It may be that if you send the whole datum to a text rendering device (say, for debugging) you get a funny result starting with the glyph Ą. If it bothers you, then don't be lazy and parse things out.
- pornel 4y agoI'm not sure what you're describing, but it's not Unicode (type code and field of data are not in Unicode jargon, so I don't know if you're just loose with terminology or describing some other system). Combining marks in Unicode are not some unrelated surprise. They're required to correctly interpret the code point they follow. Unicode requires that you to continue to consume them until they end or the input ends. In other words, Unicode is always a variable-length encoding. text[0] alone is never sufficient.