3 ms·
> Many code points are characters. For instance, everything in the ASCII range. It is definitely unclear in what sense NUL should be "a character" likewise for
by tialaramex 4y ago
> Many code points are characters. For instance, everything in the ASCII range.
It is definitely unclear in what sense NUL should be "a character" likewise for most of the ASCII control codes, are DELETE and ESCAPE characters? How about VERTICAL TAB?
> Generic text manipulating routines don't require code points to have semantics, but they require indexing.
While there have been programming languages which posit such routines, it's unclear what they mean. For example it's common to define some sort of sub-string function substr(string, A, B) and say that A and B are indexes into the code points. But, to what end? Typically such functions are dangerous (or even undefined) unless you got the indices A and B by some specialised means, suggesting actually indexing into the string isn't really how it works.
In contrast Rust offers lots of text manipulation routines which are well defined but lack such indexing. Such as split_once and trim_end_matches
We can see that UTF-8 has code units of bytes, and sure enough indexing the code units makes sense, but insisting these should be considered an analogy to code points misses why we have two different terms.
- mjw1007 4y agoFunctions that index into the string-considered-as-a-sequence-of-characters can be meaningful, and sometimes useful, in cases where you know that the string contains only characters from some simple restricted subset. (You might know that "by construction", or by checking.) In a language like Rust which doesn't directly support refinement types, it's common to end up using the standard str or String types in cases where you know such a thing. So I think there are times when that sort of indexing function would be sensible. A common example is when you're working with a very restricted subset of printable ASCII (things like stock codes, ISBNs, EANs). Or not so long ago I was working with 64-character-long strings containing only U+0020 and U+2654 ..= U+265F .
- kazinator 4y ago> in cases where you know that the string contains only characters from some simple restricted subset But I'm saying, even in cases where you don't! A string function that works with byte characters, written before Unicode existed, can do useful processing on UTF-8 data which contains characters that didn't exist when that code was written.
- mjw1007 4y agoThat's true, but in most of those cases you don't need to be able to use numeric character-count-based indexes into the string (which is what the article is arguing that you don't need). You'd typically be happy if the parsing function that you're using to find the location of (say) each comma in the string gives you an opaque token for each such location, with a way to use those tokens to get slices of the string back. So in practice we can use byte offsets into a UTF-8 string as those tokens, while the programmer doesn't really have to care that that's what they are.
- kazinator 4y agoNUL, DEL and ESC are control characters. > For example it's common to define some sort of sub-string function substr(string, A, B) and say that A and B are indexes into the code points. But, to what end? Right; this is what I'm getting at. substr(str, A, B) is a low level function: a servant. A and B have a meaning somewhere, but not to this routine. substr does not need to know what the indices mean. The indices could mean "start and end of a file suffix" or "third field in a line of CSV", or "integer part of floating-point token". Software is layered and not all the semantics can be known to every layer. Usually when we have semantic leakage/sharign across crossings, it's for some optimization purpose. For instance, the ethernet driver and hardware don't know that the frames passing through them are IP or whatever else. (Though, for instance, when you have TCP offloading to the hardware, then the hardware groks that semantics info.)
- tialaramex 4y ago> NUL, DEL and ESC are control characters. You have confused nomenclature for category. Thus begging the question. > Right; this is what I'm getting at. substr(str, A, B) is a low level function: a servant. A and B have a meaning somewhere, but not to this routine. For the purpose you're talking about the code point index is completely irrelevant, and so the byte index works fine here.
- kazinator 4y agoA character isn't just something that produces a picture, and advances the printing position to the right. Yes, a byte index will work if we have a multi-byte string such as UTF-8; and a code point index would be inconvenient since displacement calculations on it are meaningless. If we are talking about a string as an array of code points (like a wchar_t * string on sane platforms where a wchar_t can hold any code point) then we use code point indexing. C has wmemcpy, wmalloc, wstrlen, ...
- spc476 4y agoI have a graphical window, and I have a bunch of text (in whatever Unicode encoding you want---it doesn't matter to me) that I need to display in said graphical window such that the text just doesn't run off one edge of the window and can't be read. How do I break the text up into segments that will fit within the horizontal width of the graphical window without changing the meaning of any of the text? Fourty years ago I could assume ASCII and be done with it. These days? With the yearly changes to the "Rules of Unicode"? Screw that, let's build a self-landing rocket which appears to be easier these days than actually displaying text.