16 ms·
Let's stop ascribing meaning to code points (2017)
- upofadown 4y agoPython3 is a good example of what happens when you represent strings as an index of code points. I always found that a bit ironic in that one of the major justifications for python3 was to improve Unicode. In the end it failed to do that because of this.
- pxeger1 4y ago> what happens when you represent strings as an index of code points What happens? Are you referring to a specific problem that Python 3 has?
- chrismorgan 4y agoWorse still, Python strings aren’t even valid Unicode, because they’re sequences of code points rather than scalar values as they should be; and so even UTF–8‐encoding a str is fallible: >>> '\udead'.encode('utf-8') Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeEncodeError: 'utf-8' codec can't encode character '\udead' in position 0: surrogates not allowed
- kazinator 4y agoThat should Just Do What It's Told and produce ED BA AD.
- chrismorgan 4y agoThat would be baad, because it would not be valid UTF-8. Surrogates are a mess. Blame UTF-16. It was a terrible idea that never should have been implemented.
- morelisp 4y agoBoth you and parent are correct. (It should have been valid UTF-8, in practice it often is treated as such anyway. It should also never have existed in the first place.)
- chrismorgan 4y agoNo, it should not have been valid UTF-8, and nothing that validates UTF-8 (which is the considerable majority of things these days, though certainly not all) will accept it. UTF-16 fundamentally can’t represent surrogates: 0x0000 to 0xd7ff represent U+0000 to U+D7FF, and pairs of 0xd800–0xdbff (1024 code units) and 0xdc00–0xdfff (1024 code units) represent U+10000 to U+10FFFF (1024² code point), and there’s nothing left that could represent U+D800–U+DFFF. With UTF-16, there were two possible choices: either make UTF-16 an incomplete encoding, unable to represent some Unicode text, or separate surrogates out as reserved for UTF-16, not encodable and not permitted in Unicode text. Neither solution is good (because too much software doesn’t validate Unicode, and too much software treats UTF-16 more like UCS-2), and the correct answer would have been to throw away UCS-2 as a failed experiment and never invent UTF-16, but here we are now. They decided in favour of consistency between Unicode Transformation Formats, which is fairly clearly the better of the first two choices.
- account42 4y ago> No, it should not have been valid UTF-8, and nothing that validates UTF-8 (which is the considerable majority of things these days, though certainly not all) will accept it. Yes it should as those code points should have never been reserved. UTF-8 should only concern itself with encoding arbitrary 24-bit integers. And that those characters are reserved is also useful in practice because lots of things are encoded in UCS2 uhhm I mean "UTF-16" where unpaired surrogates are the reality. Hence: https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/ That UTF-16 can't represent those code points is UTF-16's problem and should have been solved by those clinging to their UCS2 codebases instead of being enshrined in Unicode. It would have certainly been possible to make a 16-bit Unicode encoding that is as much UCS2-compatible as UTF-16 is but can represent all code points. Also, almost nothing should validate UTF-8.
- mumblemumble 4y agoThey didn't get it perfect, but, compared to what it was like to deal with Unicode in Python 2, I'd say they absolutely succeeded at improving the experience of working with Unicode text.
- afiori 4y agoweren't python 2 string essentially unencoded byte-strings? In my uninformed opinion they should have simply added the u"..." and b"..." formats and forbid mixing them or using a u with a b operation and viceversa. python 3 could have simply changed the default from b to u for untagged strings.
- mumblemumble 4y agoThis is how Python 2 did it, only with the difference that you didn't need the b prefix for ascii/byte strings. The problem was that this made a muddle of all the string handling functions, which needed to be prepared to handle both types. Including ones you write yourself. And, Python being a dynamic language, getting this right was fiddly and error prone. "Forbid mixing the two" was a workable, if ergonomically annoying, in statically typed languages that predated and then subsequently had to adopt Unicode. For Python, though, it was an enormous minefield of runtime errors.
- tyingq 4y agoAnd you also had to detect which version of Python2, since Python <= 2.5.x didn't have some of the unicode support that later versions did.
- akubera 4y agoIsn't that what they ended up doing in Python3? $ python2 >>> "abc" + b"123" 'abc123' >>> "abc" + u"123" u'abc123' $ python3 >>> "abc" + b"123" Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: can only concatenate str (not "bytes") to str >>> "abc" + u"123" 'abc123' >>> type(u"123") <class 'str'> P.S. Your use of the word "simply" there is a reminder of how easy it is to underestimate the complexity of handling encoding properly, since it took years (a decade?) to port libraries from Py2 to Py3, since it doesn't just affect string literals, but general IO when reading from files and sockets.
- deleted 4y ago[deleted]
- torstenvl 4y ago> Folks start implying that code points mean something, and that O(1) indexing or slicing at code point boundaries is a useful operation. It isn't possible to determine where to slice a string at grapheme cluster boundaries without indexing into the code points to find their combining classes. The author's point here is self-contradictory, because if string slicing requires anything more than slicing at code point boundaries, you're going to need fast indexing to code points. What the author appears to want to argue is that the trade-offs are worth it. However, it's possible to make that argument without the bad faith claim that there's no value to the other perspective.
- gaganyaan 4y ago> The main time you want to be able to index by code point is if you’re implementing algorithms defined in the unicode spec that operate on unicode strings (casefolding, segmentation, NFD/NFC). > But for application logic, dealing with code points doesn’t really make sense. I think you're agreeing with him?
- morelisp 4y ago> if string slicing requires anything more than slicing at code point boundaries, you're going to need fast indexing to code points. Why? I can safely slice anywhere in UTF-8 code units, and align to the next (or previous) grapheme cluster just as easily. And this is actually useful if I have some storage limitations, unlike codepoints.
- chrismorgan 4y agoI think you’ve misunderstood Manish’s point (which is made clearer later in the article—the wording of the sentence you quoted is poor). Manish’s contention is that there is no virtue whatsoever in O(1) access to the nth code point, because code points aren’t a useful abstraction.
- torstenvl 4y agoI believe I understand his point, I just think he overstated his case. Consider the two forms of the following argument against UTF-32/UCS-4: A: Although reasoning about code points is more complex when dealing with UTF-8, the complexity can usually be remedied by an inner loop to iterate between code units, which is very simple in UTF-8, as (ch & 0b11000000) || !(ch & 0b10000000) being true will indicate the first byte of a code point. For this reason, the downside is limited and the upside - not having to convert to/from UTF-8, the most common encoding for Unicode text at rest - is significant. B: There is no downside to working in UTF-8. If you find it easier to reason about UTF-32/UCS-4, you're just wrong. There's no virtue whatsoever in doing it that way. Manish seems to be arguing B, and I simply can't agree. A strikes me as true enough without having to exaggerate like that.
- avgcorrection 4y agoYes, it’s definitely annoying that you can’t get the grapheme clusters of a string without going outside the standard library. That’s a very basic need when you are dealing with user input and you care about the individual “characters” that have been sent in.
- easrng 4y agoYou can't get grapheme clusters but you can get actual codepoints by spreading a string into an array, ex [..."abc\uD83C\uDF08"] is ["a", "b", "c", "\u{1F308}"]
- kazinator 4y agoIf you carefully manipulate the array to produce some different array, and then convert back to string/text, you can preserve the grapheme clusters just fine. Just like the way string manipulating routines that don't know anything about Unicode can work with UTF-8 without breaking the text, simply by not truncating or separating the data in the middle of a UTF-8 sequence.
- CoastalCoder 4y agoWhen I first read the headline, I thought "code point" referred to a specific location in a program's text. (E.g., file/line#/col#, or memory address.) It was fun watching my brain try to make sense of that meaning.
- finnh 4y agoAnalyzing code in terms of "function points" was something done back in the day. Maybe it still is; in my mind it's part of the same general dustbin as UML et al.
- dtgriscom 4y agohttps://en.wikipedia.org/wiki/Sequence_point https://en.wikipedia.org/wiki/Sequence_point
- CoastalCoder 4y agoThe concept is still fundamental to static code analysis.
- morelisp 4y agoHowso, given that function point analysis comes from requirements and doesn't even need code to exist? (Are you sure you know what function point analysis is? It sounds a lot like "sequence points" or "branching points", but in reality it's more like "story points"...)
- CoastalCoder 4y agoSorry, you may be right. Based on context I assumed "function point" was synonymous with static analysis' "code point". I should have checked before commenting.
- aaaaaaaaaaab 4y agoNope, that’s something totally different. Function point analysis is some handwavy enterprise bullshit that noone takes seriously anymore. https://en.m.wikipedia.org/wiki/Function_point https://en.m.wikipedia.org/wiki/Function_point
- wereHamster 4y ago> UTF-16 is mostly a “worst of both worlds” compromise at this point, and the main programming language I can think of that uses it (and exposes it in this form) is Javascript, and that too in a broken way. Unicode predates JavaScript. JS use UCS-2! From https://mathiasbynens.be/notes/javascript-encoding https://mathiasbynens.be/notes/javascript-encoding JavaScript engines are free to use UCS-2 or UTF-16 internally. Most engines that I know of use UTF-16, but whatever choice they made, it’s just an implementation detail that won’t affect the language’s characteristics. The ECMAScript/JavaScript language itself, however, exposes characters according to UCS-2, not UTF-16.
- morelisp 4y agoLike all UCS-2 users, it is using UTF-16 today. Strings are sequences of UTF-16 code units. (If you don't believe me, you can verify this by seeing that emoji exists.) The link uses the abhorrent term "characters" far too frequently to be saying anything meaningful.
- account42 4y agoLike all would-be UTF-16 users it actually uses WTF-16 in practice.
- chrismorgan 4y agoI never liked Mathias’s description there. A better description is that JavaScript uses unvalidated UTF-16, with UTF-16 code unit indexing and access in most places. (Notwithstanding this, new functionality in the language tends to use Unicode scalar values where possible, e.g. string iterator decodes valid surrogate pairs.)
- kazinator 4y ago> This is because code points have no intrinsic meaning. They are not “characters”. This is simply false. The true statement is: "Not all code points are characters". - Many code points are characters. For instance, everything in the ASCII range. - Codepoints in the ASCII range are often used for delimiters: quotes, commas, various brackets ... they have semantics. - Generic text manipulating routines don't require code points to have semantics, but they require indexing. We can make an analogy here to UTF-8. Let's pretend that "character" means "valid multi-byte UTF-8 code" and "code point" means "byte". Code point (i.e. byte) access to a UTF-8 string is extremely useful. UTF-8 strings can be processed by code that doesn't understand UTF-8 at all; for isntance you can split a UTF-8 string on commas or spaces using some function written in 1980. That function will use pointers or indices or some combination thereof into the string, using subroutines that just blindly copy ranges of bytes without caring what they mean. The UTF-8 won't be torn apart because the delimiters don't occur in the middle of a UTF-8 sequence.
- wonderbore 4y ago> Many code points are characters. For instance, everything in the ASCII range. Is 0x07 (BEL) a character?
- tialaramex 4y ago> Many code points are characters. For instance, everything in the ASCII range. It is definitely unclear in what sense NUL should be "a character" likewise for most of the ASCII control codes, are DELETE and ESCAPE characters? How about VERTICAL TAB? > Generic text manipulating routines don't require code points to have semantics, but they require indexing. While there have been programming languages which posit such routines, it's unclear what they mean. For example it's common to define some sort of sub-string function substr(string, A, B) and say that A and B are indexes into the code points. But, to what end? Typically such functions are dangerous (or even undefined) unless you got the indices A and B by some specialised means, suggesting actually indexing into the string isn't really how it works. In contrast Rust offers lots of text manipulation routines which are well defined but lack such indexing. Such as split_once and trim_end_matches We can see that UTF-8 has code units of bytes, and sure enough indexing the code units makes sense, but insisting these should be considered an analogy to code points misses why we have two different terms.
- zajio1am 4y agoFundamental problem with grapheme clusters as a basic programming language concept is that they are not uniformly defined, they depend on grapheme_extend property of unicode characters, which are known only for already defined codepoints. Which means that such splitting is problematic to be done in forward-compatible manner. It is also costly and in many cases not necessary. Which makes it more suitable to be provided by a library than to be a core language concept. Secondary issue is that with codepoints (or bytes) as basic concept one can compose grapheme clusters in the same manner as general text composing, while if grapheme clusters are basic concept then one needs specialized operation for composing them from codepoints. OTOH, pretty much agree that O(1) access to code point is not important and keeping internal representation in UTF-8 (or in input encoding) is OK for most purposes.
- dahfizz 4y agoUnpopular opinion: written languages need to modernize. The printing press and typewriter forced the Romantic language from a format optimized for hand writing (cursive) into a format optimized for the modern world (print). I think other languages would benefit from having a "print" format.
- O__________O 4y agoCursive didn’t go away with the printing press. Non-romantic languages already “print” just fine too. If ASCII is for some reason needed, most non-romantic languages have a system for romanization. Cursive versions of romantic languages exist in unicode, they’re just neither popular, nor as well designed as languages with demand for script versions. In my opinion, if any change was going to happen, switching to something similar to the International Phonetic Alphabet would make the most sense. Then any languages would be readable; already easy to encode any language in it, just not the norm.
- wlib 4y agoWhat about a Hangul inspired version of IPA? https://en.wikipedia.org/wiki/Hangul https://en.wikipedia.org/wiki/Hangul
- O__________O 4y agoHangul covers Korean language, IPA covers all languages. For example, some languages use clicks; though to be fair, IPA only currently covers 5 of the 6 clicks. Again, point is not IPA is the solution, but that having a common notation would make learning languages much easier and learning only one written notation would be needed.
- wlib 4y agoI just wanted to use a concrete example of a featural writing system. It would of course not be Hangul if it had to accommodate all phonemes. The other direction to take would be to create a sophisticated (probably ML) model that could enable certain features to be optimized, like speed of syllable recognition. If we are constructing a modern writing system then that would definitely be a good approach.
- tialaramex 4y agoIt's also easier to get away with Rust's decision to say no, strings aren't sequences of code points (which they aren't) if you do all the work to support ASCII on bytes anyway. Rust defines things like is_ascii_hexdigit() on both char (a Unicode scalar) and u8 (a byte) and so if you're writing some low-level code which cares only about bytes you aren't expected to either turn them into a string to find out if the byte you're looking at is an ASCII digit or improvise something. This sort of thing means the programmer who is moving bytes and is very angry about the notion of character encoding needn't touch Rust's Unicode stuff, while the programmer who is responsible for text rendering isn't given a bunch of undifferentiated bytes and told "Good luck". Somebody needs to figure out what the encoding is, but likely that programmer actually cares about the difference between ISO-8859-1 and Windows code page 1252 or at least is aware that somebody else might care.
- ThrowawayTestr 4y agoWhy isn't UTF-32 used everywhere for everything? Is an extra 3 bytes per character really that much?
- aaaaaaaaaaab 4y agoBecause combining characters would still be a problem in UTTF-32.
- Groxx 4y ago^ yeah, this is pretty much the answer. 32 bits is not sufficient, emoji and combining characters go further. Even with UTF-32 you still can't treat "one 32 bit sequence" as "one character/grapheme/etc". It solves nothing, but it takes more space in the vast majority of cases.
- wtetzner 4y agoWhat would it buy you? And yes, for text that’s mostly ASCII, you’d be quadrupling memory usage.
- ThrowawayTestr 4y agoRight, memory. I was only thinking of disk space.
- Groxx 4y agoCompletely agreed, and this is a pretty nice overview of the problems with thinking of strings in terms of "characters" or "code points". Graphemes are almost always the closest to what people mean when they say "character". But you also shouldn't do logic based on grapheme, unless you're contributing to harfbuzz and know enough to know exactly why this advice is wrong. Don't split or concatenate strings for any reason if you're doing internationalized stuff. E.g. ask your translators to give you a separate string for a drop-cap, and to remove it from the string that follows, do not just pluck out the first grapheme because it could look like nonsense. --- You should literally never need or want to interact with Unicode directly, unless you're building the foundational layers of other systems (rendering, Unicode normalization, etc). If you find that you have to, you're probably losing encoding information somewhere - that loss is the bug to fix, don't try to patch it somehow, you'll just cause weirder errors elsewhere - or doing something fundamentally irrational, like splitting a string somehow. The never-ending pain you encounter while doing this stuff is a sign you're Doing It Wrong™ and should step back and question the basics, not that it just needs one more fix to work correctly. If you're doing single-language logging for developers or whatever? Yeah, go wild. Though watch out for irrationally chopped user input, ya gotta make sure your log analyzer won't choke on bad UTF-8.
- thaumasiotes 4y ago> Graphemes are almost always the closest to what people mean when they say "character". But which glyphs[1] are considered graphemes and which are composites varies according to the individual writing system, which is a finer level than Unicode deals with. In French, é, è, and à are all single graphemes; they are three different vowels and those are their written forms. It's just a coincidence that é and è appear to share an "e" element, and it's also a coincidence that è and à appear to share a diacritic marking. The grave accent has no meaning in isolation, and while the "e" does, its meaning is separate from and unrelated to the meaning of either é or è. But in Hanyu Pinyin, é is two graphemes, è is two graphemes, and à is two graphemes. (With a total of four graphemes being represented across the set.) É and è share their underlying vowel, while è and à express different vowels but share their underlying tone. You could demand that strings representing Chinese always use U+0301 COMBINING ACUTE ACCENT alongside U+0065 LATIN SMALL LETTER E, while strings representing French must always use U+00E8 LATIN SMALL LETTER E WITH ACUTE. But that would be insane. Unicode can't generally distinguish graphemes from composite glyphs and doesn't want to. [1] In this comment, I will define "glyph" to be the unit of output produced by a font. You put some kind of binary data into the font and get image data out; anything that is considered a single indivisible unit of output from the font is a "glyph".
- cryptonector 4y ago> However, you don’t need code point indexing here, byte indexing works fine! UTF8 is designed so that you can check if you’re on a code point boundary even if you just byte-index directly. Yes, UTF-8 is self-resynchronizing in either direction. I.e., if you pick a random byte index into a UTF-8 string, you can check if that byte's value is the start of a codepoint, and if not then you can scan backwards (or forward) to find the start of the current (or next) codepoint. Do be careful not to overrun the bounds of the string, if you're writing C code anyways. Do note that start-of-codepoint is not the same thing as start-of-grapheme-cluster or start-of-character. Also, TFA doesn't touch at all on forms and normalization. And I think TFA is confused about "meaning". Unicode codepoints very much have meaning, and that had better not change. TFA seems to be mostly about indexing into strings, and that content is fine. > One very common misconception I’ve seen is that code points have cross-language intrinsic meaning. Well, maybe that depends on what the meaning of "meaning" is. Unicode codepoints have normative meanings -- that is, meanings assigned by the Unicode Consortium. Those "meanings" are embodied in a) their names and descriptions, b) their normative glyphs. Some codepoints are combining codepoints, and their meaning lies in the changes to the base codepoint's glyph when applied -- this is certainly a kind of "meaning". But of course people can use codepoints (really, characters) in ways which do not comport to the meanings assigned by the UC. That's fine, of course. The real meanings of the words we write and how we use them, and the glyphs they are composed of, vary over time because human language evolves. But TFA writes "cross-language intrinsic meaning". That's a more specific claim that is more likely to be true. It's certainly true that Indic language glyphs have no meaning to me when mixed with Latin scripts, since I know nothing about them, though I can look up their meaning, and I can learn about them. And it's also true that confusable codepoint assignments (e.g., some Greek charaters look just like Latin characters, and vice-versa) may not, for the reader, have the UC's intended meaning, since to the reader the only meaning will only be the rendered glyph's rather than the codepoint's! After all, being confusable, the reader isn't likely to notice that some glyph is not Latin but Greek (or whatever). But if some text does not mix confusable scripts with the intent of creating confusion, and if it is clear to the reader what is intended, then a reader familiar with the scripts (and languages) being used in the text will be able to discern intended meaning, and the UC-assigned meanings of the codepoints used will be... meaningful even though the text is mixed-script text. So, yes, with caveats, Unicode codepoints do have "cross-language intrinsic meaning". IMO, it would have been better for TFA not to mix the two things, codepoint meaning and string indexing issues. Also, string indexing is just not needed. In general you have to parse text from first code unit to last. You might tokenize text, and then indexing into a sequence of tokens might have meaning / be useful, but indexing into UTF-8/16/32 by code unit is not really useful.
- josephg 4y agoAs someone who's written a lot of unicode-aware code in multiple languages, I don't agree at all. The problem I think a lot about is collaborative text editing. I need to express an insert into a text document - eg, "Insert 'a' at position X". The obvious question that comes up is - how should we define "X"? There's a mess of options: - UTF-8 byte offset (from the start of the string). (Or UCS2 offset) - Extended grapheme cluster offset - Code point offset You can also use a line/column position, but the column position ends up being one of the 3 above. I want collaborative editing protocols to work across multiple programs, running on multiple systems. (Eg, a user is in a web browser on a phone, talking to a server written in rust.) This post says that codepoint offsets are Bad, but in my mind they're the only sane answer to this problem. Using byte offsets has two problems: 1) You have to pick an encoding, and thats problematic for cross-language compatibility. UTF-8 byte offsets are meaningless in javascript - they're slow & expensive to convert. UCS2 offsets are meaningless in rust. 2) They make it possible to express invalid data. Inserting in the middle of a codepoint is an invalid operation. I don't want to worry about different systems handling that case differently. Using codepoint offsets make it impossible to even represent the idea of inserting in the middle of a character. Using grapheme cluster offsets is problematic because the grapheme clustering rules change all the time. I want editing operations to be readable from any programming language, and across time. Saying "Insert 'a' at position 20" (measured in grapheme clusters) is ambiguous because "position 20" will drift as unicode's grapheme cluster rules change. The result is that old android phones and new iphones can't reliably edit a document together. And old editing traces can't be reliably replayed. Measuring codepoints is better than the other options because its stable, cross platform and well defined. If you aren't aware of those benefits, you aren't understanding my use case.
- jhgb 4y ago> I need to express an insert into a text document - eg, "Insert 'a' at position X". The obvious question that comes up is - how should we define "X"? Perhaps using a cursor/rider/however-you-call-it into whatever data structure you're using for editable sequences (since there's a bunch of them) is the preferred solution here? I'm hardly an editor expert but I thought that's how this has been being done for decades (and for some possibly useful but really complicated text representations, numerical indices for positions of operations wouldn't really work very well anyway).
- hgs3 4y agoThe author must have missed the part of the Unicode spec where it ascribes dozens of properties to code points: it assigns them a general category (e.g. control character, lower case letter, punctuation, and more), a case mapping (if applicable), a numeric value (if applicable) and the kind of number it is (ordinal or otherwise), the script they are typically written in, and many more. I think the confusion with Unicode is that programmers apply their non-technical preconception of what a "character" is rather than understanding the Unicode definition. Unicode defines a character as a representation of something - maybe it represents a line break (control character), a letter, an ideographic, a combining mark, or something else. A "code point" is just a number that a character is assigned for numerical representation in memory. A "grapheme cluster" is what a user perceives as a character - it's how non-programmers see text. What needs to happen is programmers need to hammer in their heads the Unicode definition of "character" just like they relearned to count from zero.
- chrismorgan 4y agoYou’ve missed the point of the article, which is that optimising for code point indexing is silly. The author is well aware of all the properties you speak of, but they’re seldom things that end-developers should be touching directly, and none of them benefit from indexing by code point offset.