11 ms·

I'm getting tired of this argument, we had it in the Rust community as well and I was tired of it there, too. I've seen the same argument over and over again,

I'm getting tired of this argument, we had it in the Rust community as well and I was tired of it there, too. I've seen the same argument over and over again, and every argument had the same hole that I see in your post: What, exactly, is a "character"? Please answer me that, and then you can write a proposal for how the API should work, exactly, and why this works with Indic and German and Korean and everything else. Strings have to be arrays. It is inevitable. We are on Von Neumann machines with L1 cache lines that are something like 64 bytes wide, so making strings into something other than arrays is a complete non-starter. Because strings are arrays, we use array indexes to slice strings. The indexes are integers, because that just makes sense on a Von Neumann machine. So your complaint is that you can slice a string in "bad" ways. What makes it bad? You're trying to prevent people from slicing up grapheme clusters, which is a noble goal. But in practice, we get the indexes from methods like str.find() or re.match(), and we treat them as more or less opaque, so we don't end up slicing grapheme clusters very often in practice. Grapheme cluster slicing requires a big Unicode table in memory anyway, so in the rare case that you need it, you can pay the high performance cost for using it. In the meantime, formats like JSON and XML are defined in terms of code points, so using code point arrays eliminates a class of bugs where you could accidentally make malformed XML or JSON, which would then get completely REJECTED by the receiving side, causing your Jabber client to quit or your web browser to show a bunch of mojibake. And let me ask you this: what do I get when I write: x = "a" y = "\u0301" Is the resulting len(x + y) == 1? But len(x) == 1 and len(y) == 1? Can you tell me what the correct behavior is? Might you end up with bugs in programs because len(x + y) != len(x) + len(y)? Or do you introduce extra code points into the string when you concatenate them? Please, tell me what you actually think is correct behavior. It is far, far more useful than pointing out that something is "wrong" on purely semantic arguments.

anon1385 11y ago
The correct answer is that strings shouldn't have length methods. There is no such thing as the 'length' of a string because as you pointed out there is no clear definition of what a 'character' is. https://www.mikeash.com/pyblog/friday-qa-2015-11-06-why-is-swifts-string-api-so-hard.html https://www.mikeash.com/pyblog/friday-qa-2015-11-06-why-is-s...
- the_mitsuhiko 11y ago
  To add to that: O(1) indexing into strings outside of the ASCII range is completely pointless and makes absolutely no sense. A language should disallow that and not encourage it.
  - Veedrac 11y ago
    I disagree with it being pointless. Being able to get indices into a string for fast slicing is useful. Consider Regex where references get stored to slices in the array. What you can drop is the idea of the index being an integer with any particular semantic meaning. Rust uses byte indexes with an assertion that the index lies on a codepoint boundary, for instance.
    - the_mitsuhiko 11y ago
      > Being able to get indices into a string for fast slicing is useful. Outside of the ASCII range? I doubt it. That most likely means you are doing something fundamentally wrong.
      - heinrich5991 11y ago
        I think something like a lexer doesn't need to know about graphemes, it just needs to know which codepoints are XID_start and XID_continue for identifiers and hardcoded codepoints for everything else. EDIT: Or a JSON parser, etc.
        the_mitsuhiko 11y ago
        A lexer scans LTR. It never jumps codepoints ahead. Indexing in a lexer never makes sense.
        tomp 11y ago
        Obviously, it does; instead of allocating a new string for every token, you can just save the start & end index of a token, and actually allocate a new string only in the case where the parser actually requests/uses that token in the AST.
hahainternet 11y ago
> Please, tell me what you actually think is correct behavior. It is far, far more useful than pointing out that something is "wrong" on purely semantic arguments. You say "wrong" as if that's a quote from me. But it's not. Yes, strings have to be arrays, but how they work depends on the semantics of your language. I'm not entirely happy with Python 3 because I don't think they really took their time to improve on the warts. I doubt there's a one-size-fits-all solution.
lucozade 11y ago
> Because strings are arrays, we use array indexes to slice strings But that's a choice and not the only one. Sure you're likely to implement your Unicode string as an array of some type. But that doesn't mean that the only sensible approach is to expose that array directly to the user. Or more specifically as the primary interface. For my money, Swift has probably the most comprehensive Unicode string API I've seen [0]. What they do is, essentially, have an opaque String type but support various "views" as properties. The main view is called .characters and represents a collection of grapheme clusters. They also have properties that present Unicode scalars, utf8 encoding etc. Their API is complex, no question. But then proper handling of Unicode is complex. But it does show that there are other options than simply exposing the Unicode scalars. BTW to your point on len(x) + len(y). The answer is 2 if you define len() on Unicode scalars but 1 if you define it on characters. Why? because len("\u301") should be 0. It's not a character, it's a base modifier. It is, of course, true that getting this right is likely to be substantially more expensive than getting it wrong but that doesn't mean it can't be done. [0] https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swift_Programming_Language/StringsAndCharacters.html https://developer.apple.com/library/ios/documentation/Swift/...
- klodolph 11y ago
  len("\u0301") == 0 is extremely surprising.
  - lucozade 11y ago
    Of course. What you were expecting was 6 right? You learnt ASCII. Then you learnt about escaping to get around limitations in ASCII. Then you learnt that Unicode got around non-Latin issues with ASCII. Each step has added cognitive load. It seems surprising to me that the next step wouldn't.
    - klodolph 11y ago
      The surprise comes with a cost, and the cost is in bugs. The point of Python 3 is to eliminate some bugs in Python 2, and forcing developers to redesign their code around a completely new string API--no matter if that new API is more semantically sound--will come with its own cost of introduced bugs and more painful conversions, which goes against the purpose of Python 3. I'm not saying that the string API in Python is perfection, but it had a specific role to fill in the larger Python ecosystem, and it does that very well. Anyway... I would say that len(x) == 0 should be the same as saying that x is empty. According to the Unicode standard, the number of grapheme clusters in "\u0301" is 1, anyway! And on most systems, the string will display as an acute accent in its own space, and you can place the cursor before or after it. Not like the set of valid "cursor positions" in a string isn't a locale-dependent value, anyway. Grapheme clusters can be thought of to be locale-independent but they are NOT the same as cursor positions.
nostrademons 11y ago
Swift has very sane way of doing this: Strings are sequences of Characters. Characters represent extended grapheme clusters. Iteration consumes as many bytes from the string as are necessary to read the next grapheme. In most cases, this is straight-up UTF-8 decoding, which can be implemented very efficiently; you only need to worry about extended grapheme clusters when you get to the non-ASCII subset of Unicode, so the code paths that require unicode table lookups are infrequently exercised. String searching & matching return an opaque String.Index type. I assume that under the hood they do Boyer-Moore on bytes and String.Index is a byte offset, but the important thing is that String.Index values are not convertible to integers, and so you never run into the case where a user passes in a byte offset that would slice a grapheme in half. Instead, String.Index has properties for the next and previous index and a method to advance by an arbitrary amount, so you'd access the 7th character after a dash in a string as myString[myString.rangeOfString("-").advanceBy(7)] Swift gives up on the idea that len(x+y) = len(x) + len(y). This is just something you have to remember; in UI programming, however, it makes a lot of sense because adding an accent to an existing string isn't going to make it take up more width in the text box. (x + y).utf8.count == x.utf8.count + y.utf8.count, however. https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swift_Programming_Language/StringsAndCharacters.html https://developer.apple.com/library/ios/documentation/Swift/... If I were to adapt this to a domain like Python's, where string processing is pretty important, I'd allow indexing & slicing by integers (including negative integers), but I'd define them in terms of String.Index types, which iterate over extended graphemes under the hood: str[2] == str[str.start.advanceBy(2)] str[-3] == str[str.end.advanceBy(-3)] str[str.find('foo') + 3] == str[str.find('foo').advanceBy(3)] str[0:-3] == str.substring(str.start, str.end.advanceBy(-3)) del str[0:3] == str.removeRange(str.start, str.start.advanceBy(3))
- Veedrac 11y ago
  > Swift gives up on the idea that len(x+y) = len(x) + len(y). This is just something you have to remember; in UI programming, however, it makes a lot of sense because adding an accent to an existing string isn't going to make it take up more width in the text box. (x + y).utf8.count == x.utf8.count + y.utf8.count, however. So do fullwidth characters have a length of 2? If not, how is this useful for finding the size of text in a text box anyway?
lmm 11y ago
Python is an abstraction a long way above the Von Neumann machine. It should not expose implementation details like the number of bytes a string happens to be stored as. The whole point of the Python 2 -> 3 transition was to disallow just treating a string as a random pile of bytes and hoping it all works out.
catnaroek 11y ago
It's possible to do better than existing mainstream solutions: (0) Dependent types: The type of valid indices into a string depends, well, on the string. It's this type, not `uint`, that should be used by functions like `str.find()` and `re.match()`. This makes it a compile error to, say, slice a string with an index to another string. Internally, or course, all indices are represented as `uint`s, but your program has no business knowing this. It's an abstract type. IMO, this solution is particularly useful for systems languages, where the increased flexibility and fine-grained control are worth the increase in complexity. (1) Coalgebras: `str.find()` shouldn't return an index. Instead, it should return in O(1) time, and allocating O(1) additional space, the left and right slices of the original string, split at the desired index. (If the string can be split in multiple places, `str.find()` should return an iterator whose element type is such left-right pairs.) For this to work, it's fundamental that the left and right slices not be allocated in their own buffers, but instead share the original string's buffer. IMO, this solution is best for high-level languages, where simplifying the API is well worth a little loss in flexibility. Neither of these APIs allows the programmer to split strings at the wrong places.
raiph 11y ago
> I'm getting tired of this argument Perhaps this is because Unicode is an exhaustive standard. :) Seriously, I hear ya. > every argument had the same hole that I see in your post: What, exactly, is a "character"? Please answer me that I'd say the answer varies depending on who's talking, what they're talking about, and who's listening. If it's a programmer interested in listening to the Unicode consortium and interested in what Unicode.org documents define for "what a user thinks of as a character" then the answer is, according to those documents, a "grapheme". > then you can write a proposal for how the API should work, exactly Right. It can get, and has indeed gotten, better than that; one can write specifications, build reference implementations, and try things out in battle for a few years. Several text processing systems (including some programming languages) have gone this route and their results should be taken in to account. (ICU is perhaps the go to reference implementation.) > and why this works with Indic and German and Korean and everything else. For designs and implementations of text processing systems that claim an aspiration of progressing toward fully following the Unicode specification, the simplest answer to "why does it work?" (or "why it is expected to work") with Indic and German and Korean and everything else is of course the Unicode standard itself. For those of us who aren't blessed with the patience of a saint and thus haven't read the entire spec from start to finish a few times, one has to rely to a degree on those who do, even if it's really hard to follow what the heck they're saying. > You're trying to prevent people from slicing up grapheme clusters, which is a noble goal. Given that grapheme clusters correspond to what the Unicode standard specifies for identifying and preserving the integrity of "what a user thinks of as a character", it seems less a noble goal than a fundamental eventual goal for any text processing system that explicitly claims to aspire to support the world's human languages via Unicode compliance. > we don't end up slicing grapheme clusters very often in practice. Which "we" are you referring to? The western world has dominated text processing to date does. This must not mean that "we", by which I mean all humans, would best continue practices that marginalize those with native languages not well served. (By ignoring the central importance of "what a user thinks of as a character"; again, I'm quoting Unicode consortium documentation.) Or, more parochially, what about the "we" that's just programmers who want to be able to increase their confidence that they are properly processing Unicode compliant text? > In the meantime, formats like JSON and XML are defined in terms of code points ... Code points are the middle level of Unicode, sandwiched between byte encodings at the bottom level and graphemes at the top. There's nothing wrong with a text processing system stopping at the middle level provided folk don't lose sight of the fact that it stops at the middle level. Thinking that code points are "what a user thinks of as a character" mostly works out OK when processing text that's English or the like, but is definitively not Unicode compliant to the degree it's used too broadly. > Can you tell me what the correct behavior is? I would suggest that the most appropriate authority is the Unicode consortium and its Unicode standard. > Might you end up with bugs in programs because len(x + y) != len(x) + len(y)? Oh for sure. But it's part of the Unicode standard. It's one of several apparently strange discontinuities that the Unicode effort introduced last century in an attempt to deal with the combination of human language reality and computer system realities. They may have chosen the wrong sweetspot. But now Unicode is a significant part of reality. I, for one, don't think it's wise or even realistic to abandon the Unicode standard as it is...

anon1385 11y ago

The correct answer is that strings shouldn't have length methods. There is no such thing as the 'length' of a string because as you pointed out there is no clear definition of what a 'character' is. https://www.mikeash.com/pyblog/friday-qa-2015-11-06-why-is-swifts-string-api-so-hard.html https://www.mikeash.com/pyblog/friday-qa-2015-11-06-why-is-s...

the_mitsuhiko 11y ago
To add to that: O(1) indexing into strings outside of the ASCII range is completely pointless and makes absolutely no sense. A language should disallow that and not encourage it.
- Veedrac 11y ago
  I disagree with it being pointless. Being able to get indices into a string for fast slicing is useful. Consider Regex where references get stored to slices in the array. What you can drop is the idea of the index being an integer with any particular semantic meaning. Rust uses byte indexes with an assertion that the index lies on a codepoint boundary, for instance.
  - the_mitsuhiko 11y ago
    > Being able to get indices into a string for fast slicing is useful. Outside of the ASCII range? I doubt it. That most likely means you are doing something fundamentally wrong.
    - heinrich5991 11y ago
      I think something like a lexer doesn't need to know about graphemes, it just needs to know which codepoints are XID_start and XID_continue for identifiers and hardcoded codepoints for everything else. EDIT: Or a JSON parser, etc.
      - the_mitsuhiko 11y ago
        A lexer scans LTR. It never jumps codepoints ahead. Indexing in a lexer never makes sense.
        tomp 11y ago
        Obviously, it does; instead of allocating a new string for every token, you can just save the start & end index of a token, and actually allocate a new string only in the case where the parser actually requests/uses that token in the AST.

hahainternet 11y ago

> Please, tell me what you actually think is correct behavior. It is far, far more useful than pointing out that something is "wrong" on purely semantic arguments. You say "wrong" as if that's a quote from me. But it's not. Yes, strings have to be arrays, but how they work depends on the semantics of your language. I'm not entirely happy with Python 3 because I don't think they really took their time to improve on the warts. I doubt there's a one-size-fits-all solution.

lucozade 11y ago

> Because strings are arrays, we use array indexes to slice strings But that's a choice and not the only one. Sure you're likely to implement your Unicode string as an array of some type. But that doesn't mean that the only sensible approach is to expose that array directly to the user. Or more specifically as the primary interface. For my money, Swift has probably the most comprehensive Unicode string API I've seen [0]. What they do is, essentially, have an opaque String type but support various "views" as properties. The main view is called .characters and represents a collection of grapheme clusters. They also have properties that present Unicode scalars, utf8 encoding etc. Their API is complex, no question. But then proper handling of Unicode is complex. But it does show that there are other options than simply exposing the Unicode scalars. BTW to your point on len(x) + len(y). The answer is 2 if you define len() on Unicode scalars but 1 if you define it on characters. Why? because len("\u301") should be 0. It's not a character, it's a base modifier. It is, of course, true that getting this right is likely to be substantially more expensive than getting it wrong but that doesn't mean it can't be done. [0] https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swift_Programming_Language/StringsAndCharacters.html https://developer.apple.com/library/ios/documentation/Swift/...

klodolph 11y ago
len("\u0301") == 0 is extremely surprising.
- lucozade 11y ago
  Of course. What you were expecting was 6 right? You learnt ASCII. Then you learnt about escaping to get around limitations in ASCII. Then you learnt that Unicode got around non-Latin issues with ASCII. Each step has added cognitive load. It seems surprising to me that the next step wouldn't.
  - klodolph 11y ago
    The surprise comes with a cost, and the cost is in bugs. The point of Python 3 is to eliminate some bugs in Python 2, and forcing developers to redesign their code around a completely new string API--no matter if that new API is more semantically sound--will come with its own cost of introduced bugs and more painful conversions, which goes against the purpose of Python 3. I'm not saying that the string API in Python is perfection, but it had a specific role to fill in the larger Python ecosystem, and it does that very well. Anyway... I would say that len(x) == 0 should be the same as saying that x is empty. According to the Unicode standard, the number of grapheme clusters in "\u0301" is 1, anyway! And on most systems, the string will display as an acute accent in its own space, and you can place the cursor before or after it. Not like the set of valid "cursor positions" in a string isn't a locale-dependent value, anyway. Grapheme clusters can be thought of to be locale-independent but they are NOT the same as cursor positions.

nostrademons 11y ago

Swift has very sane way of doing this: Strings are sequences of Characters. Characters represent extended grapheme clusters. Iteration consumes as many bytes from the string as are necessary to read the next grapheme. In most cases, this is straight-up UTF-8 decoding, which can be implemented very efficiently; you only need to worry about extended grapheme clusters when you get to the non-ASCII subset of Unicode, so the code paths that require unicode table lookups are infrequently exercised. String searching & matching return an opaque String.Index type. I assume that under the hood they do Boyer-Moore on bytes and String.Index is a byte offset, but the important thing is that String.Index values are not convertible to integers, and so you never run into the case where a user passes in a byte offset that would slice a grapheme in half. Instead, String.Index has properties for the next and previous index and a method to advance by an arbitrary amount, so you'd access the 7th character after a dash in a string as myString[myString.rangeOfString("-").advanceBy(7)] Swift gives up on the idea that len(x+y) = len(x) + len(y). This is just something you have to remember; in UI programming, however, it makes a lot of sense because adding an accent to an existing string isn't going to make it take up more width in the text box. (x + y).utf8.count == x.utf8.count + y.utf8.count, however. https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swift_Programming_Language/StringsAndCharacters.html https://developer.apple.com/library/ios/documentation/Swift/... If I were to adapt this to a domain like Python's, where string processing is pretty important, I'd allow indexing & slicing by integers (including negative integers), but I'd define them in terms of String.Index types, which iterate over extended graphemes under the hood: str[2] == str[str.start.advanceBy(2)] str[-3] == str[str.end.advanceBy(-3)] str[str.find('foo') + 3] == str[str.find('foo').advanceBy(3)] str[0:-3] == str.substring(str.start, str.end.advanceBy(-3)) del str[0:3] == str.removeRange(str.start, str.start.advanceBy(3))

Veedrac 11y ago
> Swift gives up on the idea that len(x+y) = len(x) + len(y). This is just something you have to remember; in UI programming, however, it makes a lot of sense because adding an accent to an existing string isn't going to make it take up more width in the text box. (x + y).utf8.count == x.utf8.count + y.utf8.count, however. So do fullwidth characters have a length of 2? If not, how is this useful for finding the size of text in a text box anyway?

lmm 11y ago

Python is an abstraction a long way above the Von Neumann machine. It should not expose implementation details like the number of bytes a string happens to be stored as. The whole point of the Python 2 -> 3 transition was to disallow just treating a string as a random pile of bytes and hoping it all works out.

catnaroek 11y ago

It's possible to do better than existing mainstream solutions: (0) Dependent types: The type of valid indices into a string depends, well, on the string. It's this type, not `uint`, that should be used by functions like `str.find()` and `re.match()`. This makes it a compile error to, say, slice a string with an index to another string. Internally, or course, all indices are represented as `uint`s, but your program has no business knowing this. It's an abstract type. IMO, this solution is particularly useful for systems languages, where the increased flexibility and fine-grained control are worth the increase in complexity. (1) Coalgebras: `str.find()` shouldn't return an index. Instead, it should return in O(1) time, and allocating O(1) additional space, the left and right slices of the original string, split at the desired index. (If the string can be split in multiple places, `str.find()` should return an iterator whose element type is such left-right pairs.) For this to work, it's fundamental that the left and right slices not be allocated in their own buffers, but instead share the original string's buffer. IMO, this solution is best for high-level languages, where simplifying the API is well worth a little loss in flexibility. Neither of these APIs allows the programmer to split strings at the wrong places.

raiph 11y ago

> I'm getting tired of this argument Perhaps this is because Unicode is an exhaustive standard. :) Seriously, I hear ya. > every argument had the same hole that I see in your post: What, exactly, is a "character"? Please answer me that I'd say the answer varies depending on who's talking, what they're talking about, and who's listening. If it's a programmer interested in listening to the Unicode consortium and interested in what Unicode.org documents define for "what a user thinks of as a character" then the answer is, according to those documents, a "grapheme". > then you can write a proposal for how the API should work, exactly Right. It can get, and has indeed gotten, better than that; one can write specifications, build reference implementations, and try things out in battle for a few years. Several text processing systems (including some programming languages) have gone this route and their results should be taken in to account. (ICU is perhaps the go to reference implementation.) > and why this works with Indic and German and Korean and everything else. For designs and implementations of text processing systems that claim an aspiration of progressing toward fully following the Unicode specification, the simplest answer to "why does it work?" (or "why it is expected to work") with Indic and German and Korean and everything else is of course the Unicode standard itself. For those of us who aren't blessed with the patience of a saint and thus haven't read the entire spec from start to finish a few times, one has to rely to a degree on those who do, even if it's really hard to follow what the heck they're saying. > You're trying to prevent people from slicing up grapheme clusters, which is a noble goal. Given that grapheme clusters correspond to what the Unicode standard specifies for identifying and preserving the integrity of "what a user thinks of as a character", it seems less a noble goal than a fundamental eventual goal for any text processing system that explicitly claims to aspire to support the world's human languages via Unicode compliance. > we don't end up slicing grapheme clusters very often in practice. Which "we" are you referring to? The western world has dominated text processing to date does. This must not mean that "we", by which I mean all humans, would best continue practices that marginalize those with native languages not well served. (By ignoring the central importance of "what a user thinks of as a character"; again, I'm quoting Unicode consortium documentation.) Or, more parochially, what about the "we" that's just programmers who want to be able to increase their confidence that they are properly processing Unicode compliant text? > In the meantime, formats like JSON and XML are defined in terms of code points ... Code points are the middle level of Unicode, sandwiched between byte encodings at the bottom level and graphemes at the top. There's nothing wrong with a text processing system stopping at the middle level provided folk don't lose sight of the fact that it stops at the middle level. Thinking that code points are "what a user thinks of as a character" mostly works out OK when processing text that's English or the like, but is definitively not Unicode compliant to the degree it's used too broadly. > Can you tell me what the correct behavior is? I would suggest that the most appropriate authority is the Unicode consortium and its Unicode standard. > Might you end up with bugs in programs because len(x + y) != len(x) + len(y)? Oh for sure. But it's part of the Unicode standard. It's one of several apparently strange discontinuities that the Unicode effort introduced last century in an attempt to deal with the combination of human language reality and computer system realities. They may have chosen the wrong sweetspot. But now Unicode is a significant part of reality. I, for one, don't think it's wise or even realistic to abandon the Unicode standard as it is...