51 ms·

> Java treats text as all modern systems should: as a sequence of the open-ended Universal Character Set (UCS) unfortunately, Java treats text as data encoded

by pilif 11y ago

> Java treats text as all modern systems should: as a sequence of the open-ended Universal Character Set (UCS) unfortunately, Java treats text as data encoded in UCS-2 which uses a maximum of two bytes to represent a character and all the Java string APIs assume the UCS-2 encoding. The problem with UCS-2 is that it doesn't allow to encode characters outside of the Unicode BMP. In order to solve this, Java actually uses UTF-16 encoding to encode characters outside of the BMP, but none of the APIs actually know about this and still assume UCS-2. This leads to the API being wrong about character lengths and to regular expressions wrongly matching certain strings and sometimes destroying data when you apply them to replace substrings. Case in point is this little test class here: https://gist.github.com/5745601abbfbf7068fcd https://gist.github.com/5745601abbfbf7068fcd which prints 2 on the console. I've also given a talk about this at a swiss JS conference in 2012: http://pilif.me/unicode.pdf http://pilif.me/unicode.pdf - JS has/had the same issues as Java in that regard. The only well-known contemporary languages that get this (somewhat) right are Perl (since forever), Python 3, Ruby 1.9 and Swift, though ES6 is in process of getting up to speed too. Python 3 and Swift have the issue of being totally tied to unicode, so if the politics and issues around Han Unification matter to you then those two are actually also not usable for you.

Hinrik 11y ago
> The only well-known contemporary languages that get this (somewhat) right are Python 3, Ruby 1.9 and Swift, though ES6 is in process of getting up to speed too. And Perl.
- pilif 11y ago
  You are right of course (I knew this as you can see when you look at the PDF I linked - slide 50). I have edited my original post.
aardvark179 11y ago
> unfortunately, Java treats text as data encoded in UCS-2 which uses a maximum of two bytes to represent a character and all the Java string APIs assume the UCS-2 encoding. Not entirely true. There are codePoint*, indexOf(int ch), and various other methods on String which do work correctly in code points. Unfortunately old methods can't be changed without affecting compatibility.
stickfigure 11y ago
> The only well-known contemporary languages that get this (somewhat) right are... and Ceylon.
Someone 11y ago
"that get this (somewhat) right [...] and Swift" I don't know how Perl, Ruby and Python handle this, but with Unicode, I don't think we can get better than 'somewhat right'. For example, Swift removes some invariants on string operations that many people take for granted, such as: - appending to a string of length n doesn't change its first n characters. - appending a string of length m to one of length n gives you a string of length (m+n) - strings that are equal to each other have the same byte representation (=> whenever you use a set of strings, you have to choose whether you want the ability to get the same bytes out or not) I also am not sure whether s==t and u==v implies s+u==t+v.
- raiph 11y ago
  The example you've provided is confusion/complexity related to what a "character" is. If a language supports a string type and related functions that equate "character" with a grapheme they'll deliver your first two invariants. With an appropriate approach to byte representations your third invariant can also be maintained.
  - Someone 11y ago
    If you look at Swift, it uses what Apple calls "extended grapheme cluster" as its "Character" type. That means: "e" + "`" = "è" All three strings have 1 Swift Character in them; the first is not a prefix of the last. And yes, you can normalize byte representations, but then you have to give up round-tripping through strings. I maintain convinced that there is no way to do Unicode strings without drawbacks.
    - raiph 11y ago
      I haven't tried Swift yet but my latest observations and guessing about your grave accent example are as follows. Based on direct cut/paste and inspection, the grave accent (in the "`" string above and here in these parens too) is codepoint 96. This is a non-combining (standalone) grave accent that is meant to be its own grapheme. Appending it to another grapheme such as "e" should not combine. The result should be an "e" followed by a visually separate grave character. The string should contain two separate graphemes and have a length of 2. I can't tell if the confusion/complexity in this case is due to some cut/paste/storage/browser bug, or Swift's handling of the special case (called "degenerate" in the Unicode standard) of an isolated combining character that hasn't yet been combined, or something else. So I read some of the Swift docs to see if that revealed anything. The Swift docs I read used the example of appending a combining grave accent (ie codepoint 768, which is NOT the same as the codepoint 96 stored in "`", but rather something more akin to "̀"), stored as a Character value, to an "e" string, stored as a String value. After appending a Character that is a combining character, the length of the new String is the same as the old one. And that's appropriate. I presume that the Swiftian character count concept, which is applicable to Swift Strings, and which is returned by .count, is inapplicable to Swift's Character values. Presumably, when a Swift Character value needs to be displayed, Swift generates a String that contains it and displays that. Further, perhaps, if the Character value is a combining character, Swift automatically prepends an appropriate base character (a space, value 32, or NBSP, or dotted circle, or some such) to ensure it becomes its own grapheme. Part of the confusion may stem from the decision in Swift to use "..." quotemarks for literals of both types. It seems you have to explicitly tell Swift that a "..." literal is a Character, not a String, if you want a Character. At first blush I like the sound of that. Alternatively, they've screwed something up.
SiVal 11y ago
Python 3 and Swift have the issue of being totally tied to unicode, so if the politics and issues around Han Unification matter to you then those two are actually also not usable for you. Those issues do matter to me, yet Unicode-based languages like Python 3 and Swift are not only usable, they are best general approach (among actually existing technologies, not among all theoretically possible approaches). I've been building Asian (CJK) infosystems since the 1980s, and there are far fewer problems with Unicode-based approaches than with the encoding soup we had before.

by pilif 11y ago

Hinrik 11y ago

> The only well-known contemporary languages that get this (somewhat) right are Python 3, Ruby 1.9 and Swift, though ES6 is in process of getting up to speed too. And Perl.

pilif 11y ago
You are right of course (I knew this as you can see when you look at the PDF I linked - slide 50). I have edited my original post.

aardvark179 11y ago

> unfortunately, Java treats text as data encoded in UCS-2 which uses a maximum of two bytes to represent a character and all the Java string APIs assume the UCS-2 encoding. Not entirely true. There are codePoint*, indexOf(int ch), and various other methods on String which do work correctly in code points. Unfortunately old methods can't be changed without affecting compatibility.

stickfigure 11y ago

> The only well-known contemporary languages that get this (somewhat) right are... and Ceylon.

Someone 11y ago

"that get this (somewhat) right [...] and Swift" I don't know how Perl, Ruby and Python handle this, but with Unicode, I don't think we can get better than 'somewhat right'. For example, Swift removes some invariants on string operations that many people take for granted, such as: - appending to a string of length n doesn't change its first n characters. - appending a string of length m to one of length n gives you a string of length (m+n) - strings that are equal to each other have the same byte representation (=> whenever you use a set of strings, you have to choose whether you want the ability to get the same bytes out or not) I also am not sure whether s==t and u==v implies s+u==t+v.

raiph 11y ago
The example you've provided is confusion/complexity related to what a "character" is. If a language supports a string type and related functions that equate "character" with a grapheme they'll deliver your first two invariants. With an appropriate approach to byte representations your third invariant can also be maintained.
- Someone 11y ago
  If you look at Swift, it uses what Apple calls "extended grapheme cluster" as its "Character" type. That means: "e" + "`" = "è" All three strings have 1 Swift Character in them; the first is not a prefix of the last. And yes, you can normalize byte representations, but then you have to give up round-tripping through strings. I maintain convinced that there is no way to do Unicode strings without drawbacks.
  - raiph 11y ago
    I haven't tried Swift yet but my latest observations and guessing about your grave accent example are as follows. Based on direct cut/paste and inspection, the grave accent (in the "`" string above and here in these parens too) is codepoint 96. This is a non-combining (standalone) grave accent that is meant to be its own grapheme. Appending it to another grapheme such as "e" should not combine. The result should be an "e" followed by a visually separate grave character. The string should contain two separate graphemes and have a length of 2. I can't tell if the confusion/complexity in this case is due to some cut/paste/storage/browser bug, or Swift's handling of the special case (called "degenerate" in the Unicode standard) of an isolated combining character that hasn't yet been combined, or something else. So I read some of the Swift docs to see if that revealed anything. The Swift docs I read used the example of appending a combining grave accent (ie codepoint 768, which is NOT the same as the codepoint 96 stored in "`", but rather something more akin to "̀"), stored as a Character value, to an "e" string, stored as a String value. After appending a Character that is a combining character, the length of the new String is the same as the old one. And that's appropriate. I presume that the Swiftian character count concept, which is applicable to Swift Strings, and which is returned by .count, is inapplicable to Swift's Character values. Presumably, when a Swift Character value needs to be displayed, Swift generates a String that contains it and displays that. Further, perhaps, if the Character value is a combining character, Swift automatically prepends an appropriate base character (a space, value 32, or NBSP, or dotted circle, or some such) to ensure it becomes its own grapheme. Part of the confusion may stem from the decision in Swift to use "..." quotemarks for literals of both types. It seems you have to explicitly tell Swift that a "..." literal is a Character, not a String, if you want a Character. At first blush I like the sound of that. Alternatively, they've screwed something up.

SiVal 11y ago

Python 3 and Swift have the issue of being totally tied to unicode, so if the politics and issues around Han Unification matter to you then those two are actually also not usable for you. Those issues do matter to me, yet Unicode-based languages like Python 3 and Swift are not only usable, they are best general approach (among actually existing technologies, not among all theoretically possible approaches). I've been building Asian (CJK) infosystems since the 1980s, and there are far fewer problems with Unicode-based approaches than with the encoding soup we had before.