4 ms·

A nitpick from the article >This spells trouble for languages using UTF-16 encodings (Java, C#, JavaScript). if they were using UTF-16, this wouldn't be a pro

by pilif 13y ago

A nitpick from the article >This spells trouble for languages using UTF-16 encodings (Java, C#, JavaScript). if they were using UTF-16, this wouldn't be a problem as UTF-16 can be used to perfectly well encode code points outside of the BMP (at the cost of losing ability for O(1) access to specific code points of course. If you need to know what the n-th code point is, you have to scan the string until the n-th position). They are, however, using UCS-2 which can't. If you use a library that knows about UCS-2 to work on strings encoded in UTF-16, then you will get broken characters, your counts will be off and case transformations might fail. Most languages that claim Unicode support still only have UCS-2 libraries (Python 3 is a notable exception)

Renaud 13y ago
Exactly, UTF-16 perfectly defines surrogate pairs for code points that do not fit into the 16 bit plane. A perfect implementation of UTF-16 should have no problem, unfortunately, most are broken when it comes to surrogate pairs. Many languages pre-date the introduction of UTF-16 and implemented 16 bit string encoding as UCS-2, and still do. Then there are oddities like VBA using UTF-16 internally, but converting all strings going through the Win32 API as 8-bit (relying on the current code page for character translation!)...
jeltz 13y ago
> Most languages that claim Unicode support still only have UCS-2 libraries (Python 3 is a notable exception) Most non-JVM languages[1] actually use UTF-8 as the internal encoding so they should not suffer from this. Python 3 does not use UTF-16 either, it selects an encoding based on the contents of the string. http://www.python.org/dev/peps/pep-0393/ http://www.python.org/dev/peps/pep-0393/ 1. I think .NET too uses UCS-2 or UTF-16, but I am not a Windows developer.
- pilif 13y ago
  .NET uses UCS-2 because the Windows API uses UCS-2 (so when you use Visual Studio out of the box, you will get UCS-2). ECMAScript (JS) uses UCS-2 because that's all there was when the spec was written. Other scripting languages I know for certain are - PHP doesn't care and treats strings as arrays of bytes. All the str functions operate on these byte arrays and thus happily destroy your strings if they are encoded as anything but the old 8-bit encodings. If you need to support utf-8, you have to use different library functions (mb_*) and a special syntax in their regex support (/u modifier). - Python < 3 treats strings as byte arrays or UCS-2 depending on whether you use the byte type or the Unicode type. As such, it has all the same issues as all other UCS-2 libraries - Ruby < 1.9 treats strings a byte arrays. There is some limited UTF-8 support, but it's in additional libraries. The internal API is treating strings as byte arrays. Ruby >= 1.9 lets you chose your internal encoding. Most people use utf-8, but you don't have to. - Perl I don't know enough about, but I hear it as an UTF-8 mode that is actually well-supported by the language itself and gets almost everything right. These are the more common scripting languages. Of the compiled languages, I know for certain about Go (utf-8; good library support), C (OS dependent, but the standard string API treats strings as byte arrays), C++ (dito) and Delphi (UCS-2 since 2010, byte arrays before that) I would say that there are so many exceptions to the UTF-8 rule that I wouldn't say "most" languages are using UTF-8.
  - masklinn 13y ago
    > - Python < 3 treats strings as byte arrays or UCS-2 depending on whether you use the byte type or the Unicode type. As such, it has all the same issues as all other UCS-2 libraries It's Python < 3.3 (the Flexible String Represrntation was introduced in 3.3), there's a byte array type (str in P2, bytes in P3) and a string type (unicode/str), which may be UCS2 ("narrow" builds, the default) or UCS4 ("wide" builds, set by many linux distros)
  - deleted 13y ago
    [deleted]
- berdario 13y ago
  Python <3.3 uses UCS2 or UCS4, depending on the build Ruby >1.8 lets you choose the encoding .NET UCS2/UTF-16 (I know the difference, imho if the stdlib has a .size, .length or .count that works on code units instead of code points it's broken... thus I'll mention only UCS2 from now on) Java UCS2 Clojure UCS2 Scala UCS2 QT UCS2 Haskell String UCS4 Haskell Data.Text UTF-16 (yes, not a naive UCS-2) Rust UCS4 (last time I checked) Javascript UCS2 Dart UCS2 PHP Unicode-oblivious Vala UCS4 Go UTF-8 (but it lets you call len() on strings, and it doesn't return the length of the string, but its size in bytes) I can't really think of another language that uses UTF-8 internally, are you sure?
  - sedachv 13y ago
    Common Lisp comes with two character types, base-char and character, the former being allowed to be a subset of the latter. Clozure Common Lisp uses UTF-32 for all characters and strings internally. SBCL uses base-char and simple-base-string types for ASCII and character and (simple-array character) types for UTF-32 internally. IMO having this option for two types of characters that are compatible but may have different internal representations is a really good part of the Common Lisp standard.
  - pitterpatter 13y ago
    > Rust UCS4 (last time I checked) Rust chars are 32bit Unicode codepoints. But strings themselves are utf-8. That is the string type, ~str, is basically just ~[u8], a vector of bytes and not ~[char]. `.len()` [O(1)] gives you byte length while `.char_len()` [O(n)] gives you the number of codepoints. So strings in rust are just vectors of bytes with the invariant that it's valid utf-8.
    - berdario 13y ago
      Thanks, I didn't know that
  - jeltz 13y ago
    Perl, Rust, Go and Vala. I take back the "most" part though. It seems like there are many popular solutions. Erlang uses Unicode code points and also binaries.
kvb 13y ago
.NET uses 16-bit characters, but you can use the System.Globalization.StringInfo class to iterate through a string one Unicode character at a time, index into strings by Unicode character, etc. The API's a bit awkward, but it works.
- panzi 13y ago
  One unicode character at a time or one unicode codepoint at a time? (see character composition)
  - kvb 13y ago
    Good point - I meant one codepoint at a time.
    - kvb 13y ago
      But I was wrong and it's actually by grapheme, as danbruc correctly notes.
  - masklinn 13y ago
    Codepoint. Java 5 also added new string APIs for this. IIRC, Cocoa is one of the very few frameworks/languages/whatever which provides APIs for manipulating and iterating on grapheme clusters out of the box. And provides a page explaining some of the unicode concepts and how they map to NSString: https://developer.apple.com/library/mac/documentation/Cocoa/Conceptual/Strings/Articles/stringsClusters.html#//apple_ref/doc/uid/TP40008025 https://developer.apple.com/library/mac/documentation/Cocoa/...
    - kvb 13y ago
      This is incorrect - see the definition of a text element in the remarks of http://msdn.microsoft.com/en-us/library/system.globalization.stringinfo(v=vs.110).aspx http://msdn.microsoft.com/en-us/library/system.globalization....
      - 13y ago

by pilif 13y ago

Renaud 13y ago

Exactly, UTF-16 perfectly defines surrogate pairs for code points that do not fit into the 16 bit plane. A perfect implementation of UTF-16 should have no problem, unfortunately, most are broken when it comes to surrogate pairs. Many languages pre-date the introduction of UTF-16 and implemented 16 bit string encoding as UCS-2, and still do. Then there are oddities like VBA using UTF-16 internally, but converting all strings going through the Win32 API as 8-bit (relying on the current code page for character translation!)...

jeltz 13y ago

> Most languages that claim Unicode support still only have UCS-2 libraries (Python 3 is a notable exception) Most non-JVM languages[1] actually use UTF-8 as the internal encoding so they should not suffer from this. Python 3 does not use UTF-16 either, it selects an encoding based on the contents of the string. http://www.python.org/dev/peps/pep-0393/ http://www.python.org/dev/peps/pep-0393/ 1. I think .NET too uses UCS-2 or UTF-16, but I am not a Windows developer.

pilif 13y ago
.NET uses UCS-2 because the Windows API uses UCS-2 (so when you use Visual Studio out of the box, you will get UCS-2). ECMAScript (JS) uses UCS-2 because that's all there was when the spec was written. Other scripting languages I know for certain are - PHP doesn't care and treats strings as arrays of bytes. All the str functions operate on these byte arrays and thus happily destroy your strings if they are encoded as anything but the old 8-bit encodings. If you need to support utf-8, you have to use different library functions (mb_*) and a special syntax in their regex support (/u modifier). - Python < 3 treats strings as byte arrays or UCS-2 depending on whether you use the byte type or the Unicode type. As such, it has all the same issues as all other UCS-2 libraries - Ruby < 1.9 treats strings a byte arrays. There is some limited UTF-8 support, but it's in additional libraries. The internal API is treating strings as byte arrays. Ruby >= 1.9 lets you chose your internal encoding. Most people use utf-8, but you don't have to. - Perl I don't know enough about, but I hear it as an UTF-8 mode that is actually well-supported by the language itself and gets almost everything right. These are the more common scripting languages. Of the compiled languages, I know for certain about Go (utf-8; good library support), C (OS dependent, but the standard string API treats strings as byte arrays), C++ (dito) and Delphi (UCS-2 since 2010, byte arrays before that) I would say that there are so many exceptions to the UTF-8 rule that I wouldn't say "most" languages are using UTF-8.
- masklinn 13y ago
  > - Python < 3 treats strings as byte arrays or UCS-2 depending on whether you use the byte type or the Unicode type. As such, it has all the same issues as all other UCS-2 libraries It's Python < 3.3 (the Flexible String Represrntation was introduced in 3.3), there's a byte array type (str in P2, bytes in P3) and a string type (unicode/str), which may be UCS2 ("narrow" builds, the default) or UCS4 ("wide" builds, set by many linux distros)
- deleted 13y ago
  [deleted]
berdario 13y ago
Python <3.3 uses UCS2 or UCS4, depending on the build Ruby >1.8 lets you choose the encoding .NET UCS2/UTF-16 (I know the difference, imho if the stdlib has a .size, .length or .count that works on code units instead of code points it's broken... thus I'll mention only UCS2 from now on) Java UCS2 Clojure UCS2 Scala UCS2 QT UCS2 Haskell String UCS4 Haskell Data.Text UTF-16 (yes, not a naive UCS-2) Rust UCS4 (last time I checked) Javascript UCS2 Dart UCS2 PHP Unicode-oblivious Vala UCS4 Go UTF-8 (but it lets you call len() on strings, and it doesn't return the length of the string, but its size in bytes) I can't really think of another language that uses UTF-8 internally, are you sure?
- sedachv 13y ago
  Common Lisp comes with two character types, base-char and character, the former being allowed to be a subset of the latter. Clozure Common Lisp uses UTF-32 for all characters and strings internally. SBCL uses base-char and simple-base-string types for ASCII and character and (simple-array character) types for UTF-32 internally. IMO having this option for two types of characters that are compatible but may have different internal representations is a really good part of the Common Lisp standard.
- pitterpatter 13y ago
  > Rust UCS4 (last time I checked) Rust chars are 32bit Unicode codepoints. But strings themselves are utf-8. That is the string type, ~str, is basically just ~[u8], a vector of bytes and not ~[char]. `.len()` [O(1)] gives you byte length while `.char_len()` [O(n)] gives you the number of codepoints. So strings in rust are just vectors of bytes with the invariant that it's valid utf-8.
  - berdario 13y ago
    Thanks, I didn't know that
- jeltz 13y ago
  Perl, Rust, Go and Vala. I take back the "most" part though. It seems like there are many popular solutions. Erlang uses Unicode code points and also binaries.

kvb 13y ago

.NET uses 16-bit characters, but you can use the System.Globalization.StringInfo class to iterate through a string one Unicode character at a time, index into strings by Unicode character, etc. The API's a bit awkward, but it works.

panzi 13y ago
One unicode character at a time or one unicode codepoint at a time? (see character composition)
- kvb 13y ago
  Good point - I meant one codepoint at a time.
  - kvb 13y ago
    But I was wrong and it's actually by grapheme, as danbruc correctly notes.
- masklinn 13y ago
  Codepoint. Java 5 also added new string APIs for this. IIRC, Cocoa is one of the very few frameworks/languages/whatever which provides APIs for manipulating and iterating on grapheme clusters out of the box. And provides a page explaining some of the unicode concepts and how they map to NSString: https://developer.apple.com/library/mac/documentation/Cocoa/Conceptual/Strings/Articles/stringsClusters.html#//apple_ref/doc/uid/TP40008025 https://developer.apple.com/library/mac/documentation/Cocoa/...
  - kvb 13y ago
    This is incorrect - see the definition of a text element in the remarks of http://msdn.microsoft.com/en-us/library/system.globalization.stringinfo(v=vs.110).aspx http://msdn.microsoft.com/en-us/library/system.globalization....
    - 13y ago