3 ms·

> Although in Obj-C's defence, NSString has always hidden its internal representation, which did a lot of work to protect you from this. Technically the docs n

by ken 8y ago

> Although in Obj-C's defence, NSString has always hidden its internal representation, which did a lot of work to protect you from this. Technically the docs never said what the internal representation was, but it was not very well hidden at all. As the article points out, NSString presents an API with "constant-time access to UTF-16 code units".

josephg 8y ago
The foot gun happens when you assume that the string representation's length is equal to the number of unicode characters. This is true in C if all your test data uses ASCII characters. Its true in Java / C# / etc if you only test using ASCII + any characters in the lower unicode planes. And thats most but not all of the characters used in the world today. In Javascript, all the standard APIs make the wrong assumptions about unicode. Unless you think about it and actually test using exotic characters, its very common to make mistakes where certain characters would crash your software. (I'm talking about you, iMessage). I ran into a bug just the other day where emoji in a comment in a rust file with RLS caused a weird rendering error in VSC. In javascript String#length and naive array indexing is basically always wrong, and as far as I know there's no official API that does the right thing. But because its mostly right, people use String#length and don't think about it. $ node > "𝄞".length 2 > "𝄞"[0] '�' // oops! In Objective-C, even though large NSStrings use a 2 byte encoding under the hood, the API that NSString provides encourages you to do the right thing by default a lot more than javascript does. We could talk about whether language or the documentation is responsible, but either way the difference translates into fewer bugs.
- dwaite 8y ago
  Unicode is complex enough it is often best just to not think of it in terms of a character count unless you are doing text layout (in which case you care about grapheme clusters, not characters).
  - josephg 8y ago
    I take your point, but it really depends on the application. I care about this having worked on realtime collaborative editors. In this space it definitely does make sense to describe string offsets with unicode codepoints. The problem with counting the number of grapheme clusters is that that number can change with every revision of the unicode spec, and you can get different answers on different platforms. Or the same platform, but a different version of the OS. Does a zero width space offset the position count or not? Do you get the same answer on Windows, MacOS and through whatever unix you happen to use AWS? I shudder just thinking about it. In contrast, counting unicode codepoints is simple, consistent and well defined. Counting string lengths that way is also implemented in the standard library of just about all languages.
- jcelerier 8y ago
  > '�' // oops! why oops ? what do you expect ? do you also '[0]' random image data and expect it to be the first pixel's red component ?
  - nemetroid 8y ago
    In Python and Haskell, indexing a Unicode string yields a Unicode codepoint. In Rust, there is no index operator for Strings.
pjscott 8y ago
So... that's either UTF-16, an eight-bit encoding like ASCII or Latin-1, a variable-width encoding with an offset-translation table (as mentioned in the article), or a string with a different encoding but a constant upper bound on its size. (Or a few other things that seem less practical.) The first two are used by CFStringRef (bridged to NSString), the third by Swift strings that have been hit with UTF-16 offset lookups, and the fourth by Objective C tagged pointer strings (on 64-bit devices). Yeah, I guess that NSString has mostly exhausted the possibilities for ways to fulfill their API obligations.

by ken 8y ago

josephg 8y ago

The foot gun happens when you assume that the string representation's length is equal to the number of unicode characters. This is true in C if all your test data uses ASCII characters. Its true in Java / C# / etc if you only test using ASCII + any characters in the lower unicode planes. And thats most but not all of the characters used in the world today. In Javascript, all the standard APIs make the wrong assumptions about unicode. Unless you think about it and actually test using exotic characters, its very common to make mistakes where certain characters would crash your software. (I'm talking about you, iMessage). I ran into a bug just the other day where emoji in a comment in a rust file with RLS caused a weird rendering error in VSC. In javascript String#length and naive array indexing is basically always wrong, and as far as I know there's no official API that does the right thing. But because its mostly right, people use String#length and don't think about it. $ node > "𝄞".length 2 > "𝄞"[0] '�' // oops! In Objective-C, even though large NSStrings use a 2 byte encoding under the hood, the API that NSString provides encourages you to do the right thing by default a lot more than javascript does. We could talk about whether language or the documentation is responsible, but either way the difference translates into fewer bugs.

dwaite 8y ago
Unicode is complex enough it is often best just to not think of it in terms of a character count unless you are doing text layout (in which case you care about grapheme clusters, not characters).
- josephg 8y ago
  I take your point, but it really depends on the application. I care about this having worked on realtime collaborative editors. In this space it definitely does make sense to describe string offsets with unicode codepoints. The problem with counting the number of grapheme clusters is that that number can change with every revision of the unicode spec, and you can get different answers on different platforms. Or the same platform, but a different version of the OS. Does a zero width space offset the position count or not? Do you get the same answer on Windows, MacOS and through whatever unix you happen to use AWS? I shudder just thinking about it. In contrast, counting unicode codepoints is simple, consistent and well defined. Counting string lengths that way is also implemented in the standard library of just about all languages.
jcelerier 8y ago
> '�' // oops! why oops ? what do you expect ? do you also '[0]' random image data and expect it to be the first pixel's red component ?
- nemetroid 8y ago
  In Python and Haskell, indexing a Unicode string yields a Unicode codepoint. In Rust, there is no index operator for Strings.

pjscott 8y ago

So... that's either UTF-16, an eight-bit encoding like ASCII or Latin-1, a variable-width encoding with an offset-translation table (as mentioned in the article), or a string with a different encoding but a constant upper bound on its size. (Or a few other things that seem less practical.) The first two are used by CFStringRef (bridged to NSString), the third by Swift strings that have been hit with UTF-16 offset lookups, and the fourth by Objective C tagged pointer strings (on 64-bit devices). Yeah, I guess that NSString has mostly exhausted the possibilities for ways to fulfill their API obligations.