4 ms·

I think it’s good that Rust sort of forces you to recognize the complexity in this. If you try s.len(), it’ll be in terms of UTF-8 bytes, which might be what yo

I think it’s good that Rust sort of forces you to recognize the complexity in this. If you try s.len(), it’ll be in terms of UTF-8 bytes, which might be what you want… or might be far from it. If you switch to s.chars().count(), you’ll get Unicode scalar values, which may be closer to the mark. And if you need proper segmentation by “user-perceived character,” as set out in Unicode Annex 29, you’ll have to bring in an external crate. That’s fair enough. Keep in mind that even Annex 29 is, on some level, just a starting point. Its segmentation rules don’t work “out of the box” for me with Persian and Arabic text. I’m not totally on-board with its treatment of the zero-width non-joiner (U+200C), and it doesn’t deal with manual tatweel/kashida (U+0640) at all. So you make the necessary adjustments for your use case. The rabbit hole is deep.

masklinn 7y ago
> I think it’s good that Rust sort of forces you to recognize the complexity in this. It's not bad, but Swift still does better: * String.Index is an opaque type (ideally it would be linked to a specific string too) * it returns grapheme cluster counts by default, that's probably the least surprising / useless though not necessarily super useful either * other lengths go through explicit lazy views e.g. string.utf8.count will provide the UTF8 code units count, string.unicodeScalars.count will provide the USVs count > If you switch to s.chars().count(), you’ll get UTF-8 scalar values You get unicode scalar values (which I assume is what you meant).
- theobeers 7y ago
  Good catch! I was still able to edit the comment.
- deleted 7y ago
  [deleted]
- gok 7y ago
  The big problem with extended grapheme clusters by default is that it changes behavior completely as Unicode spec changes. So Swift strings are nearly guaranteed to change their default behavior between versions.
- Manishearth 7y ago
  I think what swift does is good for its design space. I definitely would not like it if Rust did the same thing, because you need fast ways to store and reuse indices for parsing and other things. Swift is primarily geared towards UI, and in that case using EGCs is the right design choice.
- cryptonector 7y ago
  TFA makes a very good case that Swift got it wrong. Perhaps string length should be more hidden though, or parametrized with a unit to count. Perhaps it should be let nu = s._smelly_length(UTF8CodeUnits); let nc = s._smelly_length(ExtendedGraphemeClusters); Stop using "zero length" to denote empty string, just have an empty string method.
  - favorited 7y ago
    Not sure how you read "Swift’s approach to string length isn’t unambiguously the best one" and interpret it as "Swift got it wrong." The author's substantive criticism of Swift's String/Character/etc. types seems to be complications arising from the dependency on ICU (such as not necessarily being able to persist indices across icu4c versions).
atoav 7y ago
As a python user that came to Rust encodings never really made much sense to me. I knew what encodings basically were, but I neither knew why it has to be that way and how to deal with common situations without feeling insecure. Rust put the whole thing front and center and made it incredibly clear why they made that choice. The design choices around encodings also have been more well reflected than those in Python, it never becomes a “hairy” mess, but something you can rationally manage.
tapirl 7y ago
There is not the char concept in Go, only rune, which means a code point. This reduces much confusions.
- masklinn 7y ago
  > There is not the char concept in Go, only rune, which means a code point. Go named their char rune. That's a distinction without a difference.
  - hsivonen 7y ago
    Rune in Go means a scalar value i.e. the same as Rust char but not the same as C char or Java char.
    - masklinn 7y ago
      > Rune in Go means a scalar value i.e. the same as Rust char Yes. It's also the original intended meaning of java char, just didn't survive the transition to 21 bit USVs. So distinction without a difference.
- arcticbull 7y ago
  It's probably a false sense of security though. If runes are in fact code points, thats about the least helpful abstraction. They do not encode grapheme clusters meaning some items, such as emoji, may be represented by multiple runes, and if you split them along the boundary, you may end up with invalid strings or two half-characters in each substring. You can't really do much with code points -- they're useful when parsing things or when implementing unicode algorithms, and not much else.
  - HelloNurse 7y ago
    If you care about grapheme clusters, it doesn't mean using characters is wrong: you can layer a higher level API on top of characters. For example, if you want to encode emojis some non-Unicode abstract description goes in and a sequence of characters go out, and if you want to figure out the boundaries of letters clustered with the respective combining diacritical marks they are subsequences in a sequence of characters.
saghm 7y ago
I agree here! If anything, in my Rust code, I've learned that I very rarely actually need to index a string; `String::contains`, `String::starts_with`, and `String::ends_with` cover the vast majority of my use cases, and if for some reason I actually need to inspect the characters, `String::chars` will let me iterate over them.

masklinn 7y ago

> I think it’s good that Rust sort of forces you to recognize the complexity in this. It's not bad, but Swift still does better: * String.Index is an opaque type (ideally it would be linked to a specific string too) * it returns grapheme cluster counts by default, that's probably the least surprising / useless though not necessarily super useful either * other lengths go through explicit lazy views e.g. string.utf8.count will provide the UTF8 code units count, string.unicodeScalars.count will provide the USVs count > If you switch to s.chars().count(), you’ll get UTF-8 scalar values You get unicode scalar values (which I assume is what you meant).

theobeers 7y ago
Good catch! I was still able to edit the comment.
deleted 7y ago
[deleted]
gok 7y ago
The big problem with extended grapheme clusters by default is that it changes behavior completely as Unicode spec changes. So Swift strings are nearly guaranteed to change their default behavior between versions.
Manishearth 7y ago
I think what swift does is good for its design space. I definitely would not like it if Rust did the same thing, because you need fast ways to store and reuse indices for parsing and other things. Swift is primarily geared towards UI, and in that case using EGCs is the right design choice.
cryptonector 7y ago
TFA makes a very good case that Swift got it wrong. Perhaps string length should be more hidden though, or parametrized with a unit to count. Perhaps it should be let nu = s._smelly_length(UTF8CodeUnits); let nc = s._smelly_length(ExtendedGraphemeClusters); Stop using "zero length" to denote empty string, just have an empty string method.
- favorited 7y ago
  Not sure how you read "Swift’s approach to string length isn’t unambiguously the best one" and interpret it as "Swift got it wrong." The author's substantive criticism of Swift's String/Character/etc. types seems to be complications arising from the dependency on ICU (such as not necessarily being able to persist indices across icu4c versions).

atoav 7y ago

As a python user that came to Rust encodings never really made much sense to me. I knew what encodings basically were, but I neither knew why it has to be that way and how to deal with common situations without feeling insecure. Rust put the whole thing front and center and made it incredibly clear why they made that choice. The design choices around encodings also have been more well reflected than those in Python, it never becomes a “hairy” mess, but something you can rationally manage.

tapirl 7y ago

There is not the char concept in Go, only rune, which means a code point. This reduces much confusions.

masklinn 7y ago
> There is not the char concept in Go, only rune, which means a code point. Go named their char rune. That's a distinction without a difference.
- hsivonen 7y ago
  Rune in Go means a scalar value i.e. the same as Rust char but not the same as C char or Java char.
  - masklinn 7y ago
    > Rune in Go means a scalar value i.e. the same as Rust char Yes. It's also the original intended meaning of java char, just didn't survive the transition to 21 bit USVs. So distinction without a difference.
arcticbull 7y ago
It's probably a false sense of security though. If runes are in fact code points, thats about the least helpful abstraction. They do not encode grapheme clusters meaning some items, such as emoji, may be represented by multiple runes, and if you split them along the boundary, you may end up with invalid strings or two half-characters in each substring. You can't really do much with code points -- they're useful when parsing things or when implementing unicode algorithms, and not much else.
- HelloNurse 7y ago
  If you care about grapheme clusters, it doesn't mean using characters is wrong: you can layer a higher level API on top of characters. For example, if you want to encode emojis some non-Unicode abstract description goes in and a sequence of characters go out, and if you want to figure out the boundaries of letters clustered with the respective combining diacritical marks they are subsequences in a sequence of characters.

saghm 7y ago

I agree here! If anything, in my Rust code, I've learned that I very rarely actually need to index a string; `String::contains`, `String::starts_with`, and `String::ends_with` cover the vast majority of my use cases, and if for some reason I actually need to inspect the characters, `String::chars` will let me iterate over them.