5 ms·

> Just because it's a systems language? I get why you wouldn't make it the default in Rust, but the logic applies to all languages, doesn't it? You deal with

> Just because it's a systems language? I get why you wouldn't make it the default in Rust, but the logic applies to all languages, doesn't it? You deal with strings as bytes (opaque data) (edit2: didn't really mean bytes here. see child comments) more often than strings as text in systems programming, basically. Also, you'll want to write stuff like fast parsers in Rust. For a parser you're usually doing ascii matching anyway. Swift hides the actual encoding of the string from the user. That's not something we can do in Rust. > I get why you wouldn't make it the default in Rust That's all I'm saying? I'm totally for having EGC operations there, I don't want them in the default operations. "it just wouldn’t do to have expensive grapheme segmentation operations all over your string defaults" Basically, Rust tries to keep costs explicit, unlike languages like Python. This would be against that philosophy. Edit: I'd also like to point out that indexing by code point or grapheme is a rare operation in Rust anyway.

xenadu02 10y ago
In Swift you have access to the utf8 and utf16 views. For a string constructed from UTF8 the utf8 view is a view onto the underlying storage more or less. That said the String API is getting some revisions in Swift 4 to make it both faster and easier to use without sacrificing correctness. Swift supports unicode in source code so its parser is unicode aware. I'm not sure there is a good case for parsing in a non-unicode-aware way. If you're working with strings as opaque data why are you even using a String type?
- Manishearth 10y ago
  Isn't the access to the views an iterator? > I'm not sure there is a good case for parsing in a non-unicode-aware way I'm not saying that you should parse in a non-unicode-aware way, I'm saying that most parsing code deals with ASCII delimeters, so grapheme clusters are a distraction (and will mess up parsing, since most formats do not consider EGCs. An HTML closing tag followed by an accent would totally break the parser otherwise). You rarely need to index, though. Basically, in parsing, you're looking for characters which themselves are single code points. So it doesn't matter. > If you're working with strings as opaque data Well, not entirely opaque :) But the String type is basically "here is a valid piece of UTF8 data", and you can pass that around. That's a distinct enough purpose to have a String type, even if you don't mess with its innards. IME the most used String APIs are those for interpolation, concatenation, and search, all of which don't need to be EGC aware (though search being EGC aware is sometimes a good thing). Slicing sometimes gets used in conjunction with the others.
- hsivonen 10y ago
  Is the memory representation of Swift strings now documented as something that programmers can rely on for performance characteristics? I read the Swift Book when Swift first came out and it bothered me a lot that the book didn't say what the memory representation of strings is. In contrast, Rust documentation is very clear about this stuff.
  - Manishearth 10y ago
    It's a tagged union of utf8 and utf16 IIRC. I had to look at the stdlib source when I was trying to figure this out.
scrollaway 10y ago
> You deal with strings as bytes (opaque data) more often than strings as text in systems programming, basically. Now you're really confusing me. Are you talking about simply byte strings vs. unicode strings? Most modern languages, including python, have that distinction. What am I missing?
- Manishearth 10y ago
  No, no, Rust has unicode (utf8) strings (also bytestrings if you need them). "bytes" was a mischaracterization, sorry. I'm saying that most string handling in Rust is just lugging them around and copying them and concatenating/interpolating them. On top of that, the few times you do iterate through them, you're looking for specific characters or character sequences (e.g. while parsing), and EGCs aren't so important there. They can be, but usually aren't. Basically, personally the only times I've seen .chars() being used is in parsery things (where your grammar is defined over code points, not EGCs, so using EGCs is counterproductive and wrong), in the Servo DOM (where the spec tells you to operate on chars), and code where you're just verifying that the string contains or does not contain code points of a certain type (e.g. whitespace). None of this is ascribing meaning to the concept of a code point, it's just using them as a crutch to reason about things like equality. (Of course, equality in Unicode is a whole other can of worms).
hsivonen 10y ago
In addition to the Rust design making more sense for systems programming than Swift's, what Rust puts on the language-level is forward-compatible with future Unicode versions. A future version of Unicode could define combining behavior for currently unassigned code points, which means that iterating by grapheme cluster will change as Swift becomes aware of new Unicode versions.
throwawayish 10y ago
> You deal with strings as bytes (opaque data) (edit2: didn't really mean bytes here. see child comments) more often than strings as text in systems programming, basically. That's true in some sense [1] for unix-ish OSes, but doesn't hold on other operating systems. [1] Even on unix-ish OSes it's pretty obvious that filenames are supposed to be encoded in something and aren't just uint8_t*. There's just no one telling you what that encoding is, so normally it's guessed (eg. $LANG). For filesystems that actually have a defined encoding (eg. NTFS\Win32, JFS, HFS+, APFS and the whole bunch of "non-unix OS FS") these are re-coded by the filesystem driver in the unix-ish OS. Eg. ntfs-3g transcodes filenames to UTF-8; cifs takes a "iocharset" option; and we of course have convmv(1). The idea that "file names and generally everything in the OS is a bunch of bytes" is pretty much only shared by unix-ish systems.
- Manishearth 10y ago
  Oh, that's not the sense in which I was using the term "bytes". Rust has different string types for file contents and path names, with their own (platform-dependent) restrictions.

xenadu02 10y ago

In Swift you have access to the utf8 and utf16 views. For a string constructed from UTF8 the utf8 view is a view onto the underlying storage more or less. That said the String API is getting some revisions in Swift 4 to make it both faster and easier to use without sacrificing correctness. Swift supports unicode in source code so its parser is unicode aware. I'm not sure there is a good case for parsing in a non-unicode-aware way. If you're working with strings as opaque data why are you even using a String type?

Manishearth 10y ago
Isn't the access to the views an iterator? > I'm not sure there is a good case for parsing in a non-unicode-aware way I'm not saying that you should parse in a non-unicode-aware way, I'm saying that most parsing code deals with ASCII delimeters, so grapheme clusters are a distraction (and will mess up parsing, since most formats do not consider EGCs. An HTML closing tag followed by an accent would totally break the parser otherwise). You rarely need to index, though. Basically, in parsing, you're looking for characters which themselves are single code points. So it doesn't matter. > If you're working with strings as opaque data Well, not entirely opaque :) But the String type is basically "here is a valid piece of UTF8 data", and you can pass that around. That's a distinct enough purpose to have a String type, even if you don't mess with its innards. IME the most used String APIs are those for interpolation, concatenation, and search, all of which don't need to be EGC aware (though search being EGC aware is sometimes a good thing). Slicing sometimes gets used in conjunction with the others.
hsivonen 10y ago
Is the memory representation of Swift strings now documented as something that programmers can rely on for performance characteristics? I read the Swift Book when Swift first came out and it bothered me a lot that the book didn't say what the memory representation of strings is. In contrast, Rust documentation is very clear about this stuff.
- Manishearth 10y ago
  It's a tagged union of utf8 and utf16 IIRC. I had to look at the stdlib source when I was trying to figure this out.

scrollaway 10y ago

> You deal with strings as bytes (opaque data) more often than strings as text in systems programming, basically. Now you're really confusing me. Are you talking about simply byte strings vs. unicode strings? Most modern languages, including python, have that distinction. What am I missing?

Manishearth 10y ago
No, no, Rust has unicode (utf8) strings (also bytestrings if you need them). "bytes" was a mischaracterization, sorry. I'm saying that most string handling in Rust is just lugging them around and copying them and concatenating/interpolating them. On top of that, the few times you do iterate through them, you're looking for specific characters or character sequences (e.g. while parsing), and EGCs aren't so important there. They can be, but usually aren't. Basically, personally the only times I've seen .chars() being used is in parsery things (where your grammar is defined over code points, not EGCs, so using EGCs is counterproductive and wrong), in the Servo DOM (where the spec tells you to operate on chars), and code where you're just verifying that the string contains or does not contain code points of a certain type (e.g. whitespace). None of this is ascribing meaning to the concept of a code point, it's just using them as a crutch to reason about things like equality. (Of course, equality in Unicode is a whole other can of worms).

hsivonen 10y ago

In addition to the Rust design making more sense for systems programming than Swift's, what Rust puts on the language-level is forward-compatible with future Unicode versions. A future version of Unicode could define combining behavior for currently unassigned code points, which means that iterating by grapheme cluster will change as Swift becomes aware of new Unicode versions.

throwawayish 10y ago

> You deal with strings as bytes (opaque data) (edit2: didn't really mean bytes here. see child comments) more often than strings as text in systems programming, basically. That's true in some sense [1] for unix-ish OSes, but doesn't hold on other operating systems. [1] Even on unix-ish OSes it's pretty obvious that filenames are supposed to be encoded in something and aren't just uint8_t*. There's just no one telling you what that encoding is, so normally it's guessed (eg. $LANG). For filesystems that actually have a defined encoding (eg. NTFS\Win32, JFS, HFS+, APFS and the whole bunch of "non-unix OS FS") these are re-coded by the filesystem driver in the unix-ish OS. Eg. ntfs-3g transcodes filenames to UTF-8; cifs takes a "iocharset" option; and we of course have convmv(1). The idea that "file names and generally everything in the OS is a bunch of bytes" is pretty much only shared by unix-ish systems.

Manishearth 10y ago
Oh, that's not the sense in which I was using the term "bytes". Rust has different string types for file contents and path names, with their own (platform-dependent) restrictions.