4 ms·

You can't just index into a string. let hello: &[u8] = "Héllo".as_u8_slice(); // hello[1] != "é", I believe it would be the bottom half-bytes of "H".

You can't just index into a string. let hello: &[u8] = "Héllo".as_u8_slice(); // hello[1] != "é", I believe it would be the bottom half-bytes of "H". // hello[2] != "é", I believe it would be the top half-bytes of "é". To use &[u8] would be very non-ergonomic.

SAI_Peregrinus 6y ago
One could, in theory, make an ExtendedGraphemeCluster type, and make str a slice of ExtendedGraphemeClusters. So &[ExtendedGraphemeCluster] could be indexed into without having things not make sense. Of course that's much more complicated than most other primitives, and most people don't have any idea what an Extended Grapheme Cluster even is. But since they're the Unicode notion that most naturally maps to a "character" you could just call the type Character or char, and confuse the hell out of the C programmers by having a variable-width char type.
- josephg 6y ago
  Sure - but iterating by extended graphemes isn’t the only thing you want to do with strings. Sometimes you want to treat them as a bunch of UTF-8 bytes. Sometimes you want to iterate / index by Unicode code points (eg for CRDTs). And sometimes you want to render them, however the system fonts group them. It makes sense to have a special type because it can supports all of this stuff through separate methods on the type. (Or through nearby APIs). It’s confusing, but I think it’s the right choice. Although, I think the most confusing thing about rust strings isn’t that &str isn’t &[u8]. It’s that &str isn’t just &String or something like that.
  - SAI_Peregrinus 6y ago
    Well, &String is ALSO a valid Rust type, it's a reference to a (heap-allocated) String. Really, there's no single, good option that covers all use cases. Which is why Rust ended up with multiple string types, along with ways to convert between them.
    - josephg 6y ago
      > Well, &String is ALSO a valid Rust type Yeah I know. This is really confusing when learning rust. Coming from another language where there's a single rust type its really tricky to figure out when to use what! For reference, I now use String for large strings, string builders, and when the string is owned - like in a struct. And &str when taking in a string as an argument to a function. Or when iterating or something like that where you don't want to move the string. And then for small owned strings (like labels or IDs) I reach for the InlinableString crate. That handles small strings inline and puts large strings on the heap, like what Apple does for Swift and Objective-C. I understand why its this way; but its a pity the best answer rust has is so complex and confusing.
      - deleted 6y ago
        [deleted]
      - brundolf 6y ago
        It's worth noting that you can use &String anywhere that &str is accepted. The distinction is subtle and I agree it's sort of confusing, but in practice the former gets automatically coerced to the latter as needed
- comex 6y ago
  The other issue is that Rust pretty strongly expects slices to be laid out in memory as a flat array of some fixed-size element type. If something isn’t laid out that way, it can’t support all the operations a slice is supposed to support, ranging from the high-level operation of O(1) indexing, to the low-level operation of getting a raw pointer to the slice contents. In principle you could make &[ExtendedGraphemeCluster] special, but it would hamper the ability to write generic code that works on &[T] for any T. Besides, Rust likes to be explicit. Things that are different should look different. In addition, even if you gave up on O(1) indexing by grapheme cluster index, anything faster than O(n) indexing would require some sort of tree data structure to translate grapheme cluster indices to byte indices. Such a structure would be expensive to maintain and not needed the vast majority of the time; in a language that prioritizes performance, it simply couldn’t be part of the default string type no matter what the syntax.
burntsushi 6y ago
The bstr crate is designed to make handling of ASCII-compatible (but not necessarily valid UTF-8) encodings a bit more ergonomic: https://docs.rs/bstr https://docs.rs/bstr

SAI_Peregrinus 6y ago

One could, in theory, make an ExtendedGraphemeCluster type, and make str a slice of ExtendedGraphemeClusters. So &[ExtendedGraphemeCluster] could be indexed into without having things not make sense. Of course that's much more complicated than most other primitives, and most people don't have any idea what an Extended Grapheme Cluster even is. But since they're the Unicode notion that most naturally maps to a "character" you could just call the type Character or char, and confuse the hell out of the C programmers by having a variable-width char type.

josephg 6y ago
Sure - but iterating by extended graphemes isn’t the only thing you want to do with strings. Sometimes you want to treat them as a bunch of UTF-8 bytes. Sometimes you want to iterate / index by Unicode code points (eg for CRDTs). And sometimes you want to render them, however the system fonts group them. It makes sense to have a special type because it can supports all of this stuff through separate methods on the type. (Or through nearby APIs). It’s confusing, but I think it’s the right choice. Although, I think the most confusing thing about rust strings isn’t that &str isn’t &[u8]. It’s that &str isn’t just &String or something like that.
- SAI_Peregrinus 6y ago
  Well, &String is ALSO a valid Rust type, it's a reference to a (heap-allocated) String. Really, there's no single, good option that covers all use cases. Which is why Rust ended up with multiple string types, along with ways to convert between them.
  - josephg 6y ago
    > Well, &String is ALSO a valid Rust type Yeah I know. This is really confusing when learning rust. Coming from another language where there's a single rust type its really tricky to figure out when to use what! For reference, I now use String for large strings, string builders, and when the string is owned - like in a struct. And &str when taking in a string as an argument to a function. Or when iterating or something like that where you don't want to move the string. And then for small owned strings (like labels or IDs) I reach for the InlinableString crate. That handles small strings inline and puts large strings on the heap, like what Apple does for Swift and Objective-C. I understand why its this way; but its a pity the best answer rust has is so complex and confusing.
    - deleted 6y ago
      [deleted]
    - brundolf 6y ago
      It's worth noting that you can use &String anywhere that &str is accepted. The distinction is subtle and I agree it's sort of confusing, but in practice the former gets automatically coerced to the latter as needed
comex 6y ago
The other issue is that Rust pretty strongly expects slices to be laid out in memory as a flat array of some fixed-size element type. If something isn’t laid out that way, it can’t support all the operations a slice is supposed to support, ranging from the high-level operation of O(1) indexing, to the low-level operation of getting a raw pointer to the slice contents. In principle you could make &[ExtendedGraphemeCluster] special, but it would hamper the ability to write generic code that works on &[T] for any T. Besides, Rust likes to be explicit. Things that are different should look different. In addition, even if you gave up on O(1) indexing by grapheme cluster index, anything faster than O(n) indexing would require some sort of tree data structure to translate grapheme cluster indices to byte indices. Such a structure would be expensive to maintain and not needed the vast majority of the time; in a language that prioritizes performance, it simply couldn’t be part of the default string type no matter what the syntax.

burntsushi 6y ago

The bstr crate is designed to make handling of ASCII-compatible (but not necessarily valid UTF-8) encodings a bit more ergonomic: https://docs.rs/bstr https://docs.rs/bstr