5 ms·

Does this bug exist because it would be too expensive to check every string before slicing? (Being Rust-ignorant), can you not type a binary as UTF-8? Are there

Does this bug exist because it would be too expensive to check every string before slicing? (Being Rust-ignorant), can you not type a binary as UTF-8? Are there 2 versions of string functions, fast ones that assume ASCII and slow ones that assume UTF-8?

steveklabnik 8y ago
Every string is checked. But UTF8 is a multi-byte encoding, and slicing works per bytes, so you if you slice in the middle of a multi-byte character, you may get nonsense. The error happens because of this checking, not in spite of it. String always assumes full UTF-8. You could make an AsciiString type if you wanted, but it's not provided by the standard library.
- yurymik 8y ago
  Why not this? fn main() { let a = "ab早".as_bytes(); let a = &a[..3]; println!("Hello, world!"); }
  - steveklabnik 8y ago
    It's not clear to me what you're suggesting; is it that String shouldn't have supported indexing in the first place? That code does work, but you have a &[u8] not a &str.
    - deleted 8y ago
      [deleted]
    - dkarl 8y ago
      I think he's suggesting that slicing on strings should be by character, and if you want to slice on bytes, you should explicitly ask to treat the string as a byte array. It makes more sense semantically, and it's safe.
      - richardwhiuk 8y ago
        Slicing on characters is a linear time operation and indexing is meant to be cheap.
        dkarl 8y ago
        That seems like taking it too far. It's like using pointer arithmetic to index a linked list on the assumption that the nodes happen to be allocated contiguously in memory. I mean, I guess the thinking is, indexing a Unicode string isn't cheap, but indexing strings used to be cheap once upon a time, when strings were encoded in fixed one-byte-per-character representations, so let's pretend that's still the case and panic if it doesn't work out.... That's weirdly antithetical to Rust's purported focus on safety. Also, you can get the same performance from an operation that returns a byte array instead of a string. If that kind of performance is what you want, then a Unicode string is simply not the right type to use.
        Retra 8y ago
        Panics are not unsafe. Panic exists in Rust because they are safe. If you don't want a panic on index, just don't index. Indexing into a UTF-8 string doesn't serve any reasonable consistent purpose anyway, because it is an abstraction of text that doesn't provide support to the notion that a "character" is more fundamental than a word or paragraph, etc. Rust's string slicing exists solely to make ASCII text easy to handle. If your text is not ASCII, then you shouldn't be slicing it at all. Thus the panic.
        int_19h 8y ago
        I think that's too extreme. There are many legitimate reasons to slice non-ASCII text - for example, to split it on newlines.
        mlindner 8y ago
        That's not trivial and different languages vary in how they handle new line characters even. https://stackoverflow.com/questions/44995851/how-do-i-check-if-a-character-is-a-unicode-new-line-character-not-only-ascii-i https://stackoverflow.com/questions/44995851/how-do-i-check-...
    - yurymik 8y ago
      But neither AsciiString. It has a as_str method, but it's still a kludge. This example was basically a suggestion to throw0u1t: if they want to cut in the middle of the utf-sequence for whatever reason, they can [edit:] do it without extra crates. What I don't understand is why slices are indexed in bytes and not in objects. If String has an ability to check that we're cutting in the middle of the character sequence, why doesn't it provide an ability to take 3 fully formed characters.
      - steveklabnik 8y ago
        > What I don't understand is why slices are indexed in bytes and not in objects. Slicing is an O(1) operation, and that would be an O(n) operation.
      - Someone 8y ago
        I think the rust designers want to keep the implicit contract that indexing into a string is fast and O(1). If you want to find the one millionth codepoint of a UTF8-encoded string, you have to more or less (1) visit every byte of the string. If, on the other hand, you want to find the codepoint that covers the millionth byte, on the other hand, you have to read at most four bytes (read the millionth byte, and there are three cases: - it’s a full codepoint. If so, you‘re done. - it is the first byte of a multi-byte codepoint. If so, read forwards in the string for up to 3 continuation characters. - it is a continuation character. If so, search backwards in the string for the first byte, then, if necessary, read forwards to find more continuation characters. So, that is O(1) (1) you can skip continuation characters, but these typically are rare.
      - TheCoelacanth 8y ago
        It does: `s.chars().take(3)`. It just does it with iterators rather than with indexes because that better communicates the performance characteristics.
- da_chicken 8y ago
  The obvious follow up question would be: so why is slicing a string a byte-wise operation and not a character-wise operation? If a string is an array of characters, why does it let me refer to individual bytes without explicitly casting it to a byte array? How often comparatively do you want the nth byte compared to the nth character? I would suspect that's pretty rare.
  - steveklabnik 8y ago
    As stated below, indexing is an O(1) operation, and that is a O(n) operation. > If a string is an array of characters It is not, it is an array (technically vector) of bytes.
    - da_chicken 8y ago
      Who cares if it's O(1) if it causes a panic? What good is high performance if it doesn't complete or isn't safe? At the very least, shouldn't there be an O(n) method to do character-wise slicing?
      - steveklabnik 8y ago
        panics are safe. You expect the “I don’t have a bug” case to be fast. You can, but it depends on what you mean by “character”, as that’s not a concept in Unicode. Every kind of thing you could mean has a method, specific to it, since they’re different things. (char in Rust is a Unicode scalar value, and you can collect into a Vec<char> and then slice it, as an example of one of those things. And that’s still O(1) at the cost of using up to four times the memory.)
  - burntsushi 8y ago
    > How often comparatively do you want the nth byte compared to the nth character? I would suspect that's pretty rare. It's exactly the opposite of what you expect. Getting the nth codepoint is often (not always) semantically incorrect since a codepoint isn't necessarily one character. Multiple codepoints might combine to form one character. (In Unicode, these are called grapheme clusters.) Byte offsets are used a ton because you might often have the index to a position in the string from some routine, like, say, a search[1]. I've been working on text related things in both Rust and Go for several years. Both languages got this part of their strings exactly right given that their representation in memory is always a sequence of bytes. [1] - https://doc.rust-lang.org/std/primitive.str.html#method.find https://doc.rust-lang.org/std/primitive.str.html#method.find
deleted 8y ago
[deleted]

steveklabnik 8y ago

Every string is checked. But UTF8 is a multi-byte encoding, and slicing works per bytes, so you if you slice in the middle of a multi-byte character, you may get nonsense. The error happens because of this checking, not in spite of it. String always assumes full UTF-8. You could make an AsciiString type if you wanted, but it's not provided by the standard library.

yurymik 8y ago
Why not this? fn main() { let a = "ab早".as_bytes(); let a = &a[..3]; println!("Hello, world!"); }
- steveklabnik 8y ago
  It's not clear to me what you're suggesting; is it that String shouldn't have supported indexing in the first place? That code does work, but you have a &[u8] not a &str.
  - deleted 8y ago
    [deleted]
  - dkarl 8y ago
    I think he's suggesting that slicing on strings should be by character, and if you want to slice on bytes, you should explicitly ask to treat the string as a byte array. It makes more sense semantically, and it's safe.
    - richardwhiuk 8y ago
      Slicing on characters is a linear time operation and indexing is meant to be cheap.
      - dkarl 8y ago
        That seems like taking it too far. It's like using pointer arithmetic to index a linked list on the assumption that the nodes happen to be allocated contiguously in memory. I mean, I guess the thinking is, indexing a Unicode string isn't cheap, but indexing strings used to be cheap once upon a time, when strings were encoded in fixed one-byte-per-character representations, so let's pretend that's still the case and panic if it doesn't work out.... That's weirdly antithetical to Rust's purported focus on safety. Also, you can get the same performance from an operation that returns a byte array instead of a string. If that kind of performance is what you want, then a Unicode string is simply not the right type to use.
        Retra 8y ago
        Panics are not unsafe. Panic exists in Rust because they are safe. If you don't want a panic on index, just don't index. Indexing into a UTF-8 string doesn't serve any reasonable consistent purpose anyway, because it is an abstraction of text that doesn't provide support to the notion that a "character" is more fundamental than a word or paragraph, etc. Rust's string slicing exists solely to make ASCII text easy to handle. If your text is not ASCII, then you shouldn't be slicing it at all. Thus the panic.
        int_19h 8y ago
        I think that's too extreme. There are many legitimate reasons to slice non-ASCII text - for example, to split it on newlines.
        mlindner 8y ago
        That's not trivial and different languages vary in how they handle new line characters even. https://stackoverflow.com/questions/44995851/how-do-i-check-if-a-character-is-a-unicode-new-line-character-not-only-ascii-i https://stackoverflow.com/questions/44995851/how-do-i-check-...
  - yurymik 8y ago
    But neither AsciiString. It has a as_str method, but it's still a kludge. This example was basically a suggestion to throw0u1t: if they want to cut in the middle of the utf-sequence for whatever reason, they can [edit:] do it without extra crates. What I don't understand is why slices are indexed in bytes and not in objects. If String has an ability to check that we're cutting in the middle of the character sequence, why doesn't it provide an ability to take 3 fully formed characters.
    - steveklabnik 8y ago
      > What I don't understand is why slices are indexed in bytes and not in objects. Slicing is an O(1) operation, and that would be an O(n) operation.
    - Someone 8y ago
      I think the rust designers want to keep the implicit contract that indexing into a string is fast and O(1). If you want to find the one millionth codepoint of a UTF8-encoded string, you have to more or less (1) visit every byte of the string. If, on the other hand, you want to find the codepoint that covers the millionth byte, on the other hand, you have to read at most four bytes (read the millionth byte, and there are three cases: - it’s a full codepoint. If so, you‘re done. - it is the first byte of a multi-byte codepoint. If so, read forwards in the string for up to 3 continuation characters. - it is a continuation character. If so, search backwards in the string for the first byte, then, if necessary, read forwards to find more continuation characters. So, that is O(1) (1) you can skip continuation characters, but these typically are rare.
    - TheCoelacanth 8y ago
      It does: `s.chars().take(3)`. It just does it with iterators rather than with indexes because that better communicates the performance characteristics.
da_chicken 8y ago
The obvious follow up question would be: so why is slicing a string a byte-wise operation and not a character-wise operation? If a string is an array of characters, why does it let me refer to individual bytes without explicitly casting it to a byte array? How often comparatively do you want the nth byte compared to the nth character? I would suspect that's pretty rare.
- steveklabnik 8y ago
  As stated below, indexing is an O(1) operation, and that is a O(n) operation. > If a string is an array of characters It is not, it is an array (technically vector) of bytes.
  - da_chicken 8y ago
    Who cares if it's O(1) if it causes a panic? What good is high performance if it doesn't complete or isn't safe? At the very least, shouldn't there be an O(n) method to do character-wise slicing?
    - steveklabnik 8y ago
      panics are safe. You expect the “I don’t have a bug” case to be fast. You can, but it depends on what you mean by “character”, as that’s not a concept in Unicode. Every kind of thing you could mean has a method, specific to it, since they’re different things. (char in Rust is a Unicode scalar value, and you can collect into a Vec<char> and then slice it, as an example of one of those things. And that’s still O(1) at the cost of using up to four times the memory.)
- burntsushi 8y ago
  > How often comparatively do you want the nth byte compared to the nth character? I would suspect that's pretty rare. It's exactly the opposite of what you expect. Getting the nth codepoint is often (not always) semantically incorrect since a codepoint isn't necessarily one character. Multiple codepoints might combine to form one character. (In Unicode, these are called grapheme clusters.) Byte offsets are used a ton because you might often have the index to a position in the string from some routine, like, say, a search[1]. I've been working on text related things in both Rust and Go for several years. Both languages got this part of their strings exactly right given that their representation in memory is always a sequence of bytes. [1] - https://doc.rust-lang.org/std/primitive.str.html#method.find https://doc.rust-lang.org/std/primitive.str.html#method.find

deleted 8y ago

[deleted]