7 ms·

> To get rid of the checks, we can either use bytes directly (usually via Vec<u8> / &[u8]) or, if we are absolutely sure the input will be valid UTF-8, use str:

• kibwen 9y ago
  What sort of language beyond "absolutely sure", "require unsafe", and "break your code in surprising ways" would you recommend to make this warning stronger?
  • Anderkent 9y ago
    'break your code in surprising ways' is super vague. Saying 'may cause out of bounds memory access/writes should the input not be valid UTF-8' explicitly is probably more scary.
    • steveklabnik 9y ago
      The nature of undefined behavior is in fact super vague; it's not actually possible to say what will happen.
      • infogulch 9y ago
        While technically true, a specific example helps the reader conceptualize what level of danger that means, when they don't already have a concrete understanding like you do. Recall that the breadth of experience for rust's userbase is much larger than e.g. C.
        • steveklabnik 9y ago
          Oh yeah, I mean, I'm not saying I disagree with adding detail, I can just see the impulse to not, since anything you say may or may not be true.
          • infogulch 9y ago
            me talking about rust's userbase steveklabnik's profile: "Rust core team" I'm guessing you know that a lot better than I do lol.
            • steveklabnik 9y ago
              It's all good! You're absolutely right, and it's easy to forget when you're inside a bubble.
        • TickleSteve 9y ago
          > breadth of experience for rust's userbase is much larger than e.g. C Really? I think you mean the opposite here.
          • steveklabnik 9y ago
            I read this as saying "not depth", that is, C programmers will have a deep understanding here, but since Rust is drawing in a lot of new blood, they on average will not.
      • bonzini 9y ago
        Just say the worst thing that can happen, out of bounds access probably qualifies. :-)
        • lfowles 9y ago
          "Could potentially emit code to rm -rf /"
    • firethief 9y ago
      That's the kind of vagueness that should sound like a warning to anyone considering that approach.
      • kbenson 9y ago
        The nature of warnings means that any time you make assumptions about those that might need the warnings having enough knowledge to correctly assess an ambiguity, you've likely just failed a portion of the people the warning was meant to help. The correct response when warning people of potential problems is never "oh, they should be able to figure out whether this applies to their case".
        • Animats 9y ago
          Right. The correct answer here is to convert external data into UTF-8 once and keep it that way. You're probably not going to become compute-bound, even if you're just reading in text and writing it out again. The UTF-8 check is linear time.
          • burntsushi 9y ago
            Most of the Rust programs I write are compute bound, and finding ways to avoid the separate UTF-8 validation step are critical to their performance.
            • Animats 9y ago
              That's because you write string search programs. Do those even need to run in UTF-8 space, as opposed to pure byte strings?
          • gpderetta 9y ago
            If you are reading text and writing it out you generally do not want to convert to utf8 as it is potentially lossy. Think of unix filters.
      • Asooka 9y ago
  • agwa 9y ago
    Explain what might happen, as I did in my comment. People are more likely to take heed of a warning if it's concrete.
    • rabidferret 9y ago
      "requires unsafe" already implies everything in your comment. `unsafe` literally means "this might violate memory safety"
      • kbenson 9y ago
        No, I don't believe it does. There are perfectly safe operations that must be done in unsafe, because the compiler is not smart enough to determine they are safe. Thus when describing a potential operation, saying it "requires unsafe" does not imply it can blow up in your face, just that the compiler is not smart enough to determine that at this time. This is exactly why warnings should strive to be as clear as possible. Just because you think it's not ambiguous, doesn't mean you aren't just missing some context that someone else might have that makes the statement somewhat ambiguous.
        • topspin 9y ago
          "saying it "requires unsafe" does not imply it can blow up in your face" Good argument. If you advise someone to use unsafe the responsible thing to do is explain precisely what the consequences are besides "it's faster."
          • kibwen 9y ago
            For any API that requires the `unsafe` keyword to use, improper usage of that API risks memory unsafety.
            • dbaupp 9y ago
              There's a subtlety here: the "actual" memory unsafety may manifest elsewhere, if the `unsafe` keyword just allows violating constraints that other (possibly-safe) functions rely on. That is, this touches on the whole "boundary of `unsafe` is the module", where functions marked `unsafe` might do perfectly safe things internally but break invariants that other pieces of code assume are true. From the view of "an API" == "an individual function", it is true that incorrect use of `unsafe` code may not risk any memory unsafety in isolation (e.g. one can pass any integer to Vec::set_len and nothing bad will happen in that call), but it is not so true in the more conventional broader view of an API.
• zigzigzag 9y ago
  That seems like a really unfortunate design decision. I used to think that Java's use of UTF-16 for strings was just a problematic legacy thing, but compared to this it seems quite good. Strings are pretty high performance and there are no complex calculations to do indexing or bounds checks. And in Java 9 the JVM can switch between UTF-16 or Latin1 encodings on the fly, which both uses less RAM and speeds things up simultaneously. There are no memory safety issues caused by character encodings.
  • kibwen 9y ago
    What seems like an unfortunate design decision? I'm unclear what concerns you're seeing that would suggest UTF-16 as preferable to UTF-8, either in terms of performance or memory safety.
    • zigzigzag 9y ago
      To not only use UTF-8 as the internal string encoding but practically mandate it, if you want to remain safe. UTF-8 is a fine transport format, but for raw runtime performance it's obviously going to be an issue if you ever need to iterate over characters, do substring matches, things like that because you can't do constant time "next char" or indexing. UTF-16 doesn't let you do that either in the presence of combining characters, but they're pretty rare and for many operations it doesn't really matter.
      • kccqzy 9y ago
        This has been repeated so many times but UTF-16 does not allow constant time indexing either. Combining characters are one case and they are not rare at all. What about surrogates? What about grapheme clusters that are a complicated sequence of emoji and emoji modifiers with ZWJ? A better suggestion is to rethink why you need those operations in the first place.
      • burntsushi 9y ago
        I feel like your comment contains a lot of misunderstanding about UTF-8. For example, UTF-8 is self-synchronizing, which means you can indeed find the "next char" in constant time. UTF-8 is certainly not a problem for runtime performance. Substring search, for example, is as straight-forward as you might imagine. You have a needle in UTF-8 and a haystack in UTF-8, and a straight-forward application of `memmem` will work just fine (for example). In fact, UTF-8 works out great for performance , because it's very simple to apply fast routines like `memchr`. e.g., If you `memchr` for `a`, then because of UTF-8's self-synchronizing property, any and all matches for `a` actually correspond to the codepoint U+0061. Indexing works fine so long as your indices are byte offsets at valid UTF-8 boundaries. Byte offset indexing tends to be useful for mechanical transformations on a string. For example, if you know your `substring` starts as position `i` in `mystr`, then `&mystr[i + substring.len()..]` gives you the slice of `mystr` immediately following your substring in constant time. When all your APIs deal in byte offsets, this turns out to be a perfectly natural thing to do. Generally speaking, indexing by Unicode codepoint isn't an operation you want to do, because it tends to betray the problem you're trying to solve. For example, if you wanted to display a trimmed string to an end user by "selecting the first 9 characters," then selecting the first 9 codepoints would result in bad things in some circumstances, and it's not just limited to the presence of combining characters. For example, UTF-16 encodes codepoints outside the basic multilingual plane using surrogate pairs, where a surrogate pair consists of two surrogate codepoints that combine to form a single Unicode scalar value (i.e., a non-surrogate codepoint). So if you do the "obvious" thing with UTF-16, you'll wind up with bad results in not-exactly-corner cases. It's worth noting that Rust isn't alone in this. Go represents strings similarly and it also works remarkably well. (The only difference between Go and Rust is that Rust's string type is guaranteed to contain valid UTF-8 where as Go's string type is conventionally UTF-8.) Notably, you won't find "character indexing" anywhere in Go's standard library or various Unicode support libraries. :-) I would very strongly urge you to read my link in my previous comment to you. I think it would help clarify a lot of misconceptions.