3 ms·

You shouldn't misuse unsafe Rust to create `&str` or `String` with invalid UTF-8. Doing that violates assumptions some optimizations rely on. I even would conte

You shouldn't misuse unsafe Rust to create `&str` or `String` with invalid UTF-8. Doing that violates assumptions some optimizations rely on. I even would contend it's undefined behavior. At least I am convinced that you should never have invalid UTF-8 in `&str` or `String`. If you need that, there's the [bstr][0] crate for byte strings which may or may not be valid UTF-8. Essentially these are just byte slices `&[u8]` and the owned version byte vectors `Vec<u8>` and traits and methods around them. Very useful for text data which is not reliably UTF-8. [0]: https://crates.io/crates/bstr https://crates.io/crates/bstr The idea of using unsafety here is to have a way to tell the Rust compiler: "Trust me, I promise that I only fill the strings with valid UTF-8 data. You don't need to insert code to check UTF-8 validity." And the Rust compiler answers: "Okay. I accept that but if something goes wrong I can't catch that." This allows avoiding repeated validations of a byte slice for UTF-8 validity, for example.

josephg 3y ago
> You shouldn't misuse unsafe Rust to create `&str` or `String` with invalid UTF-8. Sorry - I think my comment wasn't clear. The point of from_utf8_unchecked isn't to create strings with invalid UTF8. The point is that if you know that your string has already been validated, you can skip re-validating it.
steveklabnik 3y ago
It changed, it used to be immediately considered UB, but isn’t any longer, as your grandparent says, that’s what is documented now. It still leads to UB, but isn’t inherently UB on its own. There’s a term for this but I’m forgetting at the moment and search is failing me. I agree that bstr is what you want unless you specifically understand this detail and know why it is the way it is.
- afdbcreid 3y ago
  The term is "library UB" versus "language UB". To be clear, you should still avoid it most of the times.
  - steveklabnik 3y ago
    Ah yes, thank you.
  - _nalply 3y ago
    Would you say library undefined behavior is just a violation of a library invariant? Like `&str` and `String` containing valid UTF-8? I admit that a library invariant is allowed to be violated temporarily under some strict preconditions. One example are bank account money transfers, where the credit and debit don't happen exactly simultaneously. Something similar could happen in a library. But for language UB this doesn't work. Why? Is it because of compile-time optimizations possibly deleting code restoring the invariant?

josephg 3y ago

> You shouldn't misuse unsafe Rust to create `&str` or `String` with invalid UTF-8. Sorry - I think my comment wasn't clear. The point of from_utf8_unchecked isn't to create strings with invalid UTF8. The point is that if you know that your string has already been validated, you can skip re-validating it.

steveklabnik 3y ago

It changed, it used to be immediately considered UB, but isn’t any longer, as your grandparent says, that’s what is documented now. It still leads to UB, but isn’t inherently UB on its own. There’s a term for this but I’m forgetting at the moment and search is failing me. I agree that bstr is what you want unless you specifically understand this detail and know why it is the way it is.

afdbcreid 3y ago
The term is "library UB" versus "language UB". To be clear, you should still avoid it most of the times.
- steveklabnik 3y ago
  Ah yes, thank you.
- _nalply 3y ago
  Would you say library undefined behavior is just a violation of a library invariant? Like `&str` and `String` containing valid UTF-8? I admit that a library invariant is allowed to be violated temporarily under some strict preconditions. One example are bank account money transfers, where the credit and debit don't happen exactly simultaneously. Something similar could happen in a library. But for language UB this doesn't work. Why? Is it because of compile-time optimizations possibly deleting code restoring the invariant?