4 ms·

This is a great way to see why invalid UTF-8 strings and unicode chars cause undefined behaviour in Rust. `char` is a special integer type, known to have a vali

This is a great way to see why invalid UTF-8 strings and unicode chars cause undefined behaviour in Rust. `char` is a special integer type, known to have a valid range which is a sub-range of its storage type. Outside of dataless enums, this is the only datatype with this behaviour (EDIT: I neglected NonZero<...>/NonZeroXXX and some other zero-niche types). If you manage to construct an invalid char from an invalid string or any other way, you can defeat the niche optimization code and accidentally create yourself an unsound transmute, which is game over for soundness.

NoTeslaThrow 2y ago
> This is a great way to see why invalid UTF-8 strings and unicode chars cause undefined behaviour in Rust. What does "undefined behavior" mean without a spec? Wouldn't the behavior rustc produces today be de-facto defined behavior? It seems like the contention is violating some transmute constraint, but does this not result in reproducible runtime behavior? In what context are you framing "soundness"? EDIT: I'm honestly befuddled why anyone would downvote this. I certainly don't think this is detracting from the conversation at all—how can you understand the semantics of the above comment without understanding what the intended meaning of "undefined behavior" or "soundness" is?
- mmastrac 2y ago
  > What does "undefined behavior" mean without a spec? While not as formalized as C/C++, Rust's "spec" exists in the reference, nomicon, RFCs and documentation. I believe that there is a desire for a spec, but enough resources exist that the community can continue without one with no major negative side-effects (unless you want to re-implement the compiler from scratch, I suppose). The compiler may exploit "lack of UB" for optimizations, e.g., using a known-invalid value as a niche, optimizing away safety checks, etc. > Wouldn't the behavior rustc produces today be de-facto defined behavior? Absolutely not. Bugs are fixed and the behaviour changes. Not often, but it happens. This post probably answers a lot of your reply as well: https://jacko.io/safety_and_soundness.html https://jacko.io/safety_and_soundness.html
  - NoTeslaThrow 2y ago
    EDIT: > While not as formalized as C/C++, Rust's "spec" exists in the reference, nomicon, RFCs and documentation. I believe that there is a desire for a spec, but enough resources exist that the community can continue without one with no major negative side-effects (unless you want to re-implement the compiler from scratch, I suppose). Thank you, I was unaware that this is a thing. > This post probably answers a lot of your reply as well: https://jacko.io/safety_and_soundness.html https://jacko.io/safety_and_soundness.html This appears to also rely on "undefined behavior" as a meaningful term.
    - mmastrac 2y ago
      > This appears to also rely on "undefined behavior" as a meaningful term. I assure you it is a meaningful term: https://llvm.org/docs/UndefinedBehavior.html https://llvm.org/docs/UndefinedBehavior.html
      - NoTeslaThrow 2y ago
        Ok, but in the context of the language at hand? Presumably the IR has distinct semantics from the language that generates the IR. Does UB just strictly resolve to LLVM UB? That's very reasonable!
        fc417fc802 2y ago
        No. UB is a term of art here. Consider a hypothetical non-LLVM full reimplementation of the compiler. If it optimizes and there are invalid assumptions then there is likely UB. LLVM isn't involved in that case though.
        NoTeslaThrow 2y ago
        > If it optimizes and there are invalid assumptions then there is likely UB. It's the distinguishing from bugs that concerns me.
- newpavlov 2y ago
  "Undefined behavior" means that the compiler can apply optimizations which assume that it does not happen, resulting in an incorrect code. A simpler example is `Option<NonZeroU8>`, the compiler assumes that `NonZeroU8` can never contain 0, thus it can use 0 as value for `None`. Now, if you take a reference to the inner `NonZeroU8` stored in `Some` and write 0 to it, you changed `Some` to `None`, while other optimizations may rely on the assumption that references to the content of `Some` can not flip the enum variant to `None`. You don't need a full language spec to declare something UB. And, arguably, from the compiler correctness perspective, there is no fundamental difference between walls of prose in the C/C++ "spec" and the "informal spec" currently used by Rust. (Well, there is the CompCert exception, but it's quite far from the mainstream compilers in many regards)
  - NoTeslaThrow 2y ago
    > resulting in an incorrect code. Incorrect with respect to an assumption furnished where? Your sibling comment mentions RFCs—is this behavior tied to some kind of documented expectation? > A simpler example is `Option<NonZeroU8>`, the compiler assumes that `NonZeroU8` can never contain 0, thus it can use 0 as value for `None`. Now, if you take a reference to the inner `NonZeroU8` stored in `Some` and write 0 to it, you changed `Some` to `None`, while other optimizations may rely on the assumption that references to the content of `Some` can not flip the enum variant to `None`. That seems to be the intended behavior, unless I'm reading incorrectly. Why else would you write a 0 to it? Also, does this not require using the `unsafe` keyword? So is tricking the compiler into producing the behavior you described not the expected and intended behavior?
    - newpavlov 2y ago
      >Incorrect with respect to an assumption furnished where? In the definition of the `NonZeroU8` type. Or in a more practical terms, in LLVM, when we generate LLVM IR we communicate this property to LLVM and it in turn uses it to apply optimizations to our code. >Also, does this not require using the `unsafe` keyword? Yes, it requires `unsafe` and the point is that writing 0 to `NonZeroU8` is UB since it breaks the locality principle critical for correctness of optimizations. Applying just one incorrect (because of the broken assumption) optimization together with numerous other (correct) optimizations can easily lead to very surprising results, which are practically impossible to predict and debug. This is why it's considered such anathema to have UB in code, since having UB in one place may completely break code somewhere far away.
- deleted 2y ago
  [deleted]
- duckerude 2y ago
  It means that anything strange that happens next isn't a language bug. Whether something is a bug or not is sometimes hard to pin down because there's no formal spec. Most of the time it's pretty clear though. Most software doesn't have a formal spec and manages to categorize bugs anyway.
  - NoTeslaThrow 2y ago
    > It means that anything strange that happens next isn't a language bug. This is even more vague. The language is getting blamed regardless. This makes no sense.
    - dwattttt 2y ago
      No: the language defined that e.g. a NonZeroU8 can't contain 0, and the only way it could is via illegal means. You don't need a formal proof to describe that. To try to characterise what any compiler, hypothetical or not, does if you nonetheless produce one (again, via means that aren't valid) isn't meaningful.
- ben0x539 2y ago
  > I'm honestly befuddled why anyone would downvote this. I think there's two parts to this. First, there's a bit of a history of people making disingenious jabs at Rust for not having an "ISO C++" style spec. Typically people would try to suggest that Rust can't be ready for production or shouldn't receive support in other ecosystems without being certified by some manner of international committee. Second, Rust by now has an extensive tradition of people discussing memory safety invariants, what soundness means, formal models of what is a valid memory access, desirable optimizations, etc, etc, so your question what undefined behavior means could be taken to be, like, polemically reductive or dismissive. In context I don't think it's what you're doing, but I would also not be surprised if a lot of people reading Rust-related HN discussions are just super tired of anything that even slightly looks like an effort to re-litigate undefined behavior from first principles, because it tends to derail more specific discussions.
  - NoTeslaThrow 2y ago
    Tbh, I just really hate the term "undefined behavior". It really feels like laziness in terms of what the possible damage might entail.
    - arlort 2y ago
      It is a term of art in compilers/language design though, isn't it? If you break an invariant the compiler is relying on for optimization then you can't say for sure what the effect after all optimisation passes or in future versions of the compiler will be. It's just "undefined"
    - imtringued 2y ago
      Yeah I personally think the problem isn't undefined behavior itself, but the C development culture where undefined behavior is sprinkled all over the language to the point where it has become unavoidable plus the inevitable assignment of blame onto C developers, because everyone knows there is enough time in the day for fuzzing your entire code base.
    - Dylan16807 2y ago
      In a situation like this, causing UB is basically saying you deliberately corrupted your memory. How are you supposed to be specific about what the possible damage might entail for corrupted memory? If you have a function with an "if" or a "while" or a "switch" in it, and you break the variable being evaluated, you might cause the program to skip over the choices and run whatever happens to be next in memory. What's the non-lazy listing of possible outcomes at that point?
timerol 2y ago
> Outside of dataless enums, this is the only datatype with this behaviour. Note that there are non-zero integer types that can also be used in this way, like NonZeroU8 https://doc.rust-lang.org/std/num/type.NonZeroU8.html https://doc.rust-lang.org/std/num/type.NonZeroU8.html. The NULL pointer is also used as a niche, and you can create your own as well, as documented in https://www.0xatticus.com/posts/understanding_rust_niche/ https://www.0xatticus.com/posts/understanding_rust_niche/
- mmastrac 2y ago
  Ack, yeah. I forgot about those despite having used them. That's a good point and I stand corrected. Edited post above.
  - deathanatos 2y ago
    I guess it depends on whether the sentence is only qualified as "integer" types, but bool is sort of the same way, no? A bool must be either 0 or 1 (false or true), or it's UB. (And I think for much the same reason, the niche optimization. Option<bool> is 1 B.) (And for the non-Rustaceans, the only way to get a bool to be not false or true, i.e., not 0 or 1, would be unsafe {} code. Or put differently, not having a bool be "2" is an invariant unsafe code must not violate. (IIRC, at all times, even in unsafe code.))
- tialaramex 2y ago
  Well, in practice you can't make your own non-enum types in stable Rust with this property, to unblock this Rust needs pattern types, (I wanted a path to do this a different way, but I was persuaded it's not a good idea). As that link explains the stdlib is doing this via a perma-unstable compiler use only attribute. But yes, there are the NonZero integers, and you can make your own NonBlah integer using the "XOR trick" for a relatively tiny performance overhead, as well as you can make enums which is how the current CompactString works. The link you gave mentions that Rust does this for other types, but in particular OwnedFd is often useful on Unix systems. Option<OwnedFd> has the same implementation as a C file descriptor, but the same ergonomics as a fancy high level data structure, that's the sort of optimisation we're here for. Alas the Windows equivalent can't do this because different parts of Microsoft use all zeroes and -1 to mean different things, so both are potentially valid.
hinkley 2y ago
I seem to recall someone posting a ridiculously fast utf-8 validator here based on SIMD instructions. Nothing is free but some things can be dirt cheap.
- wolf550e 2y ago
  simdutf [1] from the same people who did simdjson [2] 1 - https://simdutf.github.io/simdutf/ https://simdutf.github.io/simdutf/ 2 - https://simdjson.org/ https://simdjson.org/
tlb 2y ago
If the compiler is using a niche, it should really check every assignment that it's not accidentally the niche. That's still faster than also writing the tag.
- jenadine 2y ago
  It doesn't need to check the assignment because that type cannot be the niche by construction.
  - imtringued 2y ago
    The problem is that you need to validate every potentially written niche after an unsafe block. There is no generic way to re-validate structs in a bounded address space. You'd need something akin to a garbage collector that traces references at fixed offsets including type knowledge. This is not completely infeasible since Rust has a lot of information at compile time to avoid checks, but the extreme cases where people are writing to complicated graph like structures inside unsafe {} can realistically only be dealt with through tracing all safe references that lie inside the bounded address space. In practice it will also be a struggle to sandbox C code into a small enough CHERI style address space so that you don't have to check literally your entire computer's memory after an FFI call. It's not the enums that are the problem. unsafe can break anything if you are determined enough.
    - im3w1l 2y ago
      Isn't it the other way around? An unsafe block must respect niches.
imtringued 2y ago
Fine, I'll take the 4 byte hit for security and safety critical software then. Edit: In retrospect, the optimization doesn't actually cause any security or safety problems, because unsafe code can break any invariant, including an enum with a separated tag and value. The particular memory layout of the enum is irrelevant.
- tialaramex 2y ago
  Exactly. The correct C for working with what may or may not be a valid file descriptor produces the same machine code as the Rust with Option<OwnedFd>. But, if you mess this up and forget to handle an invalid descriptor the C compiler has no idea and your bug goes unnoticed whereas the Rust won't compile because None isn't Some(OwnedFd).

NoTeslaThrow 2y ago

> This is a great way to see why invalid UTF-8 strings and unicode chars cause undefined behaviour in Rust. What does "undefined behavior" mean without a spec? Wouldn't the behavior rustc produces today be de-facto defined behavior? It seems like the contention is violating some transmute constraint, but does this not result in reproducible runtime behavior? In what context are you framing "soundness"? EDIT: I'm honestly befuddled why anyone would downvote this. I certainly don't think this is detracting from the conversation at all—how can you understand the semantics of the above comment without understanding what the intended meaning of "undefined behavior" or "soundness" is?

mmastrac 2y ago
> What does "undefined behavior" mean without a spec? While not as formalized as C/C++, Rust's "spec" exists in the reference, nomicon, RFCs and documentation. I believe that there is a desire for a spec, but enough resources exist that the community can continue without one with no major negative side-effects (unless you want to re-implement the compiler from scratch, I suppose). The compiler may exploit "lack of UB" for optimizations, e.g., using a known-invalid value as a niche, optimizing away safety checks, etc. > Wouldn't the behavior rustc produces today be de-facto defined behavior? Absolutely not. Bugs are fixed and the behaviour changes. Not often, but it happens. This post probably answers a lot of your reply as well: https://jacko.io/safety_and_soundness.html https://jacko.io/safety_and_soundness.html
- NoTeslaThrow 2y ago
  EDIT: > While not as formalized as C/C++, Rust's "spec" exists in the reference, nomicon, RFCs and documentation. I believe that there is a desire for a spec, but enough resources exist that the community can continue without one with no major negative side-effects (unless you want to re-implement the compiler from scratch, I suppose). Thank you, I was unaware that this is a thing. > This post probably answers a lot of your reply as well: https://jacko.io/safety_and_soundness.html https://jacko.io/safety_and_soundness.html This appears to also rely on "undefined behavior" as a meaningful term.
  - mmastrac 2y ago
    > This appears to also rely on "undefined behavior" as a meaningful term. I assure you it is a meaningful term: https://llvm.org/docs/UndefinedBehavior.html https://llvm.org/docs/UndefinedBehavior.html
    - NoTeslaThrow 2y ago
      Ok, but in the context of the language at hand? Presumably the IR has distinct semantics from the language that generates the IR. Does UB just strictly resolve to LLVM UB? That's very reasonable!
      - fc417fc802 2y ago
        No. UB is a term of art here. Consider a hypothetical non-LLVM full reimplementation of the compiler. If it optimizes and there are invalid assumptions then there is likely UB. LLVM isn't involved in that case though.
        NoTeslaThrow 2y ago
        > If it optimizes and there are invalid assumptions then there is likely UB. It's the distinguishing from bugs that concerns me.
newpavlov 2y ago
"Undefined behavior" means that the compiler can apply optimizations which assume that it does not happen, resulting in an incorrect code. A simpler example is `Option<NonZeroU8>`, the compiler assumes that `NonZeroU8` can never contain 0, thus it can use 0 as value for `None`. Now, if you take a reference to the inner `NonZeroU8` stored in `Some` and write 0 to it, you changed `Some` to `None`, while other optimizations may rely on the assumption that references to the content of `Some` can not flip the enum variant to `None`. You don't need a full language spec to declare something UB. And, arguably, from the compiler correctness perspective, there is no fundamental difference between walls of prose in the C/C++ "spec" and the "informal spec" currently used by Rust. (Well, there is the CompCert exception, but it's quite far from the mainstream compilers in many regards)
- NoTeslaThrow 2y ago
  > resulting in an incorrect code. Incorrect with respect to an assumption furnished where? Your sibling comment mentions RFCs—is this behavior tied to some kind of documented expectation? > A simpler example is `Option<NonZeroU8>`, the compiler assumes that `NonZeroU8` can never contain 0, thus it can use 0 as value for `None`. Now, if you take a reference to the inner `NonZeroU8` stored in `Some` and write 0 to it, you changed `Some` to `None`, while other optimizations may rely on the assumption that references to the content of `Some` can not flip the enum variant to `None`. That seems to be the intended behavior, unless I'm reading incorrectly. Why else would you write a 0 to it? Also, does this not require using the `unsafe` keyword? So is tricking the compiler into producing the behavior you described not the expected and intended behavior?
  - newpavlov 2y ago
    >Incorrect with respect to an assumption furnished where? In the definition of the `NonZeroU8` type. Or in a more practical terms, in LLVM, when we generate LLVM IR we communicate this property to LLVM and it in turn uses it to apply optimizations to our code. >Also, does this not require using the `unsafe` keyword? Yes, it requires `unsafe` and the point is that writing 0 to `NonZeroU8` is UB since it breaks the locality principle critical for correctness of optimizations. Applying just one incorrect (because of the broken assumption) optimization together with numerous other (correct) optimizations can easily lead to very surprising results, which are practically impossible to predict and debug. This is why it's considered such anathema to have UB in code, since having UB in one place may completely break code somewhere far away.
deleted 2y ago
[deleted]
duckerude 2y ago
It means that anything strange that happens next isn't a language bug. Whether something is a bug or not is sometimes hard to pin down because there's no formal spec. Most of the time it's pretty clear though. Most software doesn't have a formal spec and manages to categorize bugs anyway.
- NoTeslaThrow 2y ago
  > It means that anything strange that happens next isn't a language bug. This is even more vague. The language is getting blamed regardless. This makes no sense.
  - dwattttt 2y ago
    No: the language defined that e.g. a NonZeroU8 can't contain 0, and the only way it could is via illegal means. You don't need a formal proof to describe that. To try to characterise what any compiler, hypothetical or not, does if you nonetheless produce one (again, via means that aren't valid) isn't meaningful.
ben0x539 2y ago
> I'm honestly befuddled why anyone would downvote this. I think there's two parts to this. First, there's a bit of a history of people making disingenious jabs at Rust for not having an "ISO C++" style spec. Typically people would try to suggest that Rust can't be ready for production or shouldn't receive support in other ecosystems without being certified by some manner of international committee. Second, Rust by now has an extensive tradition of people discussing memory safety invariants, what soundness means, formal models of what is a valid memory access, desirable optimizations, etc, etc, so your question what undefined behavior means could be taken to be, like, polemically reductive or dismissive. In context I don't think it's what you're doing, but I would also not be surprised if a lot of people reading Rust-related HN discussions are just super tired of anything that even slightly looks like an effort to re-litigate undefined behavior from first principles, because it tends to derail more specific discussions.
- NoTeslaThrow 2y ago
  Tbh, I just really hate the term "undefined behavior". It really feels like laziness in terms of what the possible damage might entail.
  - arlort 2y ago
    It is a term of art in compilers/language design though, isn't it? If you break an invariant the compiler is relying on for optimization then you can't say for sure what the effect after all optimisation passes or in future versions of the compiler will be. It's just "undefined"
  - imtringued 2y ago
    Yeah I personally think the problem isn't undefined behavior itself, but the C development culture where undefined behavior is sprinkled all over the language to the point where it has become unavoidable plus the inevitable assignment of blame onto C developers, because everyone knows there is enough time in the day for fuzzing your entire code base.
  - Dylan16807 2y ago
    In a situation like this, causing UB is basically saying you deliberately corrupted your memory. How are you supposed to be specific about what the possible damage might entail for corrupted memory? If you have a function with an "if" or a "while" or a "switch" in it, and you break the variable being evaluated, you might cause the program to skip over the choices and run whatever happens to be next in memory. What's the non-lazy listing of possible outcomes at that point?

timerol 2y ago

> Outside of dataless enums, this is the only datatype with this behaviour. Note that there are non-zero integer types that can also be used in this way, like NonZeroU8 https://doc.rust-lang.org/std/num/type.NonZeroU8.html https://doc.rust-lang.org/std/num/type.NonZeroU8.html. The NULL pointer is also used as a niche, and you can create your own as well, as documented in https://www.0xatticus.com/posts/understanding_rust_niche/ https://www.0xatticus.com/posts/understanding_rust_niche/

mmastrac 2y ago
Ack, yeah. I forgot about those despite having used them. That's a good point and I stand corrected. Edited post above.
- deathanatos 2y ago
  I guess it depends on whether the sentence is only qualified as "integer" types, but bool is sort of the same way, no? A bool must be either 0 or 1 (false or true), or it's UB. (And I think for much the same reason, the niche optimization. Option<bool> is 1 B.) (And for the non-Rustaceans, the only way to get a bool to be not false or true, i.e., not 0 or 1, would be unsafe {} code. Or put differently, not having a bool be "2" is an invariant unsafe code must not violate. (IIRC, at all times, even in unsafe code.))
tialaramex 2y ago
Well, in practice you can't make your own non-enum types in stable Rust with this property, to unblock this Rust needs pattern types, (I wanted a path to do this a different way, but I was persuaded it's not a good idea). As that link explains the stdlib is doing this via a perma-unstable compiler use only attribute. But yes, there are the NonZero integers, and you can make your own NonBlah integer using the "XOR trick" for a relatively tiny performance overhead, as well as you can make enums which is how the current CompactString works. The link you gave mentions that Rust does this for other types, but in particular OwnedFd is often useful on Unix systems. Option<OwnedFd> has the same implementation as a C file descriptor, but the same ergonomics as a fancy high level data structure, that's the sort of optimisation we're here for. Alas the Windows equivalent can't do this because different parts of Microsoft use all zeroes and -1 to mean different things, so both are potentially valid.

hinkley 2y ago

I seem to recall someone posting a ridiculously fast utf-8 validator here based on SIMD instructions. Nothing is free but some things can be dirt cheap.

wolf550e 2y ago
simdutf [1] from the same people who did simdjson [2] 1 - https://simdutf.github.io/simdutf/ https://simdutf.github.io/simdutf/ 2 - https://simdjson.org/ https://simdjson.org/

tlb 2y ago

If the compiler is using a niche, it should really check every assignment that it's not accidentally the niche. That's still faster than also writing the tag.

jenadine 2y ago
It doesn't need to check the assignment because that type cannot be the niche by construction.
- imtringued 2y ago
  The problem is that you need to validate every potentially written niche after an unsafe block. There is no generic way to re-validate structs in a bounded address space. You'd need something akin to a garbage collector that traces references at fixed offsets including type knowledge. This is not completely infeasible since Rust has a lot of information at compile time to avoid checks, but the extreme cases where people are writing to complicated graph like structures inside unsafe {} can realistically only be dealt with through tracing all safe references that lie inside the bounded address space. In practice it will also be a struggle to sandbox C code into a small enough CHERI style address space so that you don't have to check literally your entire computer's memory after an FFI call. It's not the enums that are the problem. unsafe can break anything if you are determined enough.
  - im3w1l 2y ago
    Isn't it the other way around? An unsafe block must respect niches.

imtringued 2y ago

Fine, I'll take the 4 byte hit for security and safety critical software then. Edit: In retrospect, the optimization doesn't actually cause any security or safety problems, because unsafe code can break any invariant, including an enum with a separated tag and value. The particular memory layout of the enum is irrelevant.

tialaramex 2y ago
Exactly. The correct C for working with what may or may not be a valid file descriptor produces the same machine code as the Rust with Option<OwnedFd>. But, if you mess this up and forget to handle an invalid descriptor the C compiler has no idea and your bug goes unnoticed whereas the Rust won't compile because None isn't Some(OwnedFd).