16 ms·

Uninitialized memory: Unsafe Rust is too hard

106 points by drrlvn 5y ago

remram 5y ago
> Because that raw pointer does not implement deref and because Rust has no -> operator we now need to dereference the pointer permanently to assign the fields with that awkward syntax. Absolutely not, you can still use a mutable reference: let role = &mut *uninit.as_mut_ptr(); role.name = "basic";
- mkeeter 5y ago
  The article says > A mutable reference must also never point to an invalid object, so doing let role = &mut *uninit.as_mut_ptr() if that object is not fully initialized is also wrong. I'm curious who's right here, because I've seen your pattern in code recently!
  - steveklabnik 5y ago
    The docs for as_mut_ptr: https://doc.rust-lang.org/stable/std/mem/union.MaybeUninit.html#method.as_mut_ptr https://doc.rust-lang.org/stable/std/mem/union.MaybeUninit.h... > Incorrect usage of this method: let mut x = MaybeUninit::<Vec<u32>>::uninit(); let x_vec = unsafe { &mut *x.as_mut_ptr() }; // We have created a reference to an uninitialized vector! This is undefined behavior. Also, above, it explicitly describes the intended API for partially initializing a struct: https://doc.rust-lang.org/stable/std/mem/union.MaybeUninit.html#initializing-a-struct-field-by-field https://doc.rust-lang.org/stable/std/mem/union.MaybeUninit.h...
    - mkeeter 5y ago
      Thanks Steve! It turns out I had misremembered; the cast I was thinking of is https://github.com/oxidecomputer/hubris/blob/master/sys/kern/src/startup.rs#L131-L134 https://github.com/oxidecomputer/hubris/blob/master/sys/kern... from &mut MaybeUninit<[T]> to &mut [MaybeUninit<T>], which doesn't construct a reference to something uninitialized.
  - remram 5y ago
    Aren't all the `(*role)` in their code "mutable references" too? (*role).name = "basic";
    - nickitolas 5y ago
      No, this is specifically not creating a mutable reference. So it's fine except for the fact that writing via assignment like this calls the destructor/drop for the old value, if the type has a Drop implementation. That's why &str is fine but String is not, since it will call Drop on a zeroed String (Which is not guaranteed to work) or on an uninitialized String. Using ptr::write instead explicitly does not Drop the old value that is being overwritten. Basically, if you have something like let v = vec![1,2,3]; v = vec![4,5,6]; // Here vec![1,2,3] is dropped otherwise we would be leaking
      - remram 5y ago
        Do you have a source/explanation for *role not being a mutable reference? What is this expression's type then? You mention ptr::write() but it is not used here, and I don't see how you could use it to write to a field?
        gulbanana 5y ago
        (*role) is not a reference at all, it is a value. its type is Role.
        remram 5y ago
        lvalue not mutable reference, got it. I'm surprised there are contexts where one is fine but the other is UB, but I understand the rule.
dathinab 5y ago
The scary thing is: Handling uninitialized memory is hard in C++ (and C), too. You just don't notice and accidentally do it slightly wrong (mainly in C++, in C it's harder to mess up).
- queuebert 5y ago
  Exactly this. And also avoiding initializing memory with zeroes or something is often premature optimization. Very few programs are performant enough to notice the difference.
  - dathinab 5y ago
    In my experience the most common use case for zero-memset is not optimizations but to reduce fallout if you happen to initialize a field..., like a newly added field.
staticassertion 5y ago
I think that the premise here is correct - writing unsafe Rust is too hard. There are lots of footguns. This isn't a very good motivating example but I suppose it does the job of showing the various hoops one has to jump through when using unsafe. I think right now the approach is to make unsafe "safe" (ie std::mem::uninitialized -> MaybeUninit) at the cost of complex, and eventually to build out improved helpers and abstractions. Obviously this is still ongoing. But also, just don't write unsafe? It's very easy to avoid.
- smoldesu 5y ago
  > But also, just don't write unsafe? It's very easy to avoid. Yeah, there's a weird subset of developers who insist on mixing unsafe and safe code even when they're presented equally performant, safe alternatives. One such example was the Actix framework, where the lead dev refused to merge any fixes for his unsafe code. Eventually, so many merge requests showed up to fix his broken code that he just gave up the project altogether and let the community take over. If you want to write unsafe code, I think that's perfectly fine, but Rust is not going to cater to your desires. C and C++ will give you the tools you need with the conveniences you want.
  - DSMan195276 5y ago
    > If you want to write unsafe code, I think that's perfectly fine, but Rust is not going to cater to your desires. C and C++ will give you the tools you need with the conveniences you want. This is a bit of a odd suggestion to me, honestly, you're basically saying Rust is not intended to be a C/C++ replacement. There is definitely a reality that not everything can be written in completely safe Rust, and a lot of the places that Rust could be the most beneficial (Ex. Linux Kernel) are going to require using it.
    - ivraatiems 5y ago
      Calling Rust "a C/C++ replacement" is, I think, slightly underselling what it's supposed to do. As I understand it, the goal of Rust is more "take all these things that were historically hard to do safely and make them easier to do safely" than "replace C/C++ 1 for 1".
Ericson2314 5y ago
The write_unaligned is pure FUD. Regular unpacked structs don't violate alignment on fields!
- raphlinus 5y ago
  In the spirit of being charitable, I would say that the article is highlighting a place where the guarantees are incompletely documented. And not having a solid specification and clear documentation is absolutely one of the things that makes unsafe Rust hard. I'd actually like to qualify that a bit. Doing unsafe is not especially hard, just use pointers everywhere instead of references. That's exactly like C (which doesn't even have references), but the syntax is clunkier, you have to use function calls instead of concise operators like * and ->. What is hard is finely interleaving safe and unsafe Rust, as the author is trying to do. That's difficult because the unsafe code has to upload all the safety invariants of safe Rust, and those are indeed complicated.
  - Ericson2314 5y ago
    > In the spirit of being charitable You are being nice, but even if there is a documentation error, we can prove that if safe rust isn't completely broken, and `& x.field` is allowed in safe Rust, then fields must be aligned. It is just preposterous Rust would be more broken than C in this regard. > but the syntax is clunkier, you have to use function calls instead of concise operators like * and ->. Yes I agree, the syntax does suck. I see the macros use an unstable &raw, that would be more concise. I think would be really good is if x->y in Rust matched &x->y in C. That is nicely orthogonal to dereferencing, and always safe.
    - CheezeIt 5y ago
      There could be some loophole. If the program globally never makes a reference to a field, must it be aligned? If the field is referenced, could it be that when making a reference, the field gets copied or moved out and a reference is implemented as a pointer to that copy?
eddyb 5y ago
> For instance `(*role).name` creates a `&mut &'static str` behind the scenes which is illegal, even if we can't observe it because the memory where it points to is not initialized. Where is this coming from? It's literally not true. The MIR for this has: ((*_3).0: &str) = const "basic"; ((*_3).2: u32) = const 1_u32; ((*_3).1: bool) = const false; So it's only going to do a raw offset and then assign to it, which is identical to `*ptr::addr_of_mut!((*role).field) = value`. Sadly there's no way to tell miri to consider `&mut T` valid only if `T` is valid (that choice is not settled yet, AFAIK, at the language design level), in order to demonstrate the difference (https://github.com/rust-lang/miri/issues/1638 https://github.com/rust-lang/miri/issues/1638). The other claim, "dereferencing is illegal", is more likely, but unlike popular misconception, "dereference" is a syntactic concept, that turns a (pointer/reference) "value" into a "place". There's no "operation" of "dereference" to attach dynamic semantics to. After all, `ptr::addr_of_mut!(*p).write(x)` has to remain as valid as `p.write(x)`, and it does literally contain a "dereference" operation (and so do your field projections). So it's still inaccurate. I believe what you want is to say that in `place = value` the destination `place` has to hold a valid value, as if we were doing `mem::replace(&mut place, value)`. This is indeed true for types that have destructors in them, since those would need to run (which in itself is why `write` on pointers exists - it long existed before any of the newer ideas about "indirect validity" in recent years). However, you have `Copy` types there, and those are definitely not different from `<*mut T>::write` to assign to, today. I don't see us having to change that, but I'm also not seeing any references to where these ideas are coming from. > I'm pretty sure we can depend on things being aligned What do you mean "pretty sure"? Of course you can, otherwise it would be UB to allow safe references to those fields! Anything else would be unsound. In fact, this goes hand in hand with the main significant omission of this post: this is not how you're supposed to use `MaybeUninit`. All of this raw pointer stuff is a distraction from the fact that what you want is `&mut MaybeUninit<FieldType>`. Then all of the things about reference validity are necessarily true, and you can safely initialize the value. The only `unsafe` operation in this entire blog post, that isn't unnecessarily added in, is `assume_init`. What the author doesn't mention is that Rust fails to let you convert between `&mut MaybeUninit<Struct>` and some hypothetical `&mut StructBut<replace Field with MaybeUninit<Field>>` because the language isn't powerful enough to do it automatically. This was one of the saddest things about `MaybeUninit` (and we tried to rectify it for at least arrays). This is where I was going to link to a custom derive that someone has written to generate that kind of transform manually (with the necessary check for safe field access wrt alignment). To my shock, I can't find one. Did I see one and did it have a funny name? (the one thing I did find was a macro crate but unlike a derive those have a harder time checking everything so I had to report https://github.com/youngspe/project-uninit/issues/1 https://github.com/youngspe/project-uninit/issues/1)
mlindner 5y ago
This author doesn't even seem to know C properly so it's hard to accept their reasoning. This in Rust: let mut role: Role = mem::zeroed(); Is not the same as this in C: struct role r; C does not zero initialize.
LinAGKar 5y ago
But the code isn't equivalent. The C code just has a pointer to a manually allocated buffer, while the Rust does the equivalent of zeroing out (or leaving uninitialized) a C++ std::string. Akin to: auto name = reinterpret_cast<std::string >(malloc(sizeof(std::string))); memset(name, 0, sizeof(std::string); *name = "basic"; But on the stack.
andreareina 5y ago
I don't know rust, but why isn't the answer, don't try to do what you'd do in C like construct uninitialized structs?
- dathinab 5y ago
  It's the correct answer. You still sometimes need it like: - when highly optimizing some algorithms - doing FFI So places you find it include some aync runtimes, some algorithm libraries, the standard library. Still often times you initialize it by fully writing it, not by writing fields. Anyway rules are simple: 1. use `ptr::write` instead of `ptr =` 2. use `addr_of_mut!(ptr.x)` instead of `&ptr.x` to get field pointers 3. uhm, `packed` structs are a mess, if you have some you need to take a lot of additional care, this is not limited to rust but also true for C/C++ Also you do not need `#[repr(C)]`, while the rust-specification is pending and as such `repr(Rust)` is pretty much undefined you still can expect fields to be aligned (as else you would have unaligned-`&` which is quite a problem and would likely cause a bunch of breakage through the eco-system).
  - kaba0 5y ago
    There may of course be rare cases where having it uninitialized helps, but I would wager that even than the compiler could optimize it more often than not.
    - dathinab 5y ago
      It's not that simple. Like the unused bu allocated space in a Vec is basically a `[MaybeUninit<T>]`. Or in async runtimes you often have an unsized type with an future trait object inlined (through unsized types anyway need a bit more love ;=) ). Or some C FFI patterns. But yes I would say in pure rust the use cases for `MaybeUninit` are rare, and the cases where you need pointers to fields even rarer. Though while rare in comparison to the amount of code not needing it, still needed enough to be somewhere used (e.g. in a dependency) in many projects even if ignoring std.
      - 5y ago
jcranmer 5y ago
In C, when you declare 'struct role r' (not as a static variable), it is not zeroed. The immediate Rust equivalent would be to use std::mem::uninitialized(), not std::mem::zeroed.
- wyldfire 5y ago
  But an idiom from C which might inspire some unsafe rust is to memset a struct to zero after declaration in order to guarantee that all fields are initialized before anything would access them.
  - MaxBarraclough 5y ago
    If I understand correctly, reading [0], in C99 (or later) you can do that with struct MyStruct foo = {}; This has the effect of initializing all members to zero (or, more precisely, the value which is the same as for objects that have static storage duration [0]). [0] https://gcc.gnu.org/onlinedocs/gcc/Designated-Inits.html https://gcc.gnu.org/onlinedocs/gcc/Designated-Inits.html See also Stop Memsetting Structures: https://news.ycombinator.com/item?id=19766930 https://news.ycombinator.com/item?id=19766930
    - Arch-TK 5y ago
      You need to put something in those braces. An empty initializer list is not allowed by C99 or C11. That being said, you only need to put ONE thing in there, the rest will then be initialized as you described.
      - MaxBarraclough 5y ago
        Thanks, StackOverflow confirms you're correct. [0] That's what I get for using GCC as an approximation of the C standard. (By default, GCC permits empty initializer lists.) [0] https://stackoverflow.com/a/17589839/ https://stackoverflow.com/a/17589839/
  - Lvl999Noob 5y ago
    Is initializing a NonZero field to 0 really initializing it?
sharikous 5y ago
What is the reason for the rule objects have to be always in a good state even inside unsafe?
- judofyr 5y ago
  One of the core ideas behind unsafe blocks is that they don’t actually change any semantics. All they do is allow more operations. This makes it a lot easier to reason about (for both the programmer and the compiler) since there’s not two different set of rules to remember. It does however makes things a bit clunky since the unsafe bits need to ensure a “safe” state throughout the whole block and not only by the end of it.
  - xorvoid 5y ago
    I’ve never understood why they don’t just support partially initialized structs. They’re already doing control flow analysis for initializing variables. It seems like a natural extension to do this for aggregate types (product and sum). From a type theory perspective, a struct T is not a T in unitialized state, it’s. “partial T” that at some point gets transformed into a T. So, you’d not be able to use the T in the normal sense until the compiler can prove that all fields have been init on all possible control flow paths. Why is this problematic? (I presume there is a fatal flaw as it seems too obvious of a solution..)
    - xorvoid 5y ago
      As an example, in C the following is quite common: Data dat; dat.a = 1; dat.b = “foobar”; do_thing(&dat); I seems like the compiler should be able to treat “dat” as an “maybe uninit” type until after “dat.b” gets assigned.
    - comex 5y ago
      Rust already supports partially deinitialized structs - that is, moving out of a struct field by field - and there’s no fundamental reason it can’t support partial initialization too. Indeed, there’s an open issue for it: https://github.com/rust-lang/rust/issues/54987 https://github.com/rust-lang/rust/issues/54987 But there are a lot of desired features with open issues, so don’t expect this to be implemented anytime soon unless someone takes an interest in it.
sAbakumoff 5y ago
I am under the impression that even _safe_ Rust is really hard to learn. Several years ago I started with GoLang and it was so easy to start programming even advanced things almost instantly..Rust drives me crazy. The syntax seems overcomplicated, the compiler errors are cryptic, the IDE is not helpful.
- staticassertion 5y ago
  Rust initially was much harder to learn. The compiler was considerably more strict - rejecting a lot of programs that were entirely valid. That earned rust a very negative "hard language" reputation early on. That hasn't been the case for years. The 2018 edition officially stabilized Non-Lexical Lifetimes, allowing tons of valid programs to work. There have been a lot of other improvements since then to address papercuts. At this point Rust is a pretty easy language to learn imo.
- sidkshatriya 5y ago
  > I am under the impression that even _safe_ Rust is really hard to learn. [...] > The syntax seems overcomplicated, the compiler errors are cryptic, the IDE is not helpful. Yes, Rust is hard to learn. Rust does _seem_ over-complicated. However I like to compare Rust to exercise. You need to do a bit of it before you start reaping the benefits of it. If you suspend your judgement for a bit and try to write some Rust, starting from the very beginning you will find that: - Rust is actually a small language at its core, unlike the monstrosity that is C++ . You don't really need Advanced Rust to be productive. Use Advanced Rust only when you're... advanced - Rust actually is very consistent - The Rust compiler is actually very helpful. It's the least cryptic compiler I've met. But its OK if you feel that now as you're just beginning your journey with Rust Avoid the temptation to "read" Rust from a book. Try to _do_ Rust. Otherwise it might overwhelm you. Simply keep adding Rust techniques to your arsenal as you mature in your usage of Rust. Learning Rust changed the way I look at programing. Rust is a beautiful language. As a random example, just look at the the Firecracker VMM written in Rust -- https://github.com/firecracker-microvm/firecracker https://github.com/firecracker-microvm/firecracker . It would have been able to very difficult for me to understand the codebase if it were written in C/C++! Rust is one of those rare languages I've encountered that if the code compiles, there is a high probability it will work. The type system is that good! TL;DR Persist and you will reap the rewards with Rust.
notpopcorn 5y ago
Without any unsafe code this is simply: let role = Role { name: "basic", flag: 1, disabled: false, }; The language tries to prevent you from interacting with a `Role` object that's not fully initialized. `mem::zero()` could work, but then you'll have to turn the `&'static str` into an `Option<&'static str>` or a raw pointer, to indicate that it might be null. You could also add `#[derive(Default)]` to the struct, to automatically get a `Role::default()` function to create a `Role` with and then modify the fields afterwards, if you want to set the fields in separate statements for some reason: let mut role = Role::default(); role.name = "basic"; role.flag = 1; role.disabled = false; And even with `MaybeUninit` you can initialize the whole struct (without `unsafe`!) with `MaybeUninit::write`. It's just that partially initializing something is hard to get right, which is the point of the article I guess. But I wonder how commonly you would really want that, as it easily leads to mistakes.
- ATsch 5y ago
  A much better way to do partial initialization is by splitting up the struct into multiple parts. This can be easily done in safe rust with Option, or MaybeUninit if you're really desperate for performance.
  - deleted 5y ago
    [deleted]
  - bodhiandpysics1 5y ago
    When dealing with unix syscalls, you actually sometimes need to pass structs that aren't fully initialized, or are initialized to zero with the exception of some fields. The quintessential example is the sigaction struct.
    - pjmlp 5y ago
      Another good example is Win32, in many cases only the length is initialized and the API does the rest, this allows them to change the ABI across versions without impacting the caller.
notpopcorn 5y ago
> So we use a &'static str here instead of a C string so there are some changes to the C code. > [..] > So why does this type not support zero initialization? What do we have to change? Can zeroed not be used at all? Some of you might think that the answer is #[repr(C)] on the struct to force a C layout but that won't solve the problem. The type of the first field was switched to a type (&str) that specifically promises it is never null. If the original type (a pointer) was kept, or a Option<&str> was used, mem::zero would've worked fine.
jcranmer 5y ago
Here's another perspective on why things are the way they are: One of the central philosophies of Rust is that it should not be possible to execute undefined behavior using only safe code. Rust's underlying core semantics end up being very similar to C's semantics, at least in terms of where undefined behavior can arise, and we can imagine Rust's references as being wrappers around the underlying pointer type that have extra requirements to ensure that they can be safely dereferenced in safe code without ever causing UB. So consider a simple pointer dereference in C (*p)... how could that cause UB? Well, the obvious ones are that the pointer could be out-of-bounds or pointing to an expired memory location. So references (& and &mut) most point to a live memory location, even in unsafe code. Also pretty obviously, the pointer would be UB were it unaligned, so a Rust reference must be properly aligned. Another one that should be familiar from the C context is that the memory location must be initialized. So the & reference in Rust means that the memory location must also be initialized... and since &mut implies &, so must &mut. This part is probably genuinely surprising, since it's a rule that doesn't apply to C. The most surprising rule that applies here as well is that the memory location cannot be a trap representation (to use C's terminology). Yes--C has the same requirement here, but most people probably don't come across a platform that has trap representations in C. The reason why std::mem::uninitialized was deprecated in favor of MaybeUninit was that Rust has a type all of whose representations are trap representation (that's the ! type). In short, the author is discovering two related issues here. First, the design of Rust is to push all of the burden of undefined behavior into unsafe code blocks, and the downside of that is that most programmers probably aren't sufficiently cognizant of UB rules to do that rule. Rust also pushes the UB of pointers to reference construction, whereas C makes most of its UB happen only on pointer dereference (constructing unaligned pointers being the exception). The second issue is that Rust's syntax is geared to making safe Rust ergonomic, not unsafe Rust. This means that using the "usual" syntax rules in unsafe Rust blocks is more often than not UB, even when you're trying to avoid the inherent UB construction patterns. Struct projection (given a pointer/reference to a struct, get a pointer/reference to a field) is especially implicated here. These combine when you deal with uninitialized memory references. This is a reasonably common pattern, but designing an always-safe abstraction for uninitialized memory is challenging. And Rust did screw this up, and the stability guidelines means the bad implementations are baked in for good (see, e.g., std::io::Read).
duped 5y ago
To the OP - why should creating uninitialized references with static lifetimes be easy? That is a recipe for undefined behavior - borrows aren't pointers, if you want a pointer to be zero initialized, then use a pointer. If you want safe access to that pointer then wrap it in a struct with an accessor method

106 points by drrlvn 5y ago

> Because that raw pointer does not implement deref and because Rust has no -> operator we now need to dereference the pointer permanently to assign the fields with that awkward syntax. Absolutely not, you can still use a mutable reference: let role = &mut *uninit.as_mut_ptr(); role.name = "basic";

mkeeter 5y ago
The article says > A mutable reference must also never point to an invalid object, so doing let role = &mut *uninit.as_mut_ptr() if that object is not fully initialized is also wrong. I'm curious who's right here, because I've seen your pattern in code recently!
- steveklabnik 5y ago
  The docs for as_mut_ptr: https://doc.rust-lang.org/stable/std/mem/union.MaybeUninit.html#method.as_mut_ptr https://doc.rust-lang.org/stable/std/mem/union.MaybeUninit.h... > Incorrect usage of this method: let mut x = MaybeUninit::<Vec<u32>>::uninit(); let x_vec = unsafe { &mut *x.as_mut_ptr() }; // We have created a reference to an uninitialized vector! This is undefined behavior. Also, above, it explicitly describes the intended API for partially initializing a struct: https://doc.rust-lang.org/stable/std/mem/union.MaybeUninit.html#initializing-a-struct-field-by-field https://doc.rust-lang.org/stable/std/mem/union.MaybeUninit.h...
  - mkeeter 5y ago
    Thanks Steve! It turns out I had misremembered; the cast I was thinking of is https://github.com/oxidecomputer/hubris/blob/master/sys/kern/src/startup.rs#L131-L134 https://github.com/oxidecomputer/hubris/blob/master/sys/kern... from &mut MaybeUninit<[T]> to &mut [MaybeUninit<T>], which doesn't construct a reference to something uninitialized.
- remram 5y ago
  Aren't all the `(*role)` in their code "mutable references" too? (*role).name = "basic";
  - nickitolas 5y ago
    No, this is specifically not creating a mutable reference. So it's fine except for the fact that writing via assignment like this calls the destructor/drop for the old value, if the type has a Drop implementation. That's why &str is fine but String is not, since it will call Drop on a zeroed String (Which is not guaranteed to work) or on an uninitialized String. Using ptr::write instead explicitly does not Drop the old value that is being overwritten. Basically, if you have something like let v = vec![1,2,3]; v = vec![4,5,6]; // Here vec![1,2,3] is dropped otherwise we would be leaking
    - remram 5y ago
      Do you have a source/explanation for *role not being a mutable reference? What is this expression's type then? You mention ptr::write() but it is not used here, and I don't see how you could use it to write to a field?
      - gulbanana 5y ago
        (*role) is not a reference at all, it is a value. its type is Role.
        remram 5y ago
        lvalue not mutable reference, got it. I'm surprised there are contexts where one is fine but the other is UB, but I understand the rule.

dathinab 5y ago

The scary thing is: Handling uninitialized memory is hard in C++ (and C), too. You just don't notice and accidentally do it slightly wrong (mainly in C++, in C it's harder to mess up).

queuebert 5y ago
Exactly this. And also avoiding initializing memory with zeroes or something is often premature optimization. Very few programs are performant enough to notice the difference.
- dathinab 5y ago
  In my experience the most common use case for zero-memset is not optimizations but to reduce fallout if you happen to initialize a field..., like a newly added field.

staticassertion 5y ago

I think that the premise here is correct - writing unsafe Rust is too hard. There are lots of footguns. This isn't a very good motivating example but I suppose it does the job of showing the various hoops one has to jump through when using unsafe. I think right now the approach is to make unsafe "safe" (ie std::mem::uninitialized -> MaybeUninit) at the cost of complex, and eventually to build out improved helpers and abstractions. Obviously this is still ongoing. But also, just don't write unsafe? It's very easy to avoid.

smoldesu 5y ago
> But also, just don't write unsafe? It's very easy to avoid. Yeah, there's a weird subset of developers who insist on mixing unsafe and safe code even when they're presented equally performant, safe alternatives. One such example was the Actix framework, where the lead dev refused to merge any fixes for his unsafe code. Eventually, so many merge requests showed up to fix his broken code that he just gave up the project altogether and let the community take over. If you want to write unsafe code, I think that's perfectly fine, but Rust is not going to cater to your desires. C and C++ will give you the tools you need with the conveniences you want.
- DSMan195276 5y ago
  > If you want to write unsafe code, I think that's perfectly fine, but Rust is not going to cater to your desires. C and C++ will give you the tools you need with the conveniences you want. This is a bit of a odd suggestion to me, honestly, you're basically saying Rust is not intended to be a C/C++ replacement. There is definitely a reality that not everything can be written in completely safe Rust, and a lot of the places that Rust could be the most beneficial (Ex. Linux Kernel) are going to require using it.
  - ivraatiems 5y ago
    Calling Rust "a C/C++ replacement" is, I think, slightly underselling what it's supposed to do. As I understand it, the goal of Rust is more "take all these things that were historically hard to do safely and make them easier to do safely" than "replace C/C++ 1 for 1".

Ericson2314 5y ago

The write_unaligned is pure FUD. Regular unpacked structs don't violate alignment on fields!

raphlinus 5y ago
In the spirit of being charitable, I would say that the article is highlighting a place where the guarantees are incompletely documented. And not having a solid specification and clear documentation is absolutely one of the things that makes unsafe Rust hard. I'd actually like to qualify that a bit. Doing unsafe is not especially hard, just use pointers everywhere instead of references. That's exactly like C (which doesn't even have references), but the syntax is clunkier, you have to use function calls instead of concise operators like * and ->. What is hard is finely interleaving safe and unsafe Rust, as the author is trying to do. That's difficult because the unsafe code has to upload all the safety invariants of safe Rust, and those are indeed complicated.
- Ericson2314 5y ago
  > In the spirit of being charitable You are being nice, but even if there is a documentation error, we can prove that if safe rust isn't completely broken, and `& x.field` is allowed in safe Rust, then fields must be aligned. It is just preposterous Rust would be more broken than C in this regard. > but the syntax is clunkier, you have to use function calls instead of concise operators like * and ->. Yes I agree, the syntax does suck. I see the macros use an unstable &raw, that would be more concise. I think would be really good is if x->y in Rust matched &x->y in C. That is nicely orthogonal to dereferencing, and always safe.
  - CheezeIt 5y ago
    There could be some loophole. If the program globally never makes a reference to a field, must it be aligned? If the field is referenced, could it be that when making a reference, the field gets copied or moved out and a reference is implemented as a pointer to that copy?

eddyb 5y ago

> For instance `(*role).name` creates a `&mut &'static str` behind the scenes which is illegal, even if we can't observe it because the memory where it points to is not initialized. Where is this coming from? It's literally not true. The MIR for this has: ((*_3).0: &str) = const "basic"; ((*_3).2: u32) = const 1_u32; ((*_3).1: bool) = const false; So it's only going to do a raw offset and then assign to it, which is identical to `*ptr::addr_of_mut!((*role).field) = value`. Sadly there's no way to tell miri to consider `&mut T` valid only if `T` is valid (that choice is not settled yet, AFAIK, at the language design level), in order to demonstrate the difference (https://github.com/rust-lang/miri/issues/1638 https://github.com/rust-lang/miri/issues/1638). The other claim, "dereferencing is illegal", is more likely, but unlike popular misconception, "dereference" is a syntactic concept, that turns a (pointer/reference) "value" into a "place". There's no "operation" of "dereference" to attach dynamic semantics to. After all, `ptr::addr_of_mut!(*p).write(x)` has to remain as valid as `p.write(x)`, and it does literally contain a "dereference" operation (and so do your field projections). So it's still inaccurate. I believe what you want is to say that in `place = value` the destination `place` has to hold a valid value, as if we were doing `mem::replace(&mut place, value)`. This is indeed true for types that have destructors in them, since those would need to run (which in itself is why `write` on pointers exists - it long existed before any of the newer ideas about "indirect validity" in recent years). However, you have `Copy` types there, and those are definitely not different from `<*mut T>::write` to assign to, today. I don't see us having to change that, but I'm also not seeing any references to where these ideas are coming from. > I'm pretty sure we can depend on things being aligned What do you mean "pretty sure"? Of course you can, otherwise it would be UB to allow safe references to those fields! Anything else would be unsound. In fact, this goes hand in hand with the main significant omission of this post: this is not how you're supposed to use `MaybeUninit`. All of this raw pointer stuff is a distraction from the fact that what you want is `&mut MaybeUninit<FieldType>`. Then all of the things about reference validity are necessarily true, and you can safely initialize the value. The only `unsafe` operation in this entire blog post, that isn't unnecessarily added in, is `assume_init`. What the author doesn't mention is that Rust fails to let you convert between `&mut MaybeUninit<Struct>` and some hypothetical `&mut StructBut<replace Field with MaybeUninit<Field>>` because the language isn't powerful enough to do it automatically. This was one of the saddest things about `MaybeUninit` (and we tried to rectify it for at least arrays). This is where I was going to link to a custom derive that someone has written to generate that kind of transform manually (with the necessary check for safe field access wrt alignment). To my shock, I can't find one. Did I see one and did it have a funny name? (the one thing I did find was a macro crate but unlike a derive those have a harder time checking everything so I had to report https://github.com/youngspe/project-uninit/issues/1 https://github.com/youngspe/project-uninit/issues/1)

mlindner 5y ago

This author doesn't even seem to know C properly so it's hard to accept their reasoning. This in Rust: let mut role: Role = mem::zeroed(); Is not the same as this in C: struct role r; C does not zero initialize.

LinAGKar 5y ago

But the code isn't equivalent. The C code just has a pointer to a manually allocated buffer, while the Rust does the equivalent of zeroing out (or leaving uninitialized) a C++ std::string. Akin to: auto name = reinterpret_cast<std::string >(malloc(sizeof(std::string))); memset(name, 0, sizeof(std::string); *name = "basic"; But on the stack.

andreareina 5y ago

I don't know rust, but why isn't the answer, don't try to do what you'd do in C like construct uninitialized structs?

dathinab 5y ago
It's the correct answer. You still sometimes need it like: - when highly optimizing some algorithms - doing FFI So places you find it include some aync runtimes, some algorithm libraries, the standard library. Still often times you initialize it by fully writing it, not by writing fields. Anyway rules are simple: 1. use `ptr::write` instead of `ptr =` 2. use `addr_of_mut!(ptr.x)` instead of `&ptr.x` to get field pointers 3. uhm, `packed` structs are a mess, if you have some you need to take a lot of additional care, this is not limited to rust but also true for C/C++ Also you do not need `#[repr(C)]`, while the rust-specification is pending and as such `repr(Rust)` is pretty much undefined you still can expect fields to be aligned (as else you would have unaligned-`&` which is quite a problem and would likely cause a bunch of breakage through the eco-system).
- kaba0 5y ago
  There may of course be rare cases where having it uninitialized helps, but I would wager that even than the compiler could optimize it more often than not.
  - dathinab 5y ago
    It's not that simple. Like the unused bu allocated space in a Vec is basically a `[MaybeUninit<T>]`. Or in async runtimes you often have an unsized type with an future trait object inlined (through unsized types anyway need a bit more love ;=) ). Or some C FFI patterns. But yes I would say in pure rust the use cases for `MaybeUninit` are rare, and the cases where you need pointers to fields even rarer. Though while rare in comparison to the amount of code not needing it, still needed enough to be somewhere used (e.g. in a dependency) in many projects even if ignoring std.
    - 5y ago

jcranmer 5y ago

In C, when you declare 'struct role r' (not as a static variable), it is not zeroed. The immediate Rust equivalent would be to use std::mem::uninitialized(), not std::mem::zeroed.

wyldfire 5y ago
But an idiom from C which might inspire some unsafe rust is to memset a struct to zero after declaration in order to guarantee that all fields are initialized before anything would access them.
- MaxBarraclough 5y ago
  If I understand correctly, reading [0], in C99 (or later) you can do that with struct MyStruct foo = {}; This has the effect of initializing all members to zero (or, more precisely, the value which is the same as for objects that have static storage duration [0]). [0] https://gcc.gnu.org/onlinedocs/gcc/Designated-Inits.html https://gcc.gnu.org/onlinedocs/gcc/Designated-Inits.html See also Stop Memsetting Structures: https://news.ycombinator.com/item?id=19766930 https://news.ycombinator.com/item?id=19766930
  - Arch-TK 5y ago
    You need to put something in those braces. An empty initializer list is not allowed by C99 or C11. That being said, you only need to put ONE thing in there, the rest will then be initialized as you described.
    - MaxBarraclough 5y ago
      Thanks, StackOverflow confirms you're correct. [0] That's what I get for using GCC as an approximation of the C standard. (By default, GCC permits empty initializer lists.) [0] https://stackoverflow.com/a/17589839/ https://stackoverflow.com/a/17589839/
- Lvl999Noob 5y ago
  Is initializing a NonZero field to 0 really initializing it?

sharikous 5y ago

What is the reason for the rule objects have to be always in a good state even inside unsafe?

judofyr 5y ago
One of the core ideas behind unsafe blocks is that they don’t actually change any semantics. All they do is allow more operations. This makes it a lot easier to reason about (for both the programmer and the compiler) since there’s not two different set of rules to remember. It does however makes things a bit clunky since the unsafe bits need to ensure a “safe” state throughout the whole block and not only by the end of it.
- xorvoid 5y ago
  I’ve never understood why they don’t just support partially initialized structs. They’re already doing control flow analysis for initializing variables. It seems like a natural extension to do this for aggregate types (product and sum). From a type theory perspective, a struct T is not a T in unitialized state, it’s. “partial T” that at some point gets transformed into a T. So, you’d not be able to use the T in the normal sense until the compiler can prove that all fields have been init on all possible control flow paths. Why is this problematic? (I presume there is a fatal flaw as it seems too obvious of a solution..)
  - xorvoid 5y ago
    As an example, in C the following is quite common: Data dat; dat.a = 1; dat.b = “foobar”; do_thing(&dat); I seems like the compiler should be able to treat “dat” as an “maybe uninit” type until after “dat.b” gets assigned.
  - comex 5y ago
    Rust already supports partially deinitialized structs - that is, moving out of a struct field by field - and there’s no fundamental reason it can’t support partial initialization too. Indeed, there’s an open issue for it: https://github.com/rust-lang/rust/issues/54987 https://github.com/rust-lang/rust/issues/54987 But there are a lot of desired features with open issues, so don’t expect this to be implemented anytime soon unless someone takes an interest in it.

sAbakumoff 5y ago

I am under the impression that even _safe_ Rust is really hard to learn. Several years ago I started with GoLang and it was so easy to start programming even advanced things almost instantly..Rust drives me crazy. The syntax seems overcomplicated, the compiler errors are cryptic, the IDE is not helpful.

staticassertion 5y ago
Rust initially was much harder to learn. The compiler was considerably more strict - rejecting a lot of programs that were entirely valid. That earned rust a very negative "hard language" reputation early on. That hasn't been the case for years. The 2018 edition officially stabilized Non-Lexical Lifetimes, allowing tons of valid programs to work. There have been a lot of other improvements since then to address papercuts. At this point Rust is a pretty easy language to learn imo.
sidkshatriya 5y ago
> I am under the impression that even _safe_ Rust is really hard to learn. [...] > The syntax seems overcomplicated, the compiler errors are cryptic, the IDE is not helpful. Yes, Rust is hard to learn. Rust does _seem_ over-complicated. However I like to compare Rust to exercise. You need to do a bit of it before you start reaping the benefits of it. If you suspend your judgement for a bit and try to write some Rust, starting from the very beginning you will find that: - Rust is actually a small language at its core, unlike the monstrosity that is C++ . You don't really need Advanced Rust to be productive. Use Advanced Rust only when you're... advanced - Rust actually is very consistent - The Rust compiler is actually very helpful. It's the least cryptic compiler I've met. But its OK if you feel that now as you're just beginning your journey with Rust Avoid the temptation to "read" Rust from a book. Try to _do_ Rust. Otherwise it might overwhelm you. Simply keep adding Rust techniques to your arsenal as you mature in your usage of Rust. Learning Rust changed the way I look at programing. Rust is a beautiful language. As a random example, just look at the the Firecracker VMM written in Rust -- https://github.com/firecracker-microvm/firecracker https://github.com/firecracker-microvm/firecracker . It would have been able to very difficult for me to understand the codebase if it were written in C/C++! Rust is one of those rare languages I've encountered that if the code compiles, there is a high probability it will work. The type system is that good! TL;DR Persist and you will reap the rewards with Rust.

notpopcorn 5y ago

Without any unsafe code this is simply: let role = Role { name: "basic", flag: 1, disabled: false, }; The language tries to prevent you from interacting with a `Role` object that's not fully initialized. `mem::zero()` could work, but then you'll have to turn the `&'static str` into an `Option<&'static str>` or a raw pointer, to indicate that it might be null. You could also add `#[derive(Default)]` to the struct, to automatically get a `Role::default()` function to create a `Role` with and then modify the fields afterwards, if you want to set the fields in separate statements for some reason: let mut role = Role::default(); role.name = "basic"; role.flag = 1; role.disabled = false; And even with `MaybeUninit` you can initialize the whole struct (without `unsafe`!) with `MaybeUninit::write`. It's just that partially initializing something is hard to get right, which is the point of the article I guess. But I wonder how commonly you would really want that, as it easily leads to mistakes.

ATsch 5y ago
A much better way to do partial initialization is by splitting up the struct into multiple parts. This can be easily done in safe rust with Option, or MaybeUninit if you're really desperate for performance.
- deleted 5y ago
  [deleted]
- bodhiandpysics1 5y ago
  When dealing with unix syscalls, you actually sometimes need to pass structs that aren't fully initialized, or are initialized to zero with the exception of some fields. The quintessential example is the sigaction struct.
  - pjmlp 5y ago
    Another good example is Win32, in many cases only the length is initialized and the API does the rest, this allows them to change the ABI across versions without impacting the caller.

notpopcorn 5y ago

> So we use a &'static str here instead of a C string so there are some changes to the C code. > [..] > So why does this type not support zero initialization? What do we have to change? Can zeroed not be used at all? Some of you might think that the answer is #[repr(C)] on the struct to force a C layout but that won't solve the problem. The type of the first field was switched to a type (&str) that specifically promises it is never null. If the original type (a pointer) was kept, or a Option<&str> was used, mem::zero would've worked fine.

jcranmer 5y ago

Here's another perspective on why things are the way they are: One of the central philosophies of Rust is that it should not be possible to execute undefined behavior using only safe code. Rust's underlying core semantics end up being very similar to C's semantics, at least in terms of where undefined behavior can arise, and we can imagine Rust's references as being wrappers around the underlying pointer type that have extra requirements to ensure that they can be safely dereferenced in safe code without ever causing UB. So consider a simple pointer dereference in C (*p)... how could that cause UB? Well, the obvious ones are that the pointer could be out-of-bounds or pointing to an expired memory location. So references (& and &mut) most point to a live memory location, even in unsafe code. Also pretty obviously, the pointer would be UB were it unaligned, so a Rust reference must be properly aligned. Another one that should be familiar from the C context is that the memory location must be initialized. So the & reference in Rust means that the memory location must also be initialized... and since &mut implies &, so must &mut. This part is probably genuinely surprising, since it's a rule that doesn't apply to C. The most surprising rule that applies here as well is that the memory location cannot be a trap representation (to use C's terminology). Yes--C has the same requirement here, but most people probably don't come across a platform that has trap representations in C. The reason why std::mem::uninitialized was deprecated in favor of MaybeUninit was that Rust has a type all of whose representations are trap representation (that's the ! type). In short, the author is discovering two related issues here. First, the design of Rust is to push all of the burden of undefined behavior into unsafe code blocks, and the downside of that is that most programmers probably aren't sufficiently cognizant of UB rules to do that rule. Rust also pushes the UB of pointers to reference construction, whereas C makes most of its UB happen only on pointer dereference (constructing unaligned pointers being the exception). The second issue is that Rust's syntax is geared to making safe Rust ergonomic, not unsafe Rust. This means that using the "usual" syntax rules in unsafe Rust blocks is more often than not UB, even when you're trying to avoid the inherent UB construction patterns. Struct projection (given a pointer/reference to a struct, get a pointer/reference to a field) is especially implicated here. These combine when you deal with uninitialized memory references. This is a reasonably common pattern, but designing an always-safe abstraction for uninitialized memory is challenging. And Rust did screw this up, and the stability guidelines means the bad implementations are baked in for good (see, e.g., std::io::Read).

duped 5y ago

To the OP - why should creating uninitialized references with static lifetimes be easy? That is a recipe for undefined behavior - borrows aren't pointers, if you want a pointer to be zero initialized, then use a pointer. If you want safe access to that pointer then wrap it in a struct with an accessor method