4 ms·

Unfortunately this write-up didn’t help me at all understanding pinning. Alomost feels I need to read the article to the end in order to understand the assumpt

Unfortunately this write-up didn’t help me at all understanding pinning. Alomost feels I need to read the article to the end in order to understand the assumptions at the start. I guess I’m missing some context. - what is ‘trivially moveable’? - why would I want to overload the assignment operator? - what is a move constructor? - why would this make memory safe apis tricky? - what does moving references outside unsafe code mean? - what now we suddenly talk about Pin and Unpin, without explaining them first. I’m so confused

Tamschi 5y ago
"Trivially movable" means anyone can take an instance (that they own) and copy its data (just the plain memory cells) to a new location with a new address, then continue to use that instance at that new location without breaking anything. This is a weaker guarantee than `Copy` because they're not necessarily allowed to use both copies of the data as true instances. They can still delay choosing one, but they can only ever interact with one of them through its API. I'll see if I can add a link to an explanation there. In Rust it's mostly a base assumption and many other languages (C#, Java, JS…) avoid the topic almost entirely, so they all don't explain it very prominently. C++ is much more explicit about it, but unfortunately calls this property a bunch of different names. (I've seen "trivially relocatable" as general term for it, since a "move" seems to be a very specific action in C++.) The intro section turned out a bit C++-y, maybe. You can read up on the terms by clicking any underlined text. I'm not too happy about links not being blue, but I can't change it without being in the Hashnode ambassador program. (Not much spare money means my entire web presence is strung together from free or very inexpensive platform services. Maybe I'll post about that in the future.) You can also skip the "The Problem" section entirely, though. It's not that important overall and serves mostly as on-ramp for users of other languages with manual memory management.
- sfvisser 5y ago
  Thanks, this already helps quite a lot! I came to rust from mostly higher level languages, Haskell/JVM/JS. So there is this continuous battle of Rust solving problems that I’m not necessarily sure what they are and why those problems exist in the first place.
  - Tamschi 5y ago
    That's understandable, I originally come from about the same position too, with a bit less functional programming background. I updated the intro to also cover that angle: https://github.com/Tamschi/Abstraction-Haven-Backup/commit/f29e27960c2a697e096714819eec8be0f79d52db https://github.com/Tamschi/Abstraction-Haven-Backup/commit/f... Since it turned out quite a bit longer that way, I was pretty liberal with the bold formatting to make it still easy to skip for those familiar with the issue.
- zozbot234 5y ago
  There was an interesting talk in RustConf 2021 discussing the possibility of adding in-place construction and C++-like "move constructors" for pinned data in Rust. You can find it under the title "Move Constructors: Is It Possible?" by Michel Young de la Sota.
  - Tamschi 5y ago
    There's actually a direct mention of that in my post already, but it's all the way near the bottom in the "Weakening non-moveability" section. It probably goes a bit too far to link in the intro, so I inlined a quick definition as relative clause there instead. The `moveit` library is interesting also because it would allow pinning containers to expose a wider mutable API, though there's currently no trait to update instances post-move, which would allow `realloc` use. I filed an issue for it about a week ago: https://github.com/google/moveit/issues/26 https://github.com/google/moveit/issues/26 That still leaves the issue of callback targets though, which would have to be notified before a possible move to support one-step reallocation.
aliceryhl 5y ago
Trivially moveable means that you can move the object by simply copying the bytes to a new location, and non-trivially-moveable objects are those that would be incorrect to move in this manner. A move constructor is a C++ concept for non-trivially-moveable types that lets you do something more complicated than just copying the bytes over when you want to move the value to a new location, and creating a move constructor is essentially how you overload the assignment operator in C++. The classic example of something that is not trivially moveable is something with a pointer into itself. If you just copy the bytes, that pointer still points at the old location, which would not be valid. In C++, you would handle that by having a move constructor update the pointer to point at the new location. Dealing with non-trivially-moveable types make memory safe APIs tricky because Rust generally guarantees that code with no unsafe block can never result in memory unsafety. This also applies to library design, and anyone designing a library with unsafe blocks in it should design their library to provide the same guarantee — i.e. that anyone using only the safe methods from the library may not be able to cause memory unsafety, even if they can call methods in the library containing unsafe blocks. This should hold no matter how ridiculous the thing they're doing is, as long as it doesn't require unsafe. So the reason why this is tricky is that safe Rust permits anyone to perform a "trivial move" of any object they have ownership of. Thus, if you make even the constructor of such an object safe, then safe code could call that constructor, then move the object in an unsound way. So how do you solve this? Any library that has a non-trivially-moveable type needs some way for the user to promise not to move it using a trivial move. The way they do this is by introducing a Pin type, which is a wrapper around a (typically mutable) reference. The primary constructor for Pin is called new_unchecked, and it is an unsafe method whose safety requirement for calling it is that you never move the object that the reference points at. Thus, any library that sees a reference wrapped in a Pin knows that whoever created it _must_ have called an unsafe block, promising to follow the rules for that unsafe method. It is, in a sense, a way to shift the blame if anything goes wrong in an unsafe block in the library — normally the library would be at blame because the library is the only one with an unsafe block, but when using Pin, it is the person who _created_ the Pin object who is at blame for the incorrect use of unsafe — not the library. It is worth noting that new_unchecked is not the only way to create a Pin. For example, there's a macro called pin! that uses variable shadowing to prevent the user from accessing the underlying variable, preventing the user from moving the object since they can't access the variable name anymore. This lets safe code create a Pin reference to something the safe code owns, and the unsafe block is inside the definition of the macro. Regarding moving references outside unsafe code, well that's because if someone gets hold of a mutable reference to a pinned value, then they could call std::mem::swap to perform a trivial move of that value, and this would not involve an unsafe block. Thus, if the library exposes any way of getting such a mutable reference in safe code, then that would be an incorrect unsafe block, as safe code is allowed to do anything it wants, as long as it uses only unsafe. The Pin wrapper type has methods for unwrapping it and getting back the mutable reference that it is a wrapper around, however they are unsafe, so if anyone did that and then called std::mem::swap, they would be at fault because the one who unwrapped it called an unsafe method without following its rules. As for Unpin, that's a marker on types that are trivially moveable. The Pin wrapper is essentially a no-op for references to those types and provides no guarantees. There's a safe Pin::new method that can only be used for such types. It's worth pointing out that in Rust, non-trivially-moveable types must still be trivially moveable _until_ they get pinned. So they typically have an "init" state or something like that which can be safely moved around, but then once you pin it, the object becomes non-trivially-moveable.

Tamschi 5y ago

"Trivially movable" means anyone can take an instance (that they own) and copy its data (just the plain memory cells) to a new location with a new address, then continue to use that instance at that new location without breaking anything. This is a weaker guarantee than `Copy` because they're not necessarily allowed to use both copies of the data as true instances. They can still delay choosing one, but they can only ever interact with one of them through its API. I'll see if I can add a link to an explanation there. In Rust it's mostly a base assumption and many other languages (C#, Java, JS…) avoid the topic almost entirely, so they all don't explain it very prominently. C++ is much more explicit about it, but unfortunately calls this property a bunch of different names. (I've seen "trivially relocatable" as general term for it, since a "move" seems to be a very specific action in C++.) The intro section turned out a bit C++-y, maybe. You can read up on the terms by clicking any underlined text. I'm not too happy about links not being blue, but I can't change it without being in the Hashnode ambassador program. (Not much spare money means my entire web presence is strung together from free or very inexpensive platform services. Maybe I'll post about that in the future.) You can also skip the "The Problem" section entirely, though. It's not that important overall and serves mostly as on-ramp for users of other languages with manual memory management.

sfvisser 5y ago
Thanks, this already helps quite a lot! I came to rust from mostly higher level languages, Haskell/JVM/JS. So there is this continuous battle of Rust solving problems that I’m not necessarily sure what they are and why those problems exist in the first place.
- Tamschi 5y ago
  That's understandable, I originally come from about the same position too, with a bit less functional programming background. I updated the intro to also cover that angle: https://github.com/Tamschi/Abstraction-Haven-Backup/commit/f29e27960c2a697e096714819eec8be0f79d52db https://github.com/Tamschi/Abstraction-Haven-Backup/commit/f... Since it turned out quite a bit longer that way, I was pretty liberal with the bold formatting to make it still easy to skip for those familiar with the issue.
zozbot234 5y ago
There was an interesting talk in RustConf 2021 discussing the possibility of adding in-place construction and C++-like "move constructors" for pinned data in Rust. You can find it under the title "Move Constructors: Is It Possible?" by Michel Young de la Sota.
- Tamschi 5y ago
  There's actually a direct mention of that in my post already, but it's all the way near the bottom in the "Weakening non-moveability" section. It probably goes a bit too far to link in the intro, so I inlined a quick definition as relative clause there instead. The `moveit` library is interesting also because it would allow pinning containers to expose a wider mutable API, though there's currently no trait to update instances post-move, which would allow `realloc` use. I filed an issue for it about a week ago: https://github.com/google/moveit/issues/26 https://github.com/google/moveit/issues/26 That still leaves the issue of callback targets though, which would have to be notified before a possible move to support one-step reallocation.

aliceryhl 5y ago

Trivially moveable means that you can move the object by simply copying the bytes to a new location, and non-trivially-moveable objects are those that would be incorrect to move in this manner. A move constructor is a C++ concept for non-trivially-moveable types that lets you do something more complicated than just copying the bytes over when you want to move the value to a new location, and creating a move constructor is essentially how you overload the assignment operator in C++. The classic example of something that is not trivially moveable is something with a pointer into itself. If you just copy the bytes, that pointer still points at the old location, which would not be valid. In C++, you would handle that by having a move constructor update the pointer to point at the new location. Dealing with non-trivially-moveable types make memory safe APIs tricky because Rust generally guarantees that code with no unsafe block can never result in memory unsafety. This also applies to library design, and anyone designing a library with unsafe blocks in it should design their library to provide the same guarantee — i.e. that anyone using only the safe methods from the library may not be able to cause memory unsafety, even if they can call methods in the library containing unsafe blocks. This should hold no matter how ridiculous the thing they're doing is, as long as it doesn't require unsafe. So the reason why this is tricky is that safe Rust permits anyone to perform a "trivial move" of any object they have ownership of. Thus, if you make even the constructor of such an object safe, then safe code could call that constructor, then move the object in an unsound way. So how do you solve this? Any library that has a non-trivially-moveable type needs some way for the user to promise not to move it using a trivial move. The way they do this is by introducing a Pin type, which is a wrapper around a (typically mutable) reference. The primary constructor for Pin is called new_unchecked, and it is an unsafe method whose safety requirement for calling it is that you never move the object that the reference points at. Thus, any library that sees a reference wrapped in a Pin knows that whoever created it _must_ have called an unsafe block, promising to follow the rules for that unsafe method. It is, in a sense, a way to shift the blame if anything goes wrong in an unsafe block in the library — normally the library would be at blame because the library is the only one with an unsafe block, but when using Pin, it is the person who _created_ the Pin object who is at blame for the incorrect use of unsafe — not the library. It is worth noting that new_unchecked is not the only way to create a Pin. For example, there's a macro called pin! that uses variable shadowing to prevent the user from accessing the underlying variable, preventing the user from moving the object since they can't access the variable name anymore. This lets safe code create a Pin reference to something the safe code owns, and the unsafe block is inside the definition of the macro. Regarding moving references outside unsafe code, well that's because if someone gets hold of a mutable reference to a pinned value, then they could call std::mem::swap to perform a trivial move of that value, and this would not involve an unsafe block. Thus, if the library exposes any way of getting such a mutable reference in safe code, then that would be an incorrect unsafe block, as safe code is allowed to do anything it wants, as long as it uses only unsafe. The Pin wrapper type has methods for unwrapping it and getting back the mutable reference that it is a wrapper around, however they are unsafe, so if anyone did that and then called std::mem::swap, they would be at fault because the one who unwrapped it called an unsafe method without following its rules. As for Unpin, that's a marker on types that are trivially moveable. The Pin wrapper is essentially a no-op for references to those types and provides no guarantees. There's a safe Pin::new method that can only be used for such types. It's worth pointing out that in Rust, non-trivially-moveable types must still be trivially moveable _until_ they get pinned. So they typically have an "init" state or something like that which can be safely moved around, but then once you pin it, the object becomes non-trivially-moveable.