5 ms·

Pinning in Plain English

33 points by lukastyrychtr 5y ago

2bitencryption 5y ago
Not to detract from the author's work - it does express the concept in "plain English" as promised - but boy oh boy is Pinning a confusing concept. This is coming from someone who has used rust pretty regularly since 2015. Sure, in simplest terms, I get it - "Pinning is used to guarantee something is never moved", and the utility of this is mostly (exclusively?) for guarantees in concurrency. The moment I see the symbol "!Unpin" my brain goes into a fog. "Pinning means things cannot be moved... but most things are Unpin... which can be unpinned after being pinned... but some things are !Unpin... so they can NOT be UN-pinned... so when they are Pinnned they are REALLY pinned..." It seems to me like "Pin" is a perfectly sound, logical concept, but it was designed by type-system-science nerds, that barely translates to the promise of Rust which is to empower everyone, not just type-system-science nerds, to make safe software.
- vvanders 5y ago
  I would largely agree, in the cases where I've needed something pinned I just drop to Box'ing to, keeping the box internal and managing the pinning directly. It's annoying because you lose some classes of things that you may want to pin(values on stack for short durations) but I found similar complexities in the API. For instance if you take a pinned Arc(ex: Pin<Arc<T>>) there wasn't a way I could determine to get out the mutable interior pointer(ex: via Arc::get_mut) since those APIs require a specific signature and you would need to un-pin the value to make those calls despite them being by reference(safe from a pinning perspective). Most of Rust is incredibly well designed but so it's pretty jarring that Pinning is so much harder to use. I've found work-arounds for the cases where I've needed either via boxing or dropping to unsafe but would have been nice if it had worked for my use cases.
  - cbarrick 5y ago
    > safe from a pinning perspective The reason you can't get &mut T from Pin<Arc<T>> is because the mutable reference lets you move the data in safe Rust, e.g. with std::mem::swap. So getting the &mut T from a Pin<Arc<T>> is not safe from a pinning perspective.
    - vvanders 5y ago
      Fair, but feels like that significantly reduces the utility of pinned types then if for certain values you lose the ability to mutate values when the compiler overwise knows that there is a single mutable reference.
      - Tamschi 5y ago
        There's a hole or oversight in `Arc`'s API in this regard, annoyingly. The function you're looking for would be something like pub fn get_mut_pinned(this: &mut Pin<Arc<T>>) -> Option<Pin<&mut T>> , which for pinning-aware values should give you enough mutability and for `T: Unpin` can give you `Option<&mut T>` through `Option::as_deref_mut`. Unfortunately, I haven't seen any `Arc` implementations that actually provide it, since almost none of the third-party ones are pinning-aware. (My `tiptoe::Arc`'s `get_mut` has this signature, but that's a specialty container with additional requirements for the contained value.) The same goes for `make_mut_pinned`, that's missing too (but to be fair would be much less useful, since pinning and `Clone` don't often mix that well).
        zozbot234 5y ago
        Have you tried proposing these API's on the internals.rust-lang.org forum or via posting a proposed RFC? It's not clear to me if they're sound in the general case, but if that's the case they can absolutely be added.
        Tamschi 5y ago
        I should. I'm not too familiar with the process so far, but I'll find some time for it. These functions are indeed generally sound (also for `Rc`), as any value that cares would be `!Unpin`, which would still bar access to `&mut T`. `make_mut_pinned` also shouldn't cause too much confusion, as availability of `Clone` would have to be declared explicitly just about everywhere that's relevant. They can be implemented as very thin wrappers around their non-pinning equivalents, with only a few `unsafe` operations to make the types fit.
sfvisser 5y ago
Unfortunately this write-up didn’t help me at all understanding pinning. Alomost feels I need to read the article to the end in order to understand the assumptions at the start. I guess I’m missing some context. - what is ‘trivially moveable’? - why would I want to overload the assignment operator? - what is a move constructor? - why would this make memory safe apis tricky? - what does moving references outside unsafe code mean? - what now we suddenly talk about Pin and Unpin, without explaining them first. I’m so confused
- Tamschi 5y ago
  "Trivially movable" means anyone can take an instance (that they own) and copy its data (just the plain memory cells) to a new location with a new address, then continue to use that instance at that new location without breaking anything. This is a weaker guarantee than `Copy` because they're not necessarily allowed to use both copies of the data as true instances. They can still delay choosing one, but they can only ever interact with one of them through its API. I'll see if I can add a link to an explanation there. In Rust it's mostly a base assumption and many other languages (C#, Java, JS…) avoid the topic almost entirely, so they all don't explain it very prominently. C++ is much more explicit about it, but unfortunately calls this property a bunch of different names. (I've seen "trivially relocatable" as general term for it, since a "move" seems to be a very specific action in C++.) The intro section turned out a bit C++-y, maybe. You can read up on the terms by clicking any underlined text. I'm not too happy about links not being blue, but I can't change it without being in the Hashnode ambassador program. (Not much spare money means my entire web presence is strung together from free or very inexpensive platform services. Maybe I'll post about that in the future.) You can also skip the "The Problem" section entirely, though. It's not that important overall and serves mostly as on-ramp for users of other languages with manual memory management.
  - sfvisser 5y ago
    Thanks, this already helps quite a lot! I came to rust from mostly higher level languages, Haskell/JVM/JS. So there is this continuous battle of Rust solving problems that I’m not necessarily sure what they are and why those problems exist in the first place.
dtolnay 5y ago
> Pins are very often `Unpin`, but this isn't relevant to their function. This idea is what I've seen get a lot of people lost, but it's pretty natural. By analogy, a DANGER HIGH VOLTAGE sign doesn't itself need to be dangerous or high voltage in order to be useful. If your DANGER HIGH VOLTAGE sign (the sign) is dangerous or high voltage, you'd probably want a second DANGER HIGH VOLTAGE sign pointing out this fact about the first sign. The same for Pin: if you're wanting the pinned pointer (the pointer itself) to not be Unpin, your pinned pointer would need to be pinned by some second pinned pointer.
- Tamschi 5y ago
  This is why I insist on never calling it "pinned pointer" in the post, even if not explicitly. I'm aware the term is used in the standard library docs (for context: `Pin`'s tagline is "A pinned pointer."), but in my eyes that's a misnomer 99% of the time. It might be a good idea to split the wording thoroughly and officially (into "pinned" vs. "pin"/"pinning …"), since this is such a common sticking point and seemingly mixing concepts right now. (I.e. "pinned pointers" are (usually) "`Pin`-wrapped" but not "pinned in place". It only becomes more muddled once you have inherently-pinning or "add-on" pins without `Pin` in their type.)
the_gipsy 5y ago
This is supposed to be plain English?

33 points by lukastyrychtr 5y ago

Not to detract from the author's work - it does express the concept in "plain English" as promised - but boy oh boy is Pinning a confusing concept. This is coming from someone who has used rust pretty regularly since 2015. Sure, in simplest terms, I get it - "Pinning is used to guarantee something is never moved", and the utility of this is mostly (exclusively?) for guarantees in concurrency. The moment I see the symbol "!Unpin" my brain goes into a fog. "Pinning means things cannot be moved... but most things are Unpin... which can be unpinned after being pinned... but some things are !Unpin... so they can NOT be UN-pinned... so when they are Pinnned they are REALLY pinned..." It seems to me like "Pin" is a perfectly sound, logical concept, but it was designed by type-system-science nerds, that barely translates to the promise of Rust which is to empower everyone, not just type-system-science nerds, to make safe software.

vvanders 5y ago
I would largely agree, in the cases where I've needed something pinned I just drop to Box'ing to, keeping the box internal and managing the pinning directly. It's annoying because you lose some classes of things that you may want to pin(values on stack for short durations) but I found similar complexities in the API. For instance if you take a pinned Arc(ex: Pin<Arc<T>>) there wasn't a way I could determine to get out the mutable interior pointer(ex: via Arc::get_mut) since those APIs require a specific signature and you would need to un-pin the value to make those calls despite them being by reference(safe from a pinning perspective). Most of Rust is incredibly well designed but so it's pretty jarring that Pinning is so much harder to use. I've found work-arounds for the cases where I've needed either via boxing or dropping to unsafe but would have been nice if it had worked for my use cases.
- cbarrick 5y ago
  > safe from a pinning perspective The reason you can't get &mut T from Pin<Arc<T>> is because the mutable reference lets you move the data in safe Rust, e.g. with std::mem::swap. So getting the &mut T from a Pin<Arc<T>> is not safe from a pinning perspective.
  - vvanders 5y ago
    Fair, but feels like that significantly reduces the utility of pinned types then if for certain values you lose the ability to mutate values when the compiler overwise knows that there is a single mutable reference.
    - Tamschi 5y ago
      There's a hole or oversight in `Arc`'s API in this regard, annoyingly. The function you're looking for would be something like pub fn get_mut_pinned(this: &mut Pin<Arc<T>>) -> Option<Pin<&mut T>> , which for pinning-aware values should give you enough mutability and for `T: Unpin` can give you `Option<&mut T>` through `Option::as_deref_mut`. Unfortunately, I haven't seen any `Arc` implementations that actually provide it, since almost none of the third-party ones are pinning-aware. (My `tiptoe::Arc`'s `get_mut` has this signature, but that's a specialty container with additional requirements for the contained value.) The same goes for `make_mut_pinned`, that's missing too (but to be fair would be much less useful, since pinning and `Clone` don't often mix that well).
      - zozbot234 5y ago
        Have you tried proposing these API's on the internals.rust-lang.org forum or via posting a proposed RFC? It's not clear to me if they're sound in the general case, but if that's the case they can absolutely be added.
        Tamschi 5y ago
        I should. I'm not too familiar with the process so far, but I'll find some time for it. These functions are indeed generally sound (also for `Rc`), as any value that cares would be `!Unpin`, which would still bar access to `&mut T`. `make_mut_pinned` also shouldn't cause too much confusion, as availability of `Clone` would have to be declared explicitly just about everywhere that's relevant. They can be implemented as very thin wrappers around their non-pinning equivalents, with only a few `unsafe` operations to make the types fit.

sfvisser 5y ago

Unfortunately this write-up didn’t help me at all understanding pinning. Alomost feels I need to read the article to the end in order to understand the assumptions at the start. I guess I’m missing some context. - what is ‘trivially moveable’? - why would I want to overload the assignment operator? - what is a move constructor? - why would this make memory safe apis tricky? - what does moving references outside unsafe code mean? - what now we suddenly talk about Pin and Unpin, without explaining them first. I’m so confused

Tamschi 5y ago
"Trivially movable" means anyone can take an instance (that they own) and copy its data (just the plain memory cells) to a new location with a new address, then continue to use that instance at that new location without breaking anything. This is a weaker guarantee than `Copy` because they're not necessarily allowed to use both copies of the data as true instances. They can still delay choosing one, but they can only ever interact with one of them through its API. I'll see if I can add a link to an explanation there. In Rust it's mostly a base assumption and many other languages (C#, Java, JS…) avoid the topic almost entirely, so they all don't explain it very prominently. C++ is much more explicit about it, but unfortunately calls this property a bunch of different names. (I've seen "trivially relocatable" as general term for it, since a "move" seems to be a very specific action in C++.) The intro section turned out a bit C++-y, maybe. You can read up on the terms by clicking any underlined text. I'm not too happy about links not being blue, but I can't change it without being in the Hashnode ambassador program. (Not much spare money means my entire web presence is strung together from free or very inexpensive platform services. Maybe I'll post about that in the future.) You can also skip the "The Problem" section entirely, though. It's not that important overall and serves mostly as on-ramp for users of other languages with manual memory management.
- sfvisser 5y ago
  Thanks, this already helps quite a lot! I came to rust from mostly higher level languages, Haskell/JVM/JS. So there is this continuous battle of Rust solving problems that I’m not necessarily sure what they are and why those problems exist in the first place.

dtolnay 5y ago

> Pins are very often `Unpin`, but this isn't relevant to their function. This idea is what I've seen get a lot of people lost, but it's pretty natural. By analogy, a DANGER HIGH VOLTAGE sign doesn't itself need to be dangerous or high voltage in order to be useful. If your DANGER HIGH VOLTAGE sign (the sign) is dangerous or high voltage, you'd probably want a second DANGER HIGH VOLTAGE sign pointing out this fact about the first sign. The same for Pin: if you're wanting the pinned pointer (the pointer itself) to not be Unpin, your pinned pointer would need to be pinned by some second pinned pointer.

Tamschi 5y ago
This is why I insist on never calling it "pinned pointer" in the post, even if not explicitly. I'm aware the term is used in the standard library docs (for context: `Pin`'s tagline is "A pinned pointer."), but in my eyes that's a misnomer 99% of the time. It might be a good idea to split the wording thoroughly and officially (into "pinned" vs. "pin"/"pinning …"), since this is such a common sticking point and seemingly mixing concepts right now. (I.e. "pinned pointers" are (usually) "`Pin`-wrapped" but not "pinned in place". It only becomes more muddled once you have inherently-pinning or "add-on" pins without `Pin` in their type.)

the_gipsy 5y ago

This is supposed to be plain English?