4 ms·

This is often what happens, and this is often what’s fragile. In the blog these are referred to as “lifetime extension”. The code is written as carefully as it

by cmrx64 3y ago

This is often what happens, and this is often what’s fragile. In the blog these are referred to as “lifetime extension”. The code is written as carefully as it ever is and I can confirm the observation that it’s just begging for a segfault or a leak :) Note that finalizers are asynchronous, and there’s an inversion of control/scoping issue with the way you’ve described it.

kazinator 3y ago
The pin and unpin could be tied to a reference count in the byte string object that was extracted. When blob's get_data is called to get the byte string, its pin count is bumped up. When the byte string is reclaimed by GC, it bumps down the blob's pin count.
- cmrx64 3y ago
  I don’t dispute the possibility of using pinning correctly, in practice it’s a source of bugs. Fuzzy and loose ownership regimes just don’t compose well, people are bad at running region checkers in their head and anything beyond the absolute simplest smallest scoped is prone to eventual error.
  - kazinator 3y ago
    I don't think it's difficult, if you're working inside the run-time. The difficulty is getting the behaviors if you're outside of the run-time, writing writing FFI bindings, where you don't have the option of hacking in new ownership behaviors into the target objects, and your FFI may be lacking in expressiveness also. If it's a bad problem for a certain kind of object, and that object is relatively important (lots of people want to use bindings for it), the way to go may be a lower level extension module rather than FFI, or a wrapper library around it which is more amenable to FFI.
nh2 3y ago
Haskell's FFI has `withForeignPtr :: ForeignPtr a -> (Ptr a -> IO b) -> IO b` [1]. A ForeignPtr is a GC-managed pointer with an associated finalizer. The finalizer runs when the ForeignPtr gets GC'd. `withForeignPtr` creates a scope (accepting a lambda) in which you can inspect the pointer `(Ptr a -> IO b)`. This works well in practice, so I do not really understand why "among GC implementors, it is a truth universally acknowledged that a program containing finalizers must be in want of a segfault". [1]: https://hackage.haskell.org/package/base-4.19.1.0/docs/Foreign-ForeignPtr.html#v:withForeignPtr https://hackage.haskell.org/package/base-4.19.1.0/docs/Forei...
- cmrx64 3y ago
  I’m deeply familiar with this technique, have used it plenty, have encountered the perils, and so I do not really understand why you think it works well on practice. It works well in only the case where you have perfectly well scoped small regions that you can model as your lambda. When you need to actually do anything intricate with the lifetime where you want it to escape (probably into a datastructure), the callback won’t cut it and it’s on you to ensure the foreignptr’s lifetime becomes interlinked with the returned b
  - nh2 3y ago
    I don't quite follow the argument: > it’s on you to ensure the foreignptr’s lifetime becomes interlinked with the returned b Yes; the simplest way to do that is to make sure that your data types never contain raw `Ptr`, only `ForeignPtr` -- same as in C++, where seeing `mytype * x` should ring the alarm bells. You could say "but what if I call another FFI function that needs the Ptr as an argument"? In that case, surely the function needs to document if it takes ownership of the pointer. If it doesn't document that, yes, it'll crash; but that's unrelated to finalizers (the "impossibility of composing" of which which is what the post claimed); it would also crash if no finalizers were involved.
matheusmoreira 3y ago
The possibility of segfaults is kind of a given though. I mean the whole point of foreign interfaces is to reuse existing C code. The pinning functions just expose the manual C resource management that programmers would have to deal with if they were writing C. You just turn off the automatic resource management for the objects involved so you can do it yourself, running the risk of leaking those resources. The only viable way to escape all this is to rewrite the software in the host language. A worthy goal but I don't see anyone signing up for that herculean task outside the Rust community.

by cmrx64 3y ago

kazinator 3y ago

The pin and unpin could be tied to a reference count in the byte string object that was extracted. When blob's get_data is called to get the byte string, its pin count is bumped up. When the byte string is reclaimed by GC, it bumps down the blob's pin count.

cmrx64 3y ago
I don’t dispute the possibility of using pinning correctly, in practice it’s a source of bugs. Fuzzy and loose ownership regimes just don’t compose well, people are bad at running region checkers in their head and anything beyond the absolute simplest smallest scoped is prone to eventual error.
- kazinator 3y ago
  I don't think it's difficult, if you're working inside the run-time. The difficulty is getting the behaviors if you're outside of the run-time, writing writing FFI bindings, where you don't have the option of hacking in new ownership behaviors into the target objects, and your FFI may be lacking in expressiveness also. If it's a bad problem for a certain kind of object, and that object is relatively important (lots of people want to use bindings for it), the way to go may be a lower level extension module rather than FFI, or a wrapper library around it which is more amenable to FFI.

nh2 3y ago

Haskell's FFI has `withForeignPtr :: ForeignPtr a -> (Ptr a -> IO b) -> IO b` [1]. A ForeignPtr is a GC-managed pointer with an associated finalizer. The finalizer runs when the ForeignPtr gets GC'd. `withForeignPtr` creates a scope (accepting a lambda) in which you can inspect the pointer `(Ptr a -> IO b)`. This works well in practice, so I do not really understand why "among GC implementors, it is a truth universally acknowledged that a program containing finalizers must be in want of a segfault". [1]: https://hackage.haskell.org/package/base-4.19.1.0/docs/Foreign-ForeignPtr.html#v:withForeignPtr https://hackage.haskell.org/package/base-4.19.1.0/docs/Forei...

cmrx64 3y ago
I’m deeply familiar with this technique, have used it plenty, have encountered the perils, and so I do not really understand why you think it works well on practice. It works well in only the case where you have perfectly well scoped small regions that you can model as your lambda. When you need to actually do anything intricate with the lifetime where you want it to escape (probably into a datastructure), the callback won’t cut it and it’s on you to ensure the foreignptr’s lifetime becomes interlinked with the returned b
- nh2 3y ago
  I don't quite follow the argument: > it’s on you to ensure the foreignptr’s lifetime becomes interlinked with the returned b Yes; the simplest way to do that is to make sure that your data types never contain raw `Ptr`, only `ForeignPtr` -- same as in C++, where seeing `mytype * x` should ring the alarm bells. You could say "but what if I call another FFI function that needs the Ptr as an argument"? In that case, surely the function needs to document if it takes ownership of the pointer. If it doesn't document that, yes, it'll crash; but that's unrelated to finalizers (the "impossibility of composing" of which which is what the post claimed); it would also crash if no finalizers were involved.

matheusmoreira 3y ago

The possibility of segfaults is kind of a given though. I mean the whole point of foreign interfaces is to reuse existing C code. The pinning functions just expose the manual C resource management that programmers would have to deal with if they were writing C. You just turn off the automatic resource management for the objects involved so you can do it yourself, running the risk of leaking those resources. The only viable way to escape all this is to rewrite the software in the host language. A worthy goal but I don't see anyone signing up for that herculean task outside the Rust community.