4 ms·

Anything that stops languages from just exposing some functions that solve this exact problem? function blob_contents(blob) ffi.pin(blob) -- ...

Anything that stops languages from just exposing some functions that solve this exact problem? function blob_contents(blob) ffi.pin(blob) -- ... ffi.unpin(blob) end Where pin disables garbage collection of the given object while pin re-enables it, forming the exact region where its lifetime is guaranteed which is apparently what the author needs. It's manual memory management and the code will have to be written carefully if the language has exceptions or other forms of unwinding. It should work though. Moving garbage collectors also have a concept of pinning objects since code can save pointers to them. Seems like the same problem to me.

cmrx64 3y ago
This is often what happens, and this is often what’s fragile. In the blog these are referred to as “lifetime extension”. The code is written as carefully as it ever is and I can confirm the observation that it’s just begging for a segfault or a leak :) Note that finalizers are asynchronous, and there’s an inversion of control/scoping issue with the way you’ve described it.
- kazinator 3y ago
  The pin and unpin could be tied to a reference count in the byte string object that was extracted. When blob's get_data is called to get the byte string, its pin count is bumped up. When the byte string is reclaimed by GC, it bumps down the blob's pin count.
  - cmrx64 3y ago
    I don’t dispute the possibility of using pinning correctly, in practice it’s a source of bugs. Fuzzy and loose ownership regimes just don’t compose well, people are bad at running region checkers in their head and anything beyond the absolute simplest smallest scoped is prone to eventual error.
    - kazinator 3y ago
      I don't think it's difficult, if you're working inside the run-time. The difficulty is getting the behaviors if you're outside of the run-time, writing writing FFI bindings, where you don't have the option of hacking in new ownership behaviors into the target objects, and your FFI may be lacking in expressiveness also. If it's a bad problem for a certain kind of object, and that object is relatively important (lots of people want to use bindings for it), the way to go may be a lower level extension module rather than FFI, or a wrapper library around it which is more amenable to FFI.
- nh2 3y ago
  Haskell's FFI has `withForeignPtr :: ForeignPtr a -> (Ptr a -> IO b) -> IO b` [1]. A ForeignPtr is a GC-managed pointer with an associated finalizer. The finalizer runs when the ForeignPtr gets GC'd. `withForeignPtr` creates a scope (accepting a lambda) in which you can inspect the pointer `(Ptr a -> IO b)`. This works well in practice, so I do not really understand why "among GC implementors, it is a truth universally acknowledged that a program containing finalizers must be in want of a segfault". [1]: https://hackage.haskell.org/package/base-4.19.1.0/docs/Foreign-ForeignPtr.html#v:withForeignPtr https://hackage.haskell.org/package/base-4.19.1.0/docs/Forei...
  - cmrx64 3y ago
    I’m deeply familiar with this technique, have used it plenty, have encountered the perils, and so I do not really understand why you think it works well on practice. It works well in only the case where you have perfectly well scoped small regions that you can model as your lambda. When you need to actually do anything intricate with the lifetime where you want it to escape (probably into a datastructure), the callback won’t cut it and it’s on you to ensure the foreignptr’s lifetime becomes interlinked with the returned b
    - nh2 3y ago
      I don't quite follow the argument: > it’s on you to ensure the foreignptr’s lifetime becomes interlinked with the returned b Yes; the simplest way to do that is to make sure that your data types never contain raw `Ptr`, only `ForeignPtr` -- same as in C++, where seeing `mytype * x` should ring the alarm bells. You could say "but what if I call another FFI function that needs the Ptr as an argument"? In that case, surely the function needs to document if it takes ownership of the pointer. If it doesn't document that, yes, it'll crash; but that's unrelated to finalizers (the "impossibility of composing" of which which is what the post claimed); it would also crash if no finalizers were involved.
- matheusmoreira 3y ago
  The possibility of segfaults is kind of a given though. I mean the whole point of foreign interfaces is to reuse existing C code. The pinning functions just expose the manual C resource management that programmers would have to deal with if they were writing C. You just turn off the automatic resource management for the objects involved so you can do it yourself, running the risk of leaking those resources. The only viable way to escape all this is to rewrite the software in the host language. A worthy goal but I don't see anyone signing up for that herculean task outside the Rust community.
epcoa 3y ago
This is unnecessary. The blob argument binding itself is making the object reachable throughout the function. You can easily test it with collectgarbage('collect') The author's example is simply mistaken about the relatively straightforward semantics of LuaJIT.
- matheusmoreira 3y ago
  Reading your other comments I've realized you're right about that. I'm not very familiar with LuaJIT so I assumed the garbage collection semantics were undefined. That was the impression I got from the article at least.
fweimer 3y ago
In many cases, you do not need to pin the object, only keep it alive. Current Java has a reachability fence for this: https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/ref/Reference.html#reachabilityFence(java.lang.Object) https://docs.oracle.com/en/java/javase/17/docs/api/java.base... I think I saw this first in MLton, which has a touch function for this purpose: http://mlton.org/MLtonFinalizable http://mlton.org/MLtonFinalizable I'm not convince this is particularly hard to use functionality, all things considered. Supporting explicit deallocation in a safe way is much harder, especially if FFI callbacks are involved.
sparkie 3y ago
Scheme has Guardians for this. They're available in Guile[1], and have recently been submitted as an SRFI[2] for standardization. Original proposal is from Kent Dybvig et al in 1993[3] [1]:https://www.gnu.org/software/guile//manual/html_node/Guardians.html https://www.gnu.org/software/guile//manual/html_node/Guardia... [2]:https://srfi.schemers.org/srfi-246/ https://srfi.schemers.org/srfi-246/ [3]:https://www.cs.tufts.edu/comp/250RTS/archive/kent-dybvig/guardians.pdf https://www.cs.tufts.edu/comp/250RTS/archive/kent-dybvig/gua...

cmrx64 3y ago

This is often what happens, and this is often what’s fragile. In the blog these are referred to as “lifetime extension”. The code is written as carefully as it ever is and I can confirm the observation that it’s just begging for a segfault or a leak :) Note that finalizers are asynchronous, and there’s an inversion of control/scoping issue with the way you’ve described it.

kazinator 3y ago
The pin and unpin could be tied to a reference count in the byte string object that was extracted. When blob's get_data is called to get the byte string, its pin count is bumped up. When the byte string is reclaimed by GC, it bumps down the blob's pin count.
- cmrx64 3y ago
  I don’t dispute the possibility of using pinning correctly, in practice it’s a source of bugs. Fuzzy and loose ownership regimes just don’t compose well, people are bad at running region checkers in their head and anything beyond the absolute simplest smallest scoped is prone to eventual error.
  - kazinator 3y ago
    I don't think it's difficult, if you're working inside the run-time. The difficulty is getting the behaviors if you're outside of the run-time, writing writing FFI bindings, where you don't have the option of hacking in new ownership behaviors into the target objects, and your FFI may be lacking in expressiveness also. If it's a bad problem for a certain kind of object, and that object is relatively important (lots of people want to use bindings for it), the way to go may be a lower level extension module rather than FFI, or a wrapper library around it which is more amenable to FFI.
nh2 3y ago
Haskell's FFI has `withForeignPtr :: ForeignPtr a -> (Ptr a -> IO b) -> IO b` [1]. A ForeignPtr is a GC-managed pointer with an associated finalizer. The finalizer runs when the ForeignPtr gets GC'd. `withForeignPtr` creates a scope (accepting a lambda) in which you can inspect the pointer `(Ptr a -> IO b)`. This works well in practice, so I do not really understand why "among GC implementors, it is a truth universally acknowledged that a program containing finalizers must be in want of a segfault". [1]: https://hackage.haskell.org/package/base-4.19.1.0/docs/Foreign-ForeignPtr.html#v:withForeignPtr https://hackage.haskell.org/package/base-4.19.1.0/docs/Forei...
- cmrx64 3y ago
  I’m deeply familiar with this technique, have used it plenty, have encountered the perils, and so I do not really understand why you think it works well on practice. It works well in only the case where you have perfectly well scoped small regions that you can model as your lambda. When you need to actually do anything intricate with the lifetime where you want it to escape (probably into a datastructure), the callback won’t cut it and it’s on you to ensure the foreignptr’s lifetime becomes interlinked with the returned b
  - nh2 3y ago
    I don't quite follow the argument: > it’s on you to ensure the foreignptr’s lifetime becomes interlinked with the returned b Yes; the simplest way to do that is to make sure that your data types never contain raw `Ptr`, only `ForeignPtr` -- same as in C++, where seeing `mytype * x` should ring the alarm bells. You could say "but what if I call another FFI function that needs the Ptr as an argument"? In that case, surely the function needs to document if it takes ownership of the pointer. If it doesn't document that, yes, it'll crash; but that's unrelated to finalizers (the "impossibility of composing" of which which is what the post claimed); it would also crash if no finalizers were involved.
matheusmoreira 3y ago
The possibility of segfaults is kind of a given though. I mean the whole point of foreign interfaces is to reuse existing C code. The pinning functions just expose the manual C resource management that programmers would have to deal with if they were writing C. You just turn off the automatic resource management for the objects involved so you can do it yourself, running the risk of leaking those resources. The only viable way to escape all this is to rewrite the software in the host language. A worthy goal but I don't see anyone signing up for that herculean task outside the Rust community.

epcoa 3y ago

This is unnecessary. The blob argument binding itself is making the object reachable throughout the function. You can easily test it with collectgarbage('collect') The author's example is simply mistaken about the relatively straightforward semantics of LuaJIT.

matheusmoreira 3y ago
Reading your other comments I've realized you're right about that. I'm not very familiar with LuaJIT so I assumed the garbage collection semantics were undefined. That was the impression I got from the article at least.

fweimer 3y ago

In many cases, you do not need to pin the object, only keep it alive. Current Java has a reachability fence for this: https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/ref/Reference.html#reachabilityFence(java.lang.Object) https://docs.oracle.com/en/java/javase/17/docs/api/java.base... I think I saw this first in MLton, which has a touch function for this purpose: http://mlton.org/MLtonFinalizable http://mlton.org/MLtonFinalizable I'm not convince this is particularly hard to use functionality, all things considered. Supporting explicit deallocation in a safe way is much harder, especially if FFI callbacks are involved.

sparkie 3y ago

Scheme has Guardians for this. They're available in Guile[1], and have recently been submitted as an SRFI[2] for standardization. Original proposal is from Kent Dybvig et al in 1993[3] [1]:https://www.gnu.org/software/guile//manual/html_node/Guardians.html https://www.gnu.org/software/guile//manual/html_node/Guardia... [2]:https://srfi.schemers.org/srfi-246/ https://srfi.schemers.org/srfi-246/ [3]:https://www.cs.tufts.edu/comp/250RTS/archive/kent-dybvig/guardians.pdf https://www.cs.tufts.edu/comp/250RTS/archive/kent-dybvig/gua...