4 ms·

On the impossibility of composing finalizers and FFI

46 points by ingve 3y ago

celeritascelery 3y ago
Given how much research there is into this topic, I am sure that I just don't understand the complexity of it. But to me it seems like you could have a function that associates one value with another in the GC. Something like `gc_borrows_from`. You would than write the problematic code like this: function blob_contents(blob) local len_out = ffi.new('unsigned int') local contents = gc_borrows_from(blob, hb.hb_blob_get_data(blob, len_out)) local len = len_out[0]; return ffi.string(contents, len) end This would tell the GC that the data returned by `hb_blob_get_data` is borrowed from blob, and it can't collect `blob` until `contents` is also unreachable. How to implement that would be up to the runtime, but it seems reasonable to have a wrapper type that holds a traceable reference back to blob.
- kevingadd 3y ago
  .NET calls this concept a 'dependent handle' if I understand you correctly, though allowing direct use of it is somewhat new: https://learn.microsoft.com/en-us/dotnet/api/system.runtime.dependenthandle https://learn.microsoft.com/en-us/dotnet/api/system.runtime.... And there's an older API that wraps them which I've found quite handy: https://learn.microsoft.com/en-us/dotnet/api/system.runtime.compilerservices.conditionalweaktable-2 https://learn.microsoft.com/en-us/dotnet/api/system.runtime.... Essentially, if you want to (externally - in code you wrote) associate two objects with each other without being able to modify the code of either of those objects, dependent handles are the ticket. It only creates a one-way liveness relationship, though, which may not be sufficient for every use case...
- planede 3y ago
  In C++ that would be an "aliasing shared_ptr", for a reference counting example of the same idea.
- pizlonator 3y ago
  Cute, but not a complete solution. Eventually you’ll want an object graph cycle that traverses through the non-GC heap and then you’re really screwed.
matheusmoreira 3y ago
Anything that stops languages from just exposing some functions that solve this exact problem? function blob_contents(blob) ffi.pin(blob) -- ... ffi.unpin(blob) end Where pin disables garbage collection of the given object while pin re-enables it, forming the exact region where its lifetime is guaranteed which is apparently what the author needs. It's manual memory management and the code will have to be written carefully if the language has exceptions or other forms of unwinding. It should work though. Moving garbage collectors also have a concept of pinning objects since code can save pointers to them. Seems like the same problem to me.
- cmrx64 3y ago
  This is often what happens, and this is often what’s fragile. In the blog these are referred to as “lifetime extension”. The code is written as carefully as it ever is and I can confirm the observation that it’s just begging for a segfault or a leak :) Note that finalizers are asynchronous, and there’s an inversion of control/scoping issue with the way you’ve described it.
  - kazinator 3y ago
    The pin and unpin could be tied to a reference count in the byte string object that was extracted. When blob's get_data is called to get the byte string, its pin count is bumped up. When the byte string is reclaimed by GC, it bumps down the blob's pin count.
    - cmrx64 3y ago
      I don’t dispute the possibility of using pinning correctly, in practice it’s a source of bugs. Fuzzy and loose ownership regimes just don’t compose well, people are bad at running region checkers in their head and anything beyond the absolute simplest smallest scoped is prone to eventual error.
      - kazinator 3y ago
        I don't think it's difficult, if you're working inside the run-time. The difficulty is getting the behaviors if you're outside of the run-time, writing writing FFI bindings, where you don't have the option of hacking in new ownership behaviors into the target objects, and your FFI may be lacking in expressiveness also. If it's a bad problem for a certain kind of object, and that object is relatively important (lots of people want to use bindings for it), the way to go may be a lower level extension module rather than FFI, or a wrapper library around it which is more amenable to FFI.
kazinator 3y ago
> want to copy out its contents as a byte string. That object should take care of it. Even if the parent object is reclaimed, the byte string should independently persist for as long as is necessary. Since byte strings don't contain pointers to anything, reference counting could be used. The refcount could be in the parent blob, such that when its nonzero, the blob is pinned against reclamation. Of course you can't just share out the internals of an object, such that GC doesn't know about them. It doesn't matter if it's a foreign object set up with FFI or something built into the run time.
- HelloNurse 3y ago
  Copying the data and letting the FFI data structures go is the only way to get correct behaviour without inconveniently (or impossibly) pinning objects to elude garbage collection.
adgjlsfhk1 3y ago
Julia solves this pretty well by having a gc_preserve macro that tells the compiler that an object has hidden references within the scope.
- ng55QPSK 3y ago
  which means: you tell the compiler
  - adgjlsfhk1 3y ago
    but as long as you are telling the compiler through a first class API, it's not brittle because there compiler knows the meaning and intent of the statement, and therefore can avoid optimizing it in invalid ways.
dwattttt 3y ago
This is the essence of Rusts lifetime analysis; a pointer to an object can't be live for longer than the object itself is. In this particular example, you'd make an object with a finalizer and hide the raw pointer inside of it. Then you can only touch that pointer by going through a Rust object which participates in lifetime analysis, and it'll clean it up when it's done. Any more attempts to touch that object/pointer will fail to compile. Expressed that way it makes sense that some people call it "GC at compile time".
- arcticbull 3y ago
  I like using this neat newtype for wrapping FFI objects. pub struct OwnedForeignInstance(NonNull<std::ffi::c_void>); You can then, as you say, implement Drop to give up the foreign object when it's no longer needed. impl Drop for OwnedForeignInstance { fn drop(&mut self) { unsafe { /* Destroy self.0.as_ptr() */ } } } You can define an un-owned reference to the owned foreign instance using a newtype like this. pub struct UnownedReference(PhantomData<UnsafeCell<*mut ()>>); Then, you can hand out zero-cost lifetime-checked references to the owned foreign instance like this. impl Deref for OwnedForeignInstance { type Target = UnownedReference; fn deref(&self) -> &Self::Target { unsafe { &*(self.0.as_ptr() as *mut _) } } } impl DerefMut for OwnedForeignInstance { fn deref_mut(&mut self) -> &mut Self::Target { unsafe { &mut *(self.0.as_ptr() as *mut _) } } } Once you've done that, you expose your FFI functionality on UnownedReference, relying on auto-deref. Unless it consumes the receiver, in which case you put it on the OwnedForeignInstance. This way you can't destroy the object while references to it continue to exist. It's not perfect, but it's the best way I've found so far for making FFI wrapper objects that look and feel like Rust objects while respecting the FFI contract.
lmm 3y ago
Managing FFI will always require cooperation with the GC if there is one. If the GC doesn't expose adequate APIs for doing that cooperation, that feels like more of a design problem with that GC than a fact of nature. You shouldn't be trying to "trick" the compiler/runtime to keep your thing live until you've finished using it: you should tell it when you need to keep it live until, and it should listen to you.
- nine_k 3y ago
  In other words, managing FFI resources will remain "manual" or otherwise deterministic. The mechanism will cooperate with GC by releasing an object to be collected once its interaction with FFI is done. In absence of finalizers, I suspect, FFI resources could also be garbage-collected. If looks like traditional GC-based languages (like Java or Lisp) are hurt by absence of static data flow analysis, which would guarantee that a finalizer cannot revive the object being collected (e.g. by creating a new live reference to it elsewhere). Finalizers can likely be made safe enough if their code is more restricted; that would still allow many reasonable finalizers that calmly release external resources.
  - Someone 3y ago
    > Finalizers can likely be made safe enough if their code is more restricted; that would still allow many reasonable finalizers that calmly release external resources. If a finalizer calls external code to release external resources (a not uncommon use case), there’s no way static data flow analysis can determine that external code doesn’t make a call back into the VM that revives objects, is there?
  - lmm 3y ago
    An FFI resource of this kind needs to be finalized via the FFI, almost by definition. So the problem isn't whether you can do data flow analysis in the host language, it's whether you can do sufficient analysis of the language that you're embedding (assuming what you're embedding isn't an opaque call to some library for which you only have an ABI, which is the usual way to do FFI).
- zozbot234 3y ago
epcoa 3y ago
The LuaJIT example isn't correct though, the lifetime of garbage collected objects is clearly documented: https://luajit.org/ext_ffi_semantics.html#gc https://luajit.org/ext_ffi_semantics.html#gc In the example function `blob` will not be collected because it is reachable from the `blob` argument local variable (IOW it is on the Lua stack). `ffi.string`() copies the string data into a new Lua string, and the lifetime of blob is guaranteed until the return of the function. So not sure what the issue is. function blob_contents(blob) -- <- this ensures liveness until past return local len_out = ffi.new('unsigned int') local contents = hb.hb_blob_get_data(blob, len_out) local len = len_out[0]; return ffi.string(contents, len) end
- lmm 3y ago
  The argument is that the JIT might realise that blob is never used beyond that line, and collect it early. In general that would be a desirable feature.
  - epcoa 3y ago
    I know it says this: "The semantics of LuaJIT do not prescribe when GC can happen and what values will be live, so the GC and the compiler are not constrained to extend the liveness of blob to, say, the entirety of its lexical scope. " But it is flat wrong. From the LuaJIT documentation: "All explicitly (ffi.new(), ffi.cast() etc.) or implicitly (accessors) created cdata objects are garbage collected. You need to ensure to retain valid references to cdata objects somewhere on a Lua stack, an upvalue or in a Lua table while they are still in use. Once the last reference to a cdata object is gone, the garbage collector will automatically free the memory used by it (at the end of the next GC cycle)." The Lua stack in this case includes all the local variables in that function scope. It's a non-issue/straw man and is common sense. If LuaJIT FFI worked the way the author supposed, it would be near impossible to use practically. “It is perfectly valid to collect blob after its last use” This is a useless statement. It’s perfectly “valid” for LuaJIT to not even read your source code and exit immediately, but that isn’t what it does because it would be useless. What counts as a reference is both PUC Lua and LuaJIT is defined. As far as the desirability of finer grained liveness, Lua has block scope (do end), but in practice LuaJIT does well inlining so functions ought to be short anyway.
01HNNWZ0MV43FF 3y ago
When I see all the trouble that async Rust and normal C# have had with finalizers, I must wonder if anything composes with anything else, or it's all just banging rocks and praying
LispSporks22 3y ago
I think the SDL2 bindings for Common Lisp went through a similar realization and any kind of finalization was removed.
- phoe-krk 3y ago
  Yes. Yet another problem with finalizers is the question of "which thread does actually run the finalizers"? In case of some C libraries, objects must be released exactly by the thread in which they were allocated; this means that e.g. in Steel Bank Common Lisp it is impossible to use finalizers in these cases.
light_hue_1 3y ago
In Haskell you would write a newtype that keeps a pointer back to blob along with the data that's being returned. This makes the result perfectly correct. There's nothing impossible here. You could even write yourself a small function to access the blob that ensures the results are always wrapped this way.
ptx 3y ago
If the problem with the second work-around (the reason it's not satisfactory) is that it's not supported as part of the platform, forcing you to use a "trick" to "outsmart the compiler", then C# has this solved: System.GC.KeepAlive [1] is an official part of the .NET platform and documented to do exactly this, so presumably Microsoft would not break it when making changes to the GC. [1] https://learn.microsoft.com/en-us/dotnet/api/system.gc.keepalive?view=net-8.0 https://learn.microsoft.com/en-us/dotnet/api/system.gc.keepa...
jerf 3y ago
While people spend much more time arguing matters of taste and trading urban legends about performance back and forth when discussing garbage collection, finalizers may be the most potent issue that garbage collection raises, at least from an overall correctness perspective. As near as I can tell, it has proved an intractable problem to get perfectly correct finalization with garbage collection. Many, many implementations have tried it, and all the ones I've ever seen have either ended up with a laundry list of warnings around them along with a warning that the list of warning itself may still be incomplete, or they pull them out entirely. It may be worth reading that previous sentence again. I've seen it so many times. Even in the comments on the linked article I see people trying to propose solutions to the problem. But it turns out to be one of those problems that looks simple, even super simple ("just" provide a function to do it), but then it turns out that hundreds of smart people have poured years into solving it... and failed, as far as I know, universally. Fortunately, you need to have a very particular kind of program to get hit hard by the problem, and an even more particular kind of program to not have the ability to mitigate it into an acceptable issue somehow, most often something like being more careful to close files when done with them. I think I brushed the issue once and it was addressed by exactly that, being more careful to close files when done with them because we accidentally were leaning on GC to close them and that become a de facto file handle leak. That said, the problem may be exposed by GC but it isn't entirely caused by it. Sufficiently complicated manually-managed memory programs can encounter things that stem from the same underlying issue. If the lifetime of objects that need a "finalizer" (or destructor, if you prefer) becomes sufficiently complicated, it can also become very difficult to figure out when they need to be cleaned up. But you get more runway, and you have more options to deal with it when you have the ability to more forcibly descope things and deallocate them at a known point in time. When I was programming with libpurple, the generalized IM library written in C, it used a lot of "closures" (implemented in C), and figuring out when to correctly deallocate and finalize on them was very, very difficult, because their lifetimes were complicated and not terribly well documented. This is the same underlying problem in a lot of ways, it just manifests differently. But in a manually-managed world, it is at least a solvable problem, however hard the problem may be to solve in some languages (while they had no better option at the time, I gotta say in hindsight libpurple would really have been served by Rust, and I'm not thinking the generalized "rewrite everything in Rust" here but specifically that all those closures would have been much safer to work with with lifetime annotations enforced by the compiler), and however difficult you may make the problem on yourself with your own programming practices. Alas, no silver bullet.
pizlonator 3y ago
This is my favorite problem! So cool to see it written up. The only nice way to have a GC language interop with another language is if that other language is also GC’d and that language’s GC interops with yours (what I like to call frankengc). (Once I add GC to Fil-C and give it proper frankengc hooks, you’ll be able to do destructorless FFI to whatever C code Fil-C can run. It’ll be great I promise.)

46 points by ingve 3y ago

Given how much research there is into this topic, I am sure that I just don't understand the complexity of it. But to me it seems like you could have a function that associates one value with another in the GC. Something like `gc_borrows_from`. You would than write the problematic code like this: function blob_contents(blob) local len_out = ffi.new('unsigned int') local contents = gc_borrows_from(blob, hb.hb_blob_get_data(blob, len_out)) local len = len_out[0]; return ffi.string(contents, len) end This would tell the GC that the data returned by `hb_blob_get_data` is borrowed from blob, and it can't collect `blob` until `contents` is also unreachable. How to implement that would be up to the runtime, but it seems reasonable to have a wrapper type that holds a traceable reference back to blob.

kevingadd 3y ago
.NET calls this concept a 'dependent handle' if I understand you correctly, though allowing direct use of it is somewhat new: https://learn.microsoft.com/en-us/dotnet/api/system.runtime.dependenthandle https://learn.microsoft.com/en-us/dotnet/api/system.runtime.... And there's an older API that wraps them which I've found quite handy: https://learn.microsoft.com/en-us/dotnet/api/system.runtime.compilerservices.conditionalweaktable-2 https://learn.microsoft.com/en-us/dotnet/api/system.runtime.... Essentially, if you want to (externally - in code you wrote) associate two objects with each other without being able to modify the code of either of those objects, dependent handles are the ticket. It only creates a one-way liveness relationship, though, which may not be sufficient for every use case...
planede 3y ago
In C++ that would be an "aliasing shared_ptr", for a reference counting example of the same idea.
pizlonator 3y ago
Cute, but not a complete solution. Eventually you’ll want an object graph cycle that traverses through the non-GC heap and then you’re really screwed.

matheusmoreira 3y ago

Anything that stops languages from just exposing some functions that solve this exact problem? function blob_contents(blob) ffi.pin(blob) -- ... ffi.unpin(blob) end Where pin disables garbage collection of the given object while pin re-enables it, forming the exact region where its lifetime is guaranteed which is apparently what the author needs. It's manual memory management and the code will have to be written carefully if the language has exceptions or other forms of unwinding. It should work though. Moving garbage collectors also have a concept of pinning objects since code can save pointers to them. Seems like the same problem to me.

cmrx64 3y ago
This is often what happens, and this is often what’s fragile. In the blog these are referred to as “lifetime extension”. The code is written as carefully as it ever is and I can confirm the observation that it’s just begging for a segfault or a leak :) Note that finalizers are asynchronous, and there’s an inversion of control/scoping issue with the way you’ve described it.
- kazinator 3y ago
  The pin and unpin could be tied to a reference count in the byte string object that was extracted. When blob's get_data is called to get the byte string, its pin count is bumped up. When the byte string is reclaimed by GC, it bumps down the blob's pin count.
  - cmrx64 3y ago
    I don’t dispute the possibility of using pinning correctly, in practice it’s a source of bugs. Fuzzy and loose ownership regimes just don’t compose well, people are bad at running region checkers in their head and anything beyond the absolute simplest smallest scoped is prone to eventual error.
    - kazinator 3y ago
      I don't think it's difficult, if you're working inside the run-time. The difficulty is getting the behaviors if you're outside of the run-time, writing writing FFI bindings, where you don't have the option of hacking in new ownership behaviors into the target objects, and your FFI may be lacking in expressiveness also. If it's a bad problem for a certain kind of object, and that object is relatively important (lots of people want to use bindings for it), the way to go may be a lower level extension module rather than FFI, or a wrapper library around it which is more amenable to FFI.

kazinator 3y ago

> want to copy out its contents as a byte string. That object should take care of it. Even if the parent object is reclaimed, the byte string should independently persist for as long as is necessary. Since byte strings don't contain pointers to anything, reference counting could be used. The refcount could be in the parent blob, such that when its nonzero, the blob is pinned against reclamation. Of course you can't just share out the internals of an object, such that GC doesn't know about them. It doesn't matter if it's a foreign object set up with FFI or something built into the run time.

HelloNurse 3y ago
Copying the data and letting the FFI data structures go is the only way to get correct behaviour without inconveniently (or impossibly) pinning objects to elude garbage collection.

adgjlsfhk1 3y ago

Julia solves this pretty well by having a gc_preserve macro that tells the compiler that an object has hidden references within the scope.

ng55QPSK 3y ago
which means: you tell the compiler
- adgjlsfhk1 3y ago
  but as long as you are telling the compiler through a first class API, it's not brittle because there compiler knows the meaning and intent of the statement, and therefore can avoid optimizing it in invalid ways.

dwattttt 3y ago

This is the essence of Rusts lifetime analysis; a pointer to an object can't be live for longer than the object itself is. In this particular example, you'd make an object with a finalizer and hide the raw pointer inside of it. Then you can only touch that pointer by going through a Rust object which participates in lifetime analysis, and it'll clean it up when it's done. Any more attempts to touch that object/pointer will fail to compile. Expressed that way it makes sense that some people call it "GC at compile time".

arcticbull 3y ago
I like using this neat newtype for wrapping FFI objects. pub struct OwnedForeignInstance(NonNull<std::ffi::c_void>); You can then, as you say, implement Drop to give up the foreign object when it's no longer needed. impl Drop for OwnedForeignInstance { fn drop(&mut self) { unsafe { /* Destroy self.0.as_ptr() */ } } } You can define an un-owned reference to the owned foreign instance using a newtype like this. pub struct UnownedReference(PhantomData<UnsafeCell<*mut ()>>); Then, you can hand out zero-cost lifetime-checked references to the owned foreign instance like this. impl Deref for OwnedForeignInstance { type Target = UnownedReference; fn deref(&self) -> &Self::Target { unsafe { &*(self.0.as_ptr() as *mut _) } } } impl DerefMut for OwnedForeignInstance { fn deref_mut(&mut self) -> &mut Self::Target { unsafe { &mut *(self.0.as_ptr() as *mut _) } } } Once you've done that, you expose your FFI functionality on UnownedReference, relying on auto-deref. Unless it consumes the receiver, in which case you put it on the OwnedForeignInstance. This way you can't destroy the object while references to it continue to exist. It's not perfect, but it's the best way I've found so far for making FFI wrapper objects that look and feel like Rust objects while respecting the FFI contract.

lmm 3y ago

Managing FFI will always require cooperation with the GC if there is one. If the GC doesn't expose adequate APIs for doing that cooperation, that feels like more of a design problem with that GC than a fact of nature. You shouldn't be trying to "trick" the compiler/runtime to keep your thing live until you've finished using it: you should tell it when you need to keep it live until, and it should listen to you.

nine_k 3y ago
In other words, managing FFI resources will remain "manual" or otherwise deterministic. The mechanism will cooperate with GC by releasing an object to be collected once its interaction with FFI is done. In absence of finalizers, I suspect, FFI resources could also be garbage-collected. If looks like traditional GC-based languages (like Java or Lisp) are hurt by absence of static data flow analysis, which would guarantee that a finalizer cannot revive the object being collected (e.g. by creating a new live reference to it elsewhere). Finalizers can likely be made safe enough if their code is more restricted; that would still allow many reasonable finalizers that calmly release external resources.
- Someone 3y ago
  > Finalizers can likely be made safe enough if their code is more restricted; that would still allow many reasonable finalizers that calmly release external resources. If a finalizer calls external code to release external resources (a not uncommon use case), there’s no way static data flow analysis can determine that external code doesn’t make a call back into the VM that revives objects, is there?
- lmm 3y ago
  An FFI resource of this kind needs to be finalized via the FFI, almost by definition. So the problem isn't whether you can do data flow analysis in the host language, it's whether you can do sufficient analysis of the language that you're embedding (assuming what you're embedding isn't an opaque call to some library for which you only have an ABI, which is the usual way to do FFI).
zozbot234 3y ago

epcoa 3y ago

The LuaJIT example isn't correct though, the lifetime of garbage collected objects is clearly documented: https://luajit.org/ext_ffi_semantics.html#gc https://luajit.org/ext_ffi_semantics.html#gc In the example function `blob` will not be collected because it is reachable from the `blob` argument local variable (IOW it is on the Lua stack). `ffi.string`() copies the string data into a new Lua string, and the lifetime of blob is guaranteed until the return of the function. So not sure what the issue is. function blob_contents(blob) -- <- this ensures liveness until past return local len_out = ffi.new('unsigned int') local contents = hb.hb_blob_get_data(blob, len_out) local len = len_out[0]; return ffi.string(contents, len) end

lmm 3y ago
The argument is that the JIT might realise that blob is never used beyond that line, and collect it early. In general that would be a desirable feature.
- epcoa 3y ago
  I know it says this: "The semantics of LuaJIT do not prescribe when GC can happen and what values will be live, so the GC and the compiler are not constrained to extend the liveness of blob to, say, the entirety of its lexical scope. " But it is flat wrong. From the LuaJIT documentation: "All explicitly (ffi.new(), ffi.cast() etc.) or implicitly (accessors) created cdata objects are garbage collected. You need to ensure to retain valid references to cdata objects somewhere on a Lua stack, an upvalue or in a Lua table while they are still in use. Once the last reference to a cdata object is gone, the garbage collector will automatically free the memory used by it (at the end of the next GC cycle)." The Lua stack in this case includes all the local variables in that function scope. It's a non-issue/straw man and is common sense. If LuaJIT FFI worked the way the author supposed, it would be near impossible to use practically. “It is perfectly valid to collect blob after its last use” This is a useless statement. It’s perfectly “valid” for LuaJIT to not even read your source code and exit immediately, but that isn’t what it does because it would be useless. What counts as a reference is both PUC Lua and LuaJIT is defined. As far as the desirability of finer grained liveness, Lua has block scope (do end), but in practice LuaJIT does well inlining so functions ought to be short anyway.

01HNNWZ0MV43FF 3y ago

When I see all the trouble that async Rust and normal C# have had with finalizers, I must wonder if anything composes with anything else, or it's all just banging rocks and praying

LispSporks22 3y ago

I think the SDL2 bindings for Common Lisp went through a similar realization and any kind of finalization was removed.

phoe-krk 3y ago
Yes. Yet another problem with finalizers is the question of "which thread does actually run the finalizers"? In case of some C libraries, objects must be released exactly by the thread in which they were allocated; this means that e.g. in Steel Bank Common Lisp it is impossible to use finalizers in these cases.

light_hue_1 3y ago

In Haskell you would write a newtype that keeps a pointer back to blob along with the data that's being returned. This makes the result perfectly correct. There's nothing impossible here. You could even write yourself a small function to access the blob that ensures the results are always wrapped this way.

ptx 3y ago

If the problem with the second work-around (the reason it's not satisfactory) is that it's not supported as part of the platform, forcing you to use a "trick" to "outsmart the compiler", then C# has this solved: System.GC.KeepAlive [1] is an official part of the .NET platform and documented to do exactly this, so presumably Microsoft would not break it when making changes to the GC. [1] https://learn.microsoft.com/en-us/dotnet/api/system.gc.keepalive?view=net-8.0 https://learn.microsoft.com/en-us/dotnet/api/system.gc.keepa...

jerf 3y ago

While people spend much more time arguing matters of taste and trading urban legends about performance back and forth when discussing garbage collection, finalizers may be the most potent issue that garbage collection raises, at least from an overall correctness perspective. As near as I can tell, it has proved an intractable problem to get perfectly correct finalization with garbage collection. Many, many implementations have tried it, and all the ones I've ever seen have either ended up with a laundry list of warnings around them along with a warning that the list of warning itself may still be incomplete, or they pull them out entirely. It may be worth reading that previous sentence again. I've seen it so many times. Even in the comments on the linked article I see people trying to propose solutions to the problem. But it turns out to be one of those problems that looks simple, even super simple ("just" provide a function to do it), but then it turns out that hundreds of smart people have poured years into solving it... and failed, as far as I know, universally. Fortunately, you need to have a very particular kind of program to get hit hard by the problem, and an even more particular kind of program to not have the ability to mitigate it into an acceptable issue somehow, most often something like being more careful to close files when done with them. I think I brushed the issue once and it was addressed by exactly that, being more careful to close files when done with them because we accidentally were leaning on GC to close them and that become a de facto file handle leak. That said, the problem may be exposed by GC but it isn't entirely caused by it. Sufficiently complicated manually-managed memory programs can encounter things that stem from the same underlying issue. If the lifetime of objects that need a "finalizer" (or destructor, if you prefer) becomes sufficiently complicated, it can also become very difficult to figure out when they need to be cleaned up. But you get more runway, and you have more options to deal with it when you have the ability to more forcibly descope things and deallocate them at a known point in time. When I was programming with libpurple, the generalized IM library written in C, it used a lot of "closures" (implemented in C), and figuring out when to correctly deallocate and finalize on them was very, very difficult, because their lifetimes were complicated and not terribly well documented. This is the same underlying problem in a lot of ways, it just manifests differently. But in a manually-managed world, it is at least a solvable problem, however hard the problem may be to solve in some languages (while they had no better option at the time, I gotta say in hindsight libpurple would really have been served by Rust, and I'm not thinking the generalized "rewrite everything in Rust" here but specifically that all those closures would have been much safer to work with with lifetime annotations enforced by the compiler), and however difficult you may make the problem on yourself with your own programming practices. Alas, no silver bullet.

pizlonator 3y ago

This is my favorite problem! So cool to see it written up. The only nice way to have a GC language interop with another language is if that other language is also GC’d and that language’s GC interops with yours (what I like to call frankengc). (Once I add GC to Fil-C and give it proper frankengc hooks, you’ll be able to do destructorless FFI to whatever C code Fil-C can run. It’ll be great I promise.)