4 ms·

You can also have the exact opposite problem with RAII, where a resource survives the end of a transaction, because there is still a live reference to it hidden

You can also have the exact opposite problem with RAII, where a resource survives the end of a transaction, because there is still a live reference to it hidden away somewhere (say, due to some debugging code holding on to it). This is a classical liveness vs. safety dualism. "Something good will eventually happen" and "nothing bad will ever happen" are promises whose solutions are often in conflict with one another. The general problem — to make transactional state changes and transactional control flow (i.e. expectations about these state changes) match up precisely — is not easy to solve in the general case, especially once you move on to things that are less trivial than simple resource acquisition/release matching.

jstimpfle 8y ago
That's also a problem with garbage collection, by the way. GC means memory safety, but not necessarily correctness. In fact, it invites sloppiness, and a kind of sloppiness that sits at a more conceptual level, which could be harder to fix.
- Ooveex2C 8y ago
  At least in java the recommended way is to only use GC as safety net for resource management. Print a warning when it's cleaned up due to finalizers (or cleaner-refs since v9) and use scope-based cleaning instead where possible.
hawkice 8y ago
Oddly, Rust's ownership system really does solve these problems, and Non-lexical lifetimes should eliminate accidental scope-broadening. Unless you are doing some mega-schenanigans, an e.g. MutexGuard gets released precisely when you think. Your point about this being difficult to solve in the general case is true, it's just worth pointing out Rust intends to do that hard thing anyway.
- steveklabnik 8y ago
  Non-lexical lifetimes do not affect something that implements Drop, so the MutexGuard will, by default, last till the end of its lexical scope. You can still call drop on it manually to release it earlier, though.
  - hawkice 8y ago
    Huh! Learn something new every day -- so are lifetimes still exactly describing when things get dropped? And anything implementing Drop just doesn't get optimized lifetimes?
    - steveklabnik 8y ago
      > so are lifetimes still exactly describing when things get dropped? It depends on exactly what you mean by this. In a nutshell, "non-lexical lifetimes" means "things go out of scope when after their last use. Drop implies a use at the end of the current lexical scope." Dropping Drop types earlier ("eager drop") was desired, but has significant problems, including "a large body of unsafe code exists in production which relies on knowing when Drop types go out of scope and changing this behavior may cause a ton of unsoundness in existing code."
    - oconnor663 8y ago
      Lifetimes and destructors aren't as directly related as it might feel at first. Destructors of course are a thing that happens at runtime, and sometimes the order in which things get destructed has important observable side effects. (Maybe it prints something, or maybe you care about the order locks are released in. Apart from memory safety questions.) Lifetimes really have no runtime effect at all. They only exist to prove things about the program at compile time. So all the types and function signatures get assembled together, and then a constraint solver gets run over the whole thing. As long as it returns "yes a solution exists", then no ones really cares about the details of the solution. The benefit of non-lexical lifetimes is to weaken the constraints on the system, so that code that used to appear invalid now appears valid. But I believe it will have no effect on any existing code. (There's a Rust compiler reimplementation somewhere that doesn't even check lifetimes, since you can always use the standard compiler in testing.)
- rbehrends 8y ago
  > Oddly, Rust's ownership system really does solve these problems No. Rust's ownership problem solves it for trivial cases, at the cost of making it hard to do other things (such as sharing references past the lifetime of the owner without resorting to Rc<T> or Arc<T>, at which point you don't really have lifetime guarantees anymore). The essential limitation of Rust is that (without resorting to Rc<T> and Arc<T>, which would put you back to square one) it is conceptually limited to the equivalent of reference counting with a maximum reference count of 1. In order to make this work, Rust needs move semantics and the ability to prove that an alias has a lifetime that is a subset of the lifetime of the original object) and may even sometimes have to copy objects, because it can never actually increase the (purely fictitious) reference count after object creation. This inherent limitation makes a lot of things hard (or at least hard to do without copying or explicit reference counting). Structural sharing in general, hash consing, persistent data structures, global and shared caches, cyclic data structures, and so forth. In short, you have the problem with shared references less, because Rust makes it hard to share data in the first place, for better or worse. (Again, unless you resort to reference counting, and then you get the issue back in full force.)
  - steveklabnik 8y ago
    > it is conceptually limited to the equivalent of reference counting with a maximum reference count of 1. This is a thing people say, but I think it's misleading. Reference counting can increase the lifetime of an object, but borrowing cannot. I've seen this really trip up beginners. > This inherent limitation makes a lot of things hard It can make them different, which can be hard, but these things are already hard. And some people think it can make them easier or even better; see Bodil Stokke's work on persistent data structures in Rust.
    - rbehrends 8y ago
      > This is a thing people say, but I think it's misleading. Reference counting can increase the lifetime of an object, but borrowing cannot. I've seen this really trip up beginners. "With a maximum reference count of 1." As the reference count becomes 1 upon object creation, it cannot really be increased further. Hence, only operations that keep the (virtual) reference count at 1 or reduce it to 0 are allowed. My point here is that you inherently cannot do things where you cannot prove that this virtual reference count can be capped at 1.
fjsolwmv 8y ago
Can you give in an example? Keeping a debug reference at end of transaction (via a shared-reference type, since a non-shared RAII reference type could never get into that state) isn't a coding error, it's a design error -- the development intentionally requested contradictory things. That is solved by using weak references if you don't want a debug tool to force an object to stay alive.
- rbehrends 8y ago
  It is not a design error. You are making the mistake of equating reachability of a reference with making it legal to access it. And the problem is that the two often aren't one and the same. As a simple example, you may still want to access a resource after it has been released. Closing a network connection, for example, does not mean that accessing it is invalid. The connection may still have state (such as statistics collected or whether a non-blocking close was clean) that is perfectly legal to access after release (and in fact may only be consistent/observable afterwards). The Eiffel FILE class [1], for example, allows you to call `is_closed` at any time (as well as the various `is_open` functions). This is necessary because `not is_closed` is evaluated as a precondition for many other operations. A more complex example is a resource that is shared by many threads. Whether that resource is valid is often not a question of whether a reference is reachable, but a function of complex distributed state. Sometimes this can be solved by atomic reference counting, but even then atomic reference counting is expensive. [1] https://archive.eiffel.com/products/base/classes/kernel/file.html https://archive.eiffel.com/products/base/classes/kernel/file...

jstimpfle 8y ago

That's also a problem with garbage collection, by the way. GC means memory safety, but not necessarily correctness. In fact, it invites sloppiness, and a kind of sloppiness that sits at a more conceptual level, which could be harder to fix.

Ooveex2C 8y ago
At least in java the recommended way is to only use GC as safety net for resource management. Print a warning when it's cleaned up due to finalizers (or cleaner-refs since v9) and use scope-based cleaning instead where possible.

hawkice 8y ago

Oddly, Rust's ownership system really does solve these problems, and Non-lexical lifetimes should eliminate accidental scope-broadening. Unless you are doing some mega-schenanigans, an e.g. MutexGuard gets released precisely when you think. Your point about this being difficult to solve in the general case is true, it's just worth pointing out Rust intends to do that hard thing anyway.

steveklabnik 8y ago
Non-lexical lifetimes do not affect something that implements Drop, so the MutexGuard will, by default, last till the end of its lexical scope. You can still call drop on it manually to release it earlier, though.
- hawkice 8y ago
  Huh! Learn something new every day -- so are lifetimes still exactly describing when things get dropped? And anything implementing Drop just doesn't get optimized lifetimes?
  - steveklabnik 8y ago
    > so are lifetimes still exactly describing when things get dropped? It depends on exactly what you mean by this. In a nutshell, "non-lexical lifetimes" means "things go out of scope when after their last use. Drop implies a use at the end of the current lexical scope." Dropping Drop types earlier ("eager drop") was desired, but has significant problems, including "a large body of unsafe code exists in production which relies on knowing when Drop types go out of scope and changing this behavior may cause a ton of unsoundness in existing code."
  - oconnor663 8y ago
    Lifetimes and destructors aren't as directly related as it might feel at first. Destructors of course are a thing that happens at runtime, and sometimes the order in which things get destructed has important observable side effects. (Maybe it prints something, or maybe you care about the order locks are released in. Apart from memory safety questions.) Lifetimes really have no runtime effect at all. They only exist to prove things about the program at compile time. So all the types and function signatures get assembled together, and then a constraint solver gets run over the whole thing. As long as it returns "yes a solution exists", then no ones really cares about the details of the solution. The benefit of non-lexical lifetimes is to weaken the constraints on the system, so that code that used to appear invalid now appears valid. But I believe it will have no effect on any existing code. (There's a Rust compiler reimplementation somewhere that doesn't even check lifetimes, since you can always use the standard compiler in testing.)
rbehrends 8y ago
> Oddly, Rust's ownership system really does solve these problems No. Rust's ownership problem solves it for trivial cases, at the cost of making it hard to do other things (such as sharing references past the lifetime of the owner without resorting to Rc<T> or Arc<T>, at which point you don't really have lifetime guarantees anymore). The essential limitation of Rust is that (without resorting to Rc<T> and Arc<T>, which would put you back to square one) it is conceptually limited to the equivalent of reference counting with a maximum reference count of 1. In order to make this work, Rust needs move semantics and the ability to prove that an alias has a lifetime that is a subset of the lifetime of the original object) and may even sometimes have to copy objects, because it can never actually increase the (purely fictitious) reference count after object creation. This inherent limitation makes a lot of things hard (or at least hard to do without copying or explicit reference counting). Structural sharing in general, hash consing, persistent data structures, global and shared caches, cyclic data structures, and so forth. In short, you have the problem with shared references less, because Rust makes it hard to share data in the first place, for better or worse. (Again, unless you resort to reference counting, and then you get the issue back in full force.)
- steveklabnik 8y ago
  > it is conceptually limited to the equivalent of reference counting with a maximum reference count of 1. This is a thing people say, but I think it's misleading. Reference counting can increase the lifetime of an object, but borrowing cannot. I've seen this really trip up beginners. > This inherent limitation makes a lot of things hard It can make them different, which can be hard, but these things are already hard. And some people think it can make them easier or even better; see Bodil Stokke's work on persistent data structures in Rust.
  - rbehrends 8y ago
    > This is a thing people say, but I think it's misleading. Reference counting can increase the lifetime of an object, but borrowing cannot. I've seen this really trip up beginners. "With a maximum reference count of 1." As the reference count becomes 1 upon object creation, it cannot really be increased further. Hence, only operations that keep the (virtual) reference count at 1 or reduce it to 0 are allowed. My point here is that you inherently cannot do things where you cannot prove that this virtual reference count can be capped at 1.

fjsolwmv 8y ago

Can you give in an example? Keeping a debug reference at end of transaction (via a shared-reference type, since a non-shared RAII reference type could never get into that state) isn't a coding error, it's a design error -- the development intentionally requested contradictory things. That is solved by using weak references if you don't want a debug tool to force an object to stay alive.

rbehrends 8y ago
It is not a design error. You are making the mistake of equating reachability of a reference with making it legal to access it. And the problem is that the two often aren't one and the same. As a simple example, you may still want to access a resource after it has been released. Closing a network connection, for example, does not mean that accessing it is invalid. The connection may still have state (such as statistics collected or whether a non-blocking close was clean) that is perfectly legal to access after release (and in fact may only be consistent/observable afterwards). The Eiffel FILE class [1], for example, allows you to call `is_closed` at any time (as well as the various `is_open` functions). This is necessary because `not is_closed` is evaluated as a precondition for many other operations. A more complex example is a resource that is shared by many threads. Whether that resource is valid is often not a question of whether a reference is reachable, but a function of complex distributed state. Sometimes this can be solved by atomic reference counting, but even then atomic reference counting is expensive. [1] https://archive.eiffel.com/products/base/classes/kernel/file.html https://archive.eiffel.com/products/base/classes/kernel/file...