10 ms·

For me the biggest benefit is deterministic destruction, specifically for the non-memory resources it is holding (files, locks, etc.) the same as if it was stac

For me the biggest benefit is deterministic destruction, specifically for the non-memory resources it is holding (files, locks, etc.) the same as if it was stack allocated. It is very freeing knowing that is just being taken care of. No 'defer', 'with' or other special handling needed. But possibly the bigger benefit: no heap allocation for many objects at all. If you can prove single ownership, you can stack allocate it (or "inline it" if in a struct/object).

shakna 4y ago
> No 'defer', 'with' or other special handling needed. Just as an aside - in Python, it surprised me that 'with' is _not_ a destructor. The object created will remain in-scope after the 'with' body, but the '__exit__' method will be called. Which for the canonical example of file-handling, only really closes the file.
- jnwatson 4y ago
  The done vs destroyed dichotomy is a cross-cutting concern for programs that need explicit control over resource lifetimes. RAII is actually an antipattern in Python if you actually care about resources being released.
  - shakna 4y ago
    The surprise is less about resource usage and more about the scope lifetime. 'with' introduces a new binding that outlives its body. It looks, at first glance, to be a temporary variable. But it isn't.
    - pphysch 4y ago
      Consider the common pattern: with open(path) as file: result = func(file.read()) another_func(result) If `with` used a separate scope then it would be a lot less ergonomic. How might we bring data outside that scope and be confident that `with` isn't forcibly cleaning things up? Would be a lot more verbose.
      - shakna 4y ago
        I think you've missed what I've meant by a new binding. That the body executes is entirely intuitive. However: with open(path) as file: ... if file.mode == 'r': print("Huh? file is still a thing?")
        xscott 4y ago
        Python doesn't have block scope, but most people don't seem to notice: def foo(): if False: x = 123 print("x is visible out here:", x) That will generate an error because `x` is "referenced before assignment", but the variable exists for the entire function. This can bite you in subtle ways: x = "is global out here" def foo(): print("all good:", x) def bar(): print("not good:", x) if False: x = "now x is local, even before this declaration"
    - chombier 4y ago
      I always wondered why does this variable leak outside the `with` scope, because this is really the one place where it should not. IIRC python3 fixed scope leaks in list comprehensions, does anyone know why they didn't fix this one as well? Or does it have actual uses in practice?
      - shakna 4y ago
        I don't know the _motivation_ for why it leaks, but it leaks because the method calls of a context manager are __enter__ and __exit__, but not __del__. [0] The motivation is especially opaque, because though CPython does run __exit__ immediately, it isn't guaranteed to do so by the standard. So __exit__ will happen at some point, like __del__ with the GC, but it won't actually call __del__, which makes GC'ing the value more complicated as __del__ can't safely run until __exit__ does. I _think_ the motivation is something to do with __exit__ being allowed to throw an exception that can be caught, whereas __del__ isn't allowed to do that, but I don't find it that convincing. And I've never seen an example where the introduced variable is ever referenced afterwards in Python's documentation (including in the PEP itself). [0] https://peps.python.org/pep-0343/ https://peps.python.org/pep-0343/
        chombier 4y ago
        Thinking about it, my guess is that properly scoping the variable does not really prevent the user from leaking the value anyway: with context() as foo: leak = foo This is unlike list comprehensions, where it is syntactically impossible to bind the comprehension variable to a new name accessible outside the comprehension. So maybe the reason is that it is not really worth the trouble.
    - ptx 4y ago
      Everything is always* scoped to the entire function in Python. Perhaps that's not always ideal, but at least the rule is simple. * Well, not variables in generator expressions. There might be a few other exceptions.
SemanticStrengh 4y ago
What does it bring vs Java autocloseable?
- nu11ptr 4y ago
  Java's "try with resources" is effectively the same as Go's 'defer' and Python's 'with', so the advantage is that you can't forget to do it. No matter what you do, when the object goes out of scope, the resource will be closed.
  - pjmlp 4y ago
    If you actually use static analysis, you can't forget it in Java either. And it isn't as if using languages without static analysis is still considered a best practice in the 21st century.
    - tialaramex 4y ago
      > And it isn't as if using languages without static analysis is still considered a best practice in the 21st century. So in summary, "Please just do better". Not helpful as a video game meme, and really no better for programming.
      - pjmlp 4y ago
        I guess that is why no one bothers with clippy then, it is superfluous.
        tialaramex 4y ago
        Everywhere that a clippy lint is there to catch bugs, I wish it wasn't a clippy lint because it won't always get run and those bugs will get missed. An example of a clippy lint that I like in clippy: If you try to transmute() &[u8] into a &str clippy will point out that you should probably from_utf8_unchecked() -- this lint results in code that does the same thing but is clearer for maintainers about what we intended. If our reaction to the lint is "But it isn't UTF-8" then we just got an important red flag, but that's not why the lint is there. On the other hand, clippy lints for truncating casts and I want those gone from the language so the warnings ought to at least live in the compiler not clippy. I would like to see a future Edition outlaw these casts entirely so that you need to write a truncating cast explicitly if that's what you needed.
oaiey 4y ago
I would argue that resources need always explicit handling, no matter whether you do reference counting or GC. So if I do a one-liner to dispose an object or call delete on an object is the same beauty. Wait till the variable is out of scope together with many more or hit the GC cleanup is just not the right way to clean up externally resources in a proper way.
xxs 4y ago
>the non-memory resources it is holding (files, locks, etc.) Locks are entirely a memory construct. You can trivially write one, e.g. Java locks are written entirely in java. The files descriptors (incl. socket) - they should not be relied to be GC'd by default, they do require their own lifecycle. For instance: c# Disposable/ java's try with resource. >no heap allocation for many objects at all. This has been already the case for many a year - reference escape analysis, ref. escape elission, trace compilation.
- tialaramex 4y ago
  Locks are, in fact, one of many resources your operating system can give you. Historically, in concurrent software your only option was to either use these OS allocated locks or build a spin lock from atomic operations (and thus burn CPU every time you're waiting for a lock). The OS allocated locks, such as from sem_open() or create_sem() must later be properly released or you're leaking a finite OS resource that will only be reclaimed on process exit or in some cases only at reboot. Today your operating system probably offers a lightweight "futex" style locking primitive, in which so long as it's not locked no OS resources are reserved it's just memory. But, you're not guaranteed that all your locks are these lightweight structures and even if you were you need to clean up properly because when they are locked the OS is using resources so you need to unlock properly or you'll blow up.
  - xxs 4y ago
    The only part needed to build a lock is to be able to 'sleep' the current thread and the latter to be able to be awaken, plus compare-and-swap or load linked/store conditional. There is nothing really special about the locks as long as the current thread can be made dormant. Technically a single OS lock (e.g. a futex), associated with the current thread would suffice to implement as many locks as one desires. The lock would be released when the thread dies in the clean up code. The locks that result in thread awaiting are still in-memory, of course. They are just OS provided primitives. Yet again, a well behaved/designed lock should be non-contested for the most calls (easily 99.9%+). In that case there is no mode-switch to get to the kernel. A contested lock that goes to the kernel is a context switch. In the end there is no need to do anything special about locks as they can implemented in the user-space unlike the file descriptors.

shakna 4y ago

> No 'defer', 'with' or other special handling needed. Just as an aside - in Python, it surprised me that 'with' is _not_ a destructor. The object created will remain in-scope after the 'with' body, but the '__exit__' method will be called. Which for the canonical example of file-handling, only really closes the file.

jnwatson 4y ago
The done vs destroyed dichotomy is a cross-cutting concern for programs that need explicit control over resource lifetimes. RAII is actually an antipattern in Python if you actually care about resources being released.
- shakna 4y ago
  The surprise is less about resource usage and more about the scope lifetime. 'with' introduces a new binding that outlives its body. It looks, at first glance, to be a temporary variable. But it isn't.
  - pphysch 4y ago
    Consider the common pattern: with open(path) as file: result = func(file.read()) another_func(result) If `with` used a separate scope then it would be a lot less ergonomic. How might we bring data outside that scope and be confident that `with` isn't forcibly cleaning things up? Would be a lot more verbose.
    - shakna 4y ago
      I think you've missed what I've meant by a new binding. That the body executes is entirely intuitive. However: with open(path) as file: ... if file.mode == 'r': print("Huh? file is still a thing?")
      - xscott 4y ago
        Python doesn't have block scope, but most people don't seem to notice: def foo(): if False: x = 123 print("x is visible out here:", x) That will generate an error because `x` is "referenced before assignment", but the variable exists for the entire function. This can bite you in subtle ways: x = "is global out here" def foo(): print("all good:", x) def bar(): print("not good:", x) if False: x = "now x is local, even before this declaration"
  - chombier 4y ago
    I always wondered why does this variable leak outside the `with` scope, because this is really the one place where it should not. IIRC python3 fixed scope leaks in list comprehensions, does anyone know why they didn't fix this one as well? Or does it have actual uses in practice?
    - shakna 4y ago
      I don't know the _motivation_ for why it leaks, but it leaks because the method calls of a context manager are __enter__ and __exit__, but not __del__. [0] The motivation is especially opaque, because though CPython does run __exit__ immediately, it isn't guaranteed to do so by the standard. So __exit__ will happen at some point, like __del__ with the GC, but it won't actually call __del__, which makes GC'ing the value more complicated as __del__ can't safely run until __exit__ does. I _think_ the motivation is something to do with __exit__ being allowed to throw an exception that can be caught, whereas __del__ isn't allowed to do that, but I don't find it that convincing. And I've never seen an example where the introduced variable is ever referenced afterwards in Python's documentation (including in the PEP itself). [0] https://peps.python.org/pep-0343/ https://peps.python.org/pep-0343/
      - chombier 4y ago
        Thinking about it, my guess is that properly scoping the variable does not really prevent the user from leaking the value anyway: with context() as foo: leak = foo This is unlike list comprehensions, where it is syntactically impossible to bind the comprehension variable to a new name accessible outside the comprehension. So maybe the reason is that it is not really worth the trouble.
  - ptx 4y ago
    Everything is always* scoped to the entire function in Python. Perhaps that's not always ideal, but at least the rule is simple. * Well, not variables in generator expressions. There might be a few other exceptions.

SemanticStrengh 4y ago

What does it bring vs Java autocloseable?

nu11ptr 4y ago
Java's "try with resources" is effectively the same as Go's 'defer' and Python's 'with', so the advantage is that you can't forget to do it. No matter what you do, when the object goes out of scope, the resource will be closed.
- pjmlp 4y ago
  If you actually use static analysis, you can't forget it in Java either. And it isn't as if using languages without static analysis is still considered a best practice in the 21st century.
  - tialaramex 4y ago
    > And it isn't as if using languages without static analysis is still considered a best practice in the 21st century. So in summary, "Please just do better". Not helpful as a video game meme, and really no better for programming.
    - pjmlp 4y ago
      I guess that is why no one bothers with clippy then, it is superfluous.
      - tialaramex 4y ago
        Everywhere that a clippy lint is there to catch bugs, I wish it wasn't a clippy lint because it won't always get run and those bugs will get missed. An example of a clippy lint that I like in clippy: If you try to transmute() &[u8] into a &str clippy will point out that you should probably from_utf8_unchecked() -- this lint results in code that does the same thing but is clearer for maintainers about what we intended. If our reaction to the lint is "But it isn't UTF-8" then we just got an important red flag, but that's not why the lint is there. On the other hand, clippy lints for truncating casts and I want those gone from the language so the warnings ought to at least live in the compiler not clippy. I would like to see a future Edition outlaw these casts entirely so that you need to write a truncating cast explicitly if that's what you needed.

oaiey 4y ago

I would argue that resources need always explicit handling, no matter whether you do reference counting or GC. So if I do a one-liner to dispose an object or call delete on an object is the same beauty. Wait till the variable is out of scope together with many more or hit the GC cleanup is just not the right way to clean up externally resources in a proper way.

xxs 4y ago

>the non-memory resources it is holding (files, locks, etc.) Locks are entirely a memory construct. You can trivially write one, e.g. Java locks are written entirely in java. The files descriptors (incl. socket) - they should not be relied to be GC'd by default, they do require their own lifecycle. For instance: c# Disposable/ java's try with resource. >no heap allocation for many objects at all. This has been already the case for many a year - reference escape analysis, ref. escape elission, trace compilation.

tialaramex 4y ago
Locks are, in fact, one of many resources your operating system can give you. Historically, in concurrent software your only option was to either use these OS allocated locks or build a spin lock from atomic operations (and thus burn CPU every time you're waiting for a lock). The OS allocated locks, such as from sem_open() or create_sem() must later be properly released or you're leaking a finite OS resource that will only be reclaimed on process exit or in some cases only at reboot. Today your operating system probably offers a lightweight "futex" style locking primitive, in which so long as it's not locked no OS resources are reserved it's just memory. But, you're not guaranteed that all your locks are these lightweight structures and even if you were you need to clean up properly because when they are locked the OS is using resources so you need to unlock properly or you'll blow up.
- xxs 4y ago
  The only part needed to build a lock is to be able to 'sleep' the current thread and the latter to be able to be awaken, plus compare-and-swap or load linked/store conditional. There is nothing really special about the locks as long as the current thread can be made dormant. Technically a single OS lock (e.g. a futex), associated with the current thread would suffice to implement as many locks as one desires. The lock would be released when the thread dies in the clean up code. The locks that result in thread awaiting are still in-memory, of course. They are just OS provided primitives. Yet again, a well behaved/designed lock should be non-contested for the most calls (easily 99.9%+). In that case there is no mode-switch to get to the kernel. A contested lock that goes to the kernel is a context switch. In the end there is no need to do anything special about locks as they can implemented in the user-space unlike the file descriptors.