5 ms·

For files python[0] does not need an explicit with to close the file. It gets closed when references reach zero. [0] For the pedants repeating each other in th

by illumen 12y ago

For files python[0] does not need an explicit with to close the file. It gets closed when references reach zero. [0] For the pedants repeating each other in the replies... Where by python, I mean the implementation that 98% of people use, called python, which the competition calls CPython. Yes it is a great feature of the implementation, like the deterministic memory management through reference counting.

Wilduck 12y ago
This is currently true in CPython, but is not necessarily true in other implementations. It's good hygiene in python to use a `with` statement for this reason, as well as helping to enforce that you don't accidentally hold on to a reference, and keep the file open. I think that context managers in python are generally a very nice abstraction over these sorts of concepts, and it's cool to see how Rust is handling these problems very cleanly.
wycats 12y ago
What happens if there's a cycle? It will get closed when the cycle is finally collected. You can rely on a GC (and refcounting + a cycle collector is absolutely a GC) to finalize your resources for you, but it is quite difficult to reason about when the file will eventually be closed. Also, the reference counting semantics of CPython are implementation-specific. From the PEP that introduced `with`: > Note that we're not guaranteeing that the finally-clause is executed immediately after the generator object becomes unused, even though this is how it will work in CPython. This is similar to auto-closing files: while a reference-counting implementation like CPython deallocates an object as soon as the last reference to it goes away, implementations that use other GC algorithms do not make the same guarantee. This applies to Jython, IronPython, and probably to Python running on Parrot.
- illumen 12y ago
  CPython is the implementation that 98% of the market uses, but you are partially correct for that other 2%. You are also wrong for CPython which does NOT require the garbage collector to be on, and DOES release the file in a deterministic way. Simply, the article is misleading with regards to python.
  - mwcampbell 12y ago
    Still, do you want to bake assumptions based on CPython's implementation into your codebase, then have those assumptions invalidated when you later switch to PyPy to take advantage of the JIT compiler?
  - apendleton 12y ago
    If the garbage collector were turned off and a cycle were to be present, the file would simply never be closed. Though I suppose that would, indeed, be deterministic.
- phunge 12y ago
  Props to Rust for tackling this head on, more languages should provide resource allocation that's deterministic, predictable and syntactically convenient. CPython's behavior is nice, but it seems to me it came about by accident. Big heavy resources use refcounting because everything uses refcounting. Plus, if CPython had true concurrency, across-the-board refcounting probably wouldn't have lasted nearly as long (reason being: multithreaded refcounting requires atomic ops which make them much more expensive). But in the general language design world, IMHO, refcounting is actually a pretty good compromise for resource management. Because: 1) The efficiency lost compared to Rust's approach is probably immaterial (since the Big Expensive Object that you're mananaging dwarfs the refcounting -- syscalls and file descriptors are expensive). So I don't think zero overhead is critical. 2) Cycles are pretty easy to avoid when you're only refcounting Big Expensive Objects (FDs, mmapped buffers, etc. etc.) For a general runtime, refcounting is tricky. But if it's a special part of the language that's small, simple, and only for resource management, it's pretty convenient. I think it's worlds better than the approach of mixing GC and resource management -- language designers should admit that's a terrible idea.
  - kibwen 12y ago
    I agree that refcounting is a great and usable compromise in this space, but your own comment flirts with why I don't think modern languages are chomping at the bit to base resource management on it: concurrency. Granted, I have no idea how much overhead is imposed by atomic operations vs. a stop-the-world or concurrent GC (if anyone has some data, I'd love to see it!). But given how it's become de rigeur for new languages to come with a baked-in concurrency story and emphasize concurrent applications, I don't blame them for not wanting to tie themselves to the RC cart.
    - chetanahuja 12y ago
      "Granted, I have no idea how much overhead is imposed by atomic operations vs. a stop-the-world or concurrent GC." An atomic read/write/increment/decrement is a non-blocking (from the point of the view of a user code running on the CPU) CPU operation. The CAS operations required to do atomic inc/dec are relatively expensive compared to normal memory read/writes of course but nowhere near the scale of stop-the-world GC (where all threads in the runtime have to be blocked for a potentially large amount of time, sometimes measured in whole seconds or even 10's of seconds for large heaps). Anyone who has had to troubleshoot JVM's handling large heaps (in the ~10GB range) quickly learns to love deterministic costs of refcounted resource management.
masklinn 12y ago
That's an implementation detail of CPython. You rely on that behaviour, your code is broken: * on pypy * on jython * on ironpython That is, it's broken on Python, and it's not correct Python code. It's only correct CPython code.
- Derbasti 12y ago
  Why? Wouldn't any garbage collector collect the file object when it goes out of scope? def foolish(): f = open('filename.txt') Once `foolish` returns, `f` goes out of scope, refcounting or not. Once `f` goes out of scope, its destructor closes the file. In CPython, the destructor is called when the refcount reaches zero. In Pypy, the destructor is called when there are no more references to `f`. Both happen when `foolish` returns, in both Pypy and CPython. This is an implementation detail of file, not CPython. Or did I fundamentally misunderstand something here?
  - Jweb_Guru 12y ago
    Yes, you misunderstood how tracing garbage collection works. Cleanup of data allocated on the heap occurs during tracing phases, not when a reference goes out of scope. Otherwise, imagine if before it returned f copied the object into some global structure g--how would the garbage collector know it was safe to close? (Some garbage collected languages can do automated "escape analysis" to figure out whether an object "escapes" the scope, and if it doesn't place it on the stack instead of the heap and deterministically free it when it goes out of scope. But that's surprisingly limited in real world programs and is generally speaking an implementation / optimization detail, not a core guarantee of the language).
    - Derbasti 12y ago
      How does that contradict what I was saying? The garbage collector would still close the file eventually, right? Unless of course it was copied somewhere else in the meantime, but in the code example, that is clearly not the case.
      - 12y ago

by illumen 12y ago

Wilduck 12y ago

This is currently true in CPython, but is not necessarily true in other implementations. It's good hygiene in python to use a `with` statement for this reason, as well as helping to enforce that you don't accidentally hold on to a reference, and keep the file open. I think that context managers in python are generally a very nice abstraction over these sorts of concepts, and it's cool to see how Rust is handling these problems very cleanly.

wycats 12y ago

What happens if there's a cycle? It will get closed when the cycle is finally collected. You can rely on a GC (and refcounting + a cycle collector is absolutely a GC) to finalize your resources for you, but it is quite difficult to reason about when the file will eventually be closed. Also, the reference counting semantics of CPython are implementation-specific. From the PEP that introduced `with`: > Note that we're not guaranteeing that the finally-clause is executed immediately after the generator object becomes unused, even though this is how it will work in CPython. This is similar to auto-closing files: while a reference-counting implementation like CPython deallocates an object as soon as the last reference to it goes away, implementations that use other GC algorithms do not make the same guarantee. This applies to Jython, IronPython, and probably to Python running on Parrot.

illumen 12y ago
CPython is the implementation that 98% of the market uses, but you are partially correct for that other 2%. You are also wrong for CPython which does NOT require the garbage collector to be on, and DOES release the file in a deterministic way. Simply, the article is misleading with regards to python.
- mwcampbell 12y ago
  Still, do you want to bake assumptions based on CPython's implementation into your codebase, then have those assumptions invalidated when you later switch to PyPy to take advantage of the JIT compiler?
- apendleton 12y ago
  If the garbage collector were turned off and a cycle were to be present, the file would simply never be closed. Though I suppose that would, indeed, be deterministic.
phunge 12y ago
Props to Rust for tackling this head on, more languages should provide resource allocation that's deterministic, predictable and syntactically convenient. CPython's behavior is nice, but it seems to me it came about by accident. Big heavy resources use refcounting because everything uses refcounting. Plus, if CPython had true concurrency, across-the-board refcounting probably wouldn't have lasted nearly as long (reason being: multithreaded refcounting requires atomic ops which make them much more expensive). But in the general language design world, IMHO, refcounting is actually a pretty good compromise for resource management. Because: 1) The efficiency lost compared to Rust's approach is probably immaterial (since the Big Expensive Object that you're mananaging dwarfs the refcounting -- syscalls and file descriptors are expensive). So I don't think zero overhead is critical. 2) Cycles are pretty easy to avoid when you're only refcounting Big Expensive Objects (FDs, mmapped buffers, etc. etc.) For a general runtime, refcounting is tricky. But if it's a special part of the language that's small, simple, and only for resource management, it's pretty convenient. I think it's worlds better than the approach of mixing GC and resource management -- language designers should admit that's a terrible idea.
- kibwen 12y ago
  I agree that refcounting is a great and usable compromise in this space, but your own comment flirts with why I don't think modern languages are chomping at the bit to base resource management on it: concurrency. Granted, I have no idea how much overhead is imposed by atomic operations vs. a stop-the-world or concurrent GC (if anyone has some data, I'd love to see it!). But given how it's become de rigeur for new languages to come with a baked-in concurrency story and emphasize concurrent applications, I don't blame them for not wanting to tie themselves to the RC cart.
  - chetanahuja 12y ago
    "Granted, I have no idea how much overhead is imposed by atomic operations vs. a stop-the-world or concurrent GC." An atomic read/write/increment/decrement is a non-blocking (from the point of the view of a user code running on the CPU) CPU operation. The CAS operations required to do atomic inc/dec are relatively expensive compared to normal memory read/writes of course but nowhere near the scale of stop-the-world GC (where all threads in the runtime have to be blocked for a potentially large amount of time, sometimes measured in whole seconds or even 10's of seconds for large heaps). Anyone who has had to troubleshoot JVM's handling large heaps (in the ~10GB range) quickly learns to love deterministic costs of refcounted resource management.

masklinn 12y ago

That's an implementation detail of CPython. You rely on that behaviour, your code is broken: * on pypy * on jython * on ironpython That is, it's broken on Python, and it's not correct Python code. It's only correct CPython code.

Derbasti 12y ago
Why? Wouldn't any garbage collector collect the file object when it goes out of scope? def foolish(): f = open('filename.txt') Once `foolish` returns, `f` goes out of scope, refcounting or not. Once `f` goes out of scope, its destructor closes the file. In CPython, the destructor is called when the refcount reaches zero. In Pypy, the destructor is called when there are no more references to `f`. Both happen when `foolish` returns, in both Pypy and CPython. This is an implementation detail of file, not CPython. Or did I fundamentally misunderstand something here?
- Jweb_Guru 12y ago
  Yes, you misunderstood how tracing garbage collection works. Cleanup of data allocated on the heap occurs during tracing phases, not when a reference goes out of scope. Otherwise, imagine if before it returned f copied the object into some global structure g--how would the garbage collector know it was safe to close? (Some garbage collected languages can do automated "escape analysis" to figure out whether an object "escapes" the scope, and if it doesn't place it on the stack instead of the heap and deterministically free it when it goes out of scope. But that's surprisingly limited in real world programs and is generally speaking an implementation / optimization detail, not a core guarantee of the language).
  - Derbasti 12y ago
    How does that contradict what I was saying? The garbage collector would still close the file eventually, right? Unless of course it was copied somewhere else in the meantime, but in the code example, that is clearly not the case.
    - 12y ago