5 ms·

Horrendous advice on many levels - reference counting is an implementation detail, it is not part of the spec - that's because real implementations exist wher

Horrendous advice on many levels - reference counting is an implementation detail, it is not part of the spec - that's because real implementations exist where refcounting does not exist (ironpython, pypy, jython) - even if it were part of the spec, the schedule for object destruction also is not part of the spec. Your file object was noticed as part of an unused graph at T+0, even mainline CPython can perpetually defer its destruction to T+infinity due to its handling of reference cycles - the random system resource will eventually be garbage collected, but perhaps not if your code throws the wrong exception and/or one of those cute syntax sugar libraries takes a reference to its frame object you know nothing about. Or how about pretty much any debugging support library like Sentry? Best to have audited them all.. - the random system resource was garbage a few tens of opcodes ago, but you left it locked, and some future change caused your code to immediately attempt to reopen and relock it. surprise, deadlock in production The comparison with 'dict.clear()' makes no sense whatsoever because explicit external resource control is precisely required because those external resources aren't automatically managed like normal in-memory objects. They have material external state beyond whether they are allocated or deallocated, and require explicit initialization and de-initialization in order to avoid leaving them in unspecified states. Python's memory allocator provides none of these behaviours as guarantees. It's not even guaranteed to always call __del__, this is most commonly observable during interpreter shutdown. Perhaps a better way of looking at it is this: try/finally and with: exist for you to demand stronger guarantees from the runtime than it normally offers you. "Hey Mr. Python, I know you'll /probably/ clean up this object eventually, but I also know there are 100 edge cases where you won't bother. For this object, you are on the hook for every case, including the cases where I wrote buggy code" Meanwhile, feel free to follow this advice, and good luck at 4am debugging your prod system that for some inexplicable reason recently began crashing due to running out of file descriptors..

epage 6y ago
Another problem: Windows support. Windows' file semantics are different and knowing when a file is closed becomes critical.
- gruez 6y ago
  Elaborate?
  - plorkyeran 6y ago
    By default Windows locks a file while it is open and does not let you delete it. Other platforms do not, so code which naively reads and then deletes a file without explicitly closing it will work fine until you try to run it on Windows.
  - Pxtl 6y ago
    Unclosed file handles are incredibly painful on Windows. The user can't edit or delete the file while it's in use and gets cryptic errors about it. There is no UI showing which process is holding the file handle. The standard practice on Windows when faced with a file that's got a stuck handle is to reboot.
    - grenoire 6y ago
      Wow, thank you for putting into writing one of my greatest pet peeves about Windows. Never thought about how often I encounter this issue until now, and how bad the UX is.
      - Pxtl 6y ago
        Honestly, everything about the Explorer feels 2 decades behind he rest of Windows 10. Like, the rest of MS is has moved on, but their file browser is like an IE6-esque albatross. Everything makes it lock up.
        cameronbrown 6y ago
        Explorer is the only Windows 10 app that doesn't make me hate life. I'll take 2010 functionality over "metro" UI apps running in UWP anyday.
        6y ago
masklinn 6y ago
> - reference counting is an implementation detail, it is not part of the spec > - that's because real implementations exist where refcounting does not exist (ironpython, pypy, jython) Exactly. That was a major if not the major drive for context managers & properly closing files: the PFS and the language team were getting much more supportive of third-party implementations, and much more aware of issues in "normal" Python code e.g. though it didn't work out well in the end, ctypes also got merged into the stdlib in 2.5 with the express purpose of limiting the need for writing cpython extensions to bind to native libraries. > - the random system resource will eventually be garbage collected Not necessarily! In fact this is one of the major problems for non-memory resources in GC'd languages: the GC is triggered by memory pressure and that's it. Because it knows nothing about fd pressure, if fd consumption is not matched by enough memory consumption (or a properly matching pattern thereof) it's rather possible to keep leaking fds without the GC ever running (or ever running a collection which reclaims your file objects). For instance, back then trying to install some packages would crash under pypy due to file exhaustion, because setuptools leaked fds all over the place: https://bitbucket.org/pypy/pypy/issues/878 https://bitbucket.org/pypy/pypy/issues/878 The stdlib also did: https://bugs.python.org/issue13133 https://bugs.python.org/issue13133
- catern 6y ago
  >the GC is triggered by memory pressure and that's it. Because it knows nothing about fd pressure, So why not teach it about fd pressure? There's no reason not to teach a GC about fd pressure - the reason that's rarely done is that language implementers generally don't care that much about the system interface - they just link against libc and call it done.
  - dragonwriter 6y ago
    > You can't even define a custom hashing function All objects subject to GC have a memory impact, very few of them have an fd impact. You either have to separately track those that impact fds, or do a lot (up to 100%) unnecessary work in response to fd pressure. Even if user code can disable the fd pressure response, you've got to carry the baggage of a GC with extra code to handle it around. It's very easy to see “teaching a GC about fd pressure” as not worth the time and effort for the main runtime GC for a general purpose language. If you had a facility for swapping in special purpose GCs for particular applications, building a specialty GC that added fd awareness might be a net win for some apps, but...

epage 6y ago

Another problem: Windows support. Windows' file semantics are different and knowing when a file is closed becomes critical.

gruez 6y ago
Elaborate?
- plorkyeran 6y ago
  By default Windows locks a file while it is open and does not let you delete it. Other platforms do not, so code which naively reads and then deletes a file without explicitly closing it will work fine until you try to run it on Windows.
- Pxtl 6y ago
  Unclosed file handles are incredibly painful on Windows. The user can't edit or delete the file while it's in use and gets cryptic errors about it. There is no UI showing which process is holding the file handle. The standard practice on Windows when faced with a file that's got a stuck handle is to reboot.
  - grenoire 6y ago
    Wow, thank you for putting into writing one of my greatest pet peeves about Windows. Never thought about how often I encounter this issue until now, and how bad the UX is.
    - Pxtl 6y ago
      Honestly, everything about the Explorer feels 2 decades behind he rest of Windows 10. Like, the rest of MS is has moved on, but their file browser is like an IE6-esque albatross. Everything makes it lock up.
      - cameronbrown 6y ago
        Explorer is the only Windows 10 app that doesn't make me hate life. I'll take 2010 functionality over "metro" UI apps running in UWP anyday.
        6y ago

masklinn 6y ago

> - reference counting is an implementation detail, it is not part of the spec > - that's because real implementations exist where refcounting does not exist (ironpython, pypy, jython) Exactly. That was a major if not the major drive for context managers & properly closing files: the PFS and the language team were getting much more supportive of third-party implementations, and much more aware of issues in "normal" Python code e.g. though it didn't work out well in the end, ctypes also got merged into the stdlib in 2.5 with the express purpose of limiting the need for writing cpython extensions to bind to native libraries. > - the random system resource will eventually be garbage collected Not necessarily! In fact this is one of the major problems for non-memory resources in GC'd languages: the GC is triggered by memory pressure and that's it. Because it knows nothing about fd pressure, if fd consumption is not matched by enough memory consumption (or a properly matching pattern thereof) it's rather possible to keep leaking fds without the GC ever running (or ever running a collection which reclaims your file objects). For instance, back then trying to install some packages would crash under pypy due to file exhaustion, because setuptools leaked fds all over the place: https://bitbucket.org/pypy/pypy/issues/878 https://bitbucket.org/pypy/pypy/issues/878 The stdlib also did: https://bugs.python.org/issue13133 https://bugs.python.org/issue13133

catern 6y ago
>the GC is triggered by memory pressure and that's it. Because it knows nothing about fd pressure, So why not teach it about fd pressure? There's no reason not to teach a GC about fd pressure - the reason that's rarely done is that language implementers generally don't care that much about the system interface - they just link against libc and call it done.
- dragonwriter 6y ago
  > You can't even define a custom hashing function All objects subject to GC have a memory impact, very few of them have an fd impact. You either have to separately track those that impact fds, or do a lot (up to 100%) unnecessary work in response to fd pressure. Even if user code can disable the fd pressure response, you've got to carry the baggage of a GC with extra code to handle it around. It's very easy to see “teaching a GC about fd pressure” as not worth the time and effort for the main runtime GC for a general purpose language. If you had a facility for swapping in special purpose GCs for particular applications, building a specialty GC that added fd awareness might be a net win for some apps, but...