14 ms·

Contrarian view on closing files

91 points by coady 6y ago

graton 6y ago
Hmm. I just ran out of file descriptors on a little personal project of mine just a few days ago. I kept calling pty.openpty() and I forgot to close them. My program was calling it three times (stdout, stderr, stdin) every 10 seconds :( I ended up writing a context manager for it and now it works well. Unfortunately contextlib.closing() wouldn't work as pty.openpty() returns two objects.
bjourne 6y ago
Pretty ballsy posting a link to your own blog post on HN on such a controversial topic. Prepare to get smashed in the comments!
Izkata 6y ago
> Moreover, according to this best practice, it's impossible to write the csv version correctly. Um, just consume the iterator immediately, then it's the same as the json one: def load(filepath): with open(filepath) as f: return list(csv.reader(f)) The equivalent for the latter json example that separates the read and parse would be: def load(filepath): with open(filepath) as f: contents = f.readlines() return csv.reader(contents)
- panarky 6y ago
  > Um, just consume the iterator immediately That's fine for some use cases, but TFA wanted a lazy iterator.
  - Skunkleton 6y ago
    If you want a lazy iterator, then you will need to keep the file handle open. I don’t see how this one use case invalidates the general guideline of not sitting on file descriptors that you are finished with.
    - dmurray 6y ago
      How you feel about this is key to the article's argument. It makes the point that you shouldn't have to know if the iterator is lazy or not, that's an implementation detail. That does seem...generally desirable? But given Python already gives you easy and different syntax to create lazy vs greedy iterators, and you often can't use them interchangeably, it's clearly not a top design goal of the language.
      - joshuamorton 6y ago
        It sounds desirable, but isn't feasible unless the language is stateless. If you iterate over a thing that is stateful, you need to know whether the iterator is lazy or not to decide if you can modify the underlying thing.
        dmurray 6y ago
        I don't think that's right. You can always turn it into a greedy iterable (e.g. convert it to a list) if you need the certainty that you can modify the underlying thing, while still reaping some performance benefits from the maybe-lazy iterator in the rest of your use cases. Assuming the iterator wasn't infinite, of course, but in that case you already knew it was lazy.
        joshuamorton 6y ago
        Right but you have to know the iterator was lazy to know that you need to convert it.
        hansvm 6y ago
        Is it really that arduous to have a high-level understanding of what the method you're calling does, at least at the level of knowing whether or not it returns the type of object you want and whether it takes ownership of its arguments? There's a separate question of whether the standard library should be inconsistent with itself, but it seems like a reasonable choice -- you can't efficiently use a json file without doing _something_ invasive to it like loading it all into ram, but a csv file isn't burdened the same way, and you probably don't want to be limited by available ram when all you need to do is enumerate a csv.
  - Izkata 6y ago
    The article did not imply that: > """with a different file format""" The assumption behind the paragraph following the example does not imply lazy evaluation, but rather the opposite - the broken one is obvious if you have lazy evaluation in mind, whereas the docstrings imply the caller would be using these the same and not be expecting lazy evaluation.
- compressedgas 6y ago
  Or you can go the other way and force the caller of the load function to ensure that the iterator is closed Pass the responsibility as far up as required. Make the `load` function return a context manager. So any use of load would be like: with load(x) as it: pass `load` could be written with `contextlib.contextmanager` as: @contextlib.contextmanager def load(filepath): with open(filepath) as f: yield csv.reader(f)
BeefySwain 6y ago
Many good points are made here, but I would point out that there is a significant difference between working with JSON and CSV: data that can be operated on can typically be extracted from a CSV by working iteratively over each line. The same can not be said about JSON files. It is normal and (almost always ?) required to parse/de-serialize an entire JSON file at once and hold the contents of it in memory, while the same is often impractical and unnecessary for a CSV (when it is millions of lines). The standard methods and idioms of doing these things in Python reflects this.
- dragonwriter 6y ago
  > It is normal and (almost always ?) required to parse/de-serialize an entire JSON file at once and hold the contents of it in memory It may be normal, but there's no reason it is required. Streaming JSON processing makes sense just like streaming XML processing.
- tomjakubowski 6y ago
  It's not uncommon to see files or streams in "JSON lines" format. For files where each line is a record and the records are sorted or indexed, it's possible to seek to a record of interest just as in CSV. http://jsonlines.org/ http://jsonlines.org/
  - coldtea 6y ago
    Those files are individual JSON entries though, as per the spec. They're not read as a JSON (since they aren't) they are read per line, and each line is parsed as a JSON.
    - orf 6y ago
      Same as CSV then?
      - coldtea 6y ago
        Yes, but not the same as JSON. Which was my point. In other words, collection of JSON strings in a file != a JSON file.
j88439h84 6y ago
If you're reading the whole file at once, I'd use `Path().read_text()`.
eximius 6y ago
The point of a file context manager is to close the file when you're done with it. The CSV reader isn't done with it. Given that the CSV reader does not control ownership of the file handle, it shouldn't return a lazy iterator when given a file handle. It should accept a path, then it could own the file handle until the iterator is complete or dropped. So, yea, the CSV reader does a bad. But so what? That is a deficiency of the CSV library, not context managers.
- dmurray 6y ago
  The CSV library is part of the standard library, though. Its way of doing things has been blessed by the Python maintainers. OK, it might still be correct to say the problem is with the CSV library. But if the context manager "best practices" lead to popular parts of the standard library being bad and unidiomatic, that's at best unpythonic. After all, "There should be one -- and preferably only one -- obvious way to do it".
  - arugulum 6y ago
    By the Python standard library already does contain things that don't follow best practice. See: logging doesn't abide by PEP 8.
- throwaway894345 6y ago
  Libraries that accept filepaths (except perhaps as a convenience) are bad. The CSV library is well-designed. It isn’t responsible for closing files because it doesn’t take files, it takes TextIO. None of this is to say context managers are bad; I think they’re great but they only work as intended.
  - solidasparagus 6y ago
    That's being a little pedantic IMO. From a practical, user-facing perspective, the CSV library Reader takes in files. And they know this, which is why the documentation specifically calls out gotchas related to passing in a file object and the example code shows how it should work with a file object [1]. [1] https://docs.python.org/3/library/csv.html#csv.reader https://docs.python.org/3/library/csv.html#csv.reader
nemetroid 6y ago
> But in all seriousness, this advice is applying a notably higher standard of premature optimization to file descriptors than to any other kind of resource. I don't think it's premature, but properly closing files is definitely not about optimization. Maybe "a notably higher standard of code quality"?
- shawnz 6y ago
  How does it positivity affect code quality? It seems to me like it would be the opposite, automatic closing means less boilerplate and more expressive code
  - tcbasche 6y ago
    By automatic closing do you mean not at all and expecting the GC to do it? Or with a context manager. I think trusting an implementation detail of the underlying interpreter is lazy at best and an anti-pattern at worst. We should aim to show our intent as much as possible, and closing a file or using a context manager certainly expresses our intent.
    - shawnz 6y ago
      I mean imagine a hypothetical language where it was idiomatic to let the GC close files for you and the language was specified in such a way that it was guaranteed to work efficiently and wasn't just an implementation detail. Would code in that language have lower code quality than code in a language where files have to be closed manually? That's what the other poster seemed to be implying, but I think the opposite is true
      - nemetroid 6y ago
        I don’t quite see what you mean by ”efficiently”. A language where garbage collection of file objects is guaranteed to happen within N milliseconds? The issue is holding on to an external resource for longer than needed. and by definition, garbage collection is deferred cleanup.
wwright 6y ago
IMO part of what’s happening here is that Python’s GC model works well for memory and leaks for other resources. Rust’s ownership model works very well for many non-memory resources IMO, and even for memory it is very safe and efficient, with the only cost being mental overhead at times. Something like Rust’s ownership system but with an easy escape hatch for “loose” resource management (ie GC) could solve a lot of the cognitive load with resources, I think. Rc is helpful, but not nearly as ergonomic as a GC language. I think withoutboats had a similar blog post a year or two ago.
- gizmo686 6y ago
  I'm not sure it is fair to attribute this to Python's GC model, in that what is commonly thought of as garbage collection is not involved with the issue being discussed. However, in principle, this issue is very simmilar to that which garbage collectors set out to solve. In effect, python's context managers combined with the "with" syntax recreate the sementics of C's local variables in that: 1) They can be passed outside the lexical scope in which they were defined but 2) They're usage is illegal after execution has exited the "block" in which they are defined. 3) There is nothing which prevents a programmer from keeping and attempt to use a reference to them after execution has left the code block. The approach I take in python is the same I take in C. Most of the time, I only need to file for a short period, so I can "stack" allocate it with a context manager and let the runtime handle closing it for me. In cases where stack-like sementic do not fit, you need to use heap-like semantics and manually open/close it.
  - wwright 6y ago
    If you’re not familiar with Rust, one of the core features is that you can “pass” or “borrow” that stack-like scoping in a safe way. IE, if you create a file handle, it gets automatically cleaned up when the current scope ends… unless you transfer ownership to other code, and then it is cleaned when THAT scope ends. It also prevents references from dangling after the resource has been cleaned, even in other classes or functions. This lets you model ownership rather than memory, and the compiler will translate that (for the most part) to memory management for you. What I was trying to say is that I think managing ownership as a first-class language feature is more generally useful than trying to abstract away the concern with something like GC.
Skunkleton 6y ago
There are lots of reasons to close a file when you are done with it, especially if you are writing to that file. These considerations are not about “pythonicity”, but rather about best practices for unix.
dragonwriter 6y ago
> Any resource may be limited or require exclusivity; that's why they're called resources. Similarly one should always explicitly call dict.clear when finished with a dict. I think there is an important distinction between resources for which there may be contention within the process, vs. external resources. It's tractable to provide sufficient certainty that in an internal resource isn't subject to problematic contention either analytically or by testing or a combination, but that's less tractable with external resources. So, use guaranteed freeing of resources (context managers, if possible, in Python) for external resources or for internal resources that you can't otherwise be reasonably sure are free of contention issues.
lsorber 6y ago
As a counterexample: we did actually run out of file descriptors on Lambda by not closing file descriptors.
viraptor 6y ago
I'm lost between the no-true-scotsmen and strawmen the post tries to construct. The advice about closing the descriptors is for practical reasons. There's no need for "true believers" taking that advice to extreme. Just like clearing the dict if you're dropping the reference anyway is useless. If you can consume the file immediately, you can close it. If you're doing lazy iteration you have to keep it open, and I don't understand why that's an issue.
- Someone 6y ago
  “If you're doing lazy iteration you have to keep it open, and I don't understand why that's an issue.” It isn’t an issue if you process a few files that way (unless you’re running your system very close to the limit), but it is an issue if you have a long running process that lazily iterates over files often. Examples are iterating over a file system (say a kind of grep) or a web server where an often called entry point does this.
  - viraptor 6y ago
    Sure. But I was getting at something different. You can choose your open files limit on one side and the approach (lazy or not) on the other. If you choose lazy there's no way around having to leave the file open.
superbatfish 6y ago
I'd like to emphasize the point even further for CPython users. Here's the author's closing sentence: >Your setup script is not going to run out of file descriptors because you wrote open("README.md").read(). In CPython, files are closed (if necessary) in __del__, so the descriptor is released immediately after that line executes, thanks to CPython's reference-counting mechanism. That idiom wouldn't cause you to run out of descriptors even in a long program. (The author knows this. Their point is that even if you're NOT using CPython, you STILL shouldn't worry about it too much, and I think I agree.) I know, I know, CPython's reference-counting behavior is considered an "implementation detail", but for RAM-intensive workloads which are common in Scientific Computing (especially biological imaging), it's often a necessity. So if I were to force myself to use context managers in cases like this, I'd be losing the benefits of terse code without actually preserving portability. (I didn't have portability in the first place.) In other words, if you're dependent on ref-counting anyway, you may as well admit it and enjoy the benefits that come with it!
- orf 6y ago
  It won’t be released immediately, that’s the point. It will be “released at some point in the future and definitely not during interpreter shutdown”. A “with” statement ensures the release happens deterministically, which it turns out is a desirable trait to have.
  - superbatfish 6y ago
    The Python standard makes no guarantee, but in CPython, the example above is deterministic. The cycle-detecting garbage collector is not involved in this case. Just refcounting.
    - james412 6y ago
      If open() throws an exception, perhaps after the FD is already open (Python does non-trivial IO to a file immediately after open), or read() throws an exception, the file object is at the mercy of any try/except higher in the stack, any bugs/"features" contained in those, and any sys.excepthook installed like Sentry The example above is deterministic only on current versions of CPython and only if it is guaranteed to succeed and if it is the only line of code in the program now and in perpetuity.
james412 6y ago
Horrendous advice on many levels - reference counting is an implementation detail, it is not part of the spec - that's because real implementations exist where refcounting does not exist (ironpython, pypy, jython) - even if it were part of the spec, the schedule for object destruction also is not part of the spec. Your file object was noticed as part of an unused graph at T+0, even mainline CPython can perpetually defer its destruction to T+infinity due to its handling of reference cycles - the random system resource will eventually be garbage collected, but perhaps not if your code throws the wrong exception and/or one of those cute syntax sugar libraries takes a reference to its frame object you know nothing about. Or how about pretty much any debugging support library like Sentry? Best to have audited them all.. - the random system resource was garbage a few tens of opcodes ago, but you left it locked, and some future change caused your code to immediately attempt to reopen and relock it. surprise, deadlock in production The comparison with 'dict.clear()' makes no sense whatsoever because explicit external resource control is precisely required because those external resources aren't automatically managed like normal in-memory objects. They have material external state beyond whether they are allocated or deallocated, and require explicit initialization and de-initialization in order to avoid leaving them in unspecified states. Python's memory allocator provides none of these behaviours as guarantees. It's not even guaranteed to always call __del__, this is most commonly observable during interpreter shutdown. Perhaps a better way of looking at it is this: try/finally and with: exist for you to demand stronger guarantees from the runtime than it normally offers you. "Hey Mr. Python, I know you'll /probably/ clean up this object eventually, but I also know there are 100 edge cases where you won't bother. For this object, you are on the hook for every case, including the cases where I wrote buggy code" Meanwhile, feel free to follow this advice, and good luck at 4am debugging your prod system that for some inexplicable reason recently began crashing due to running out of file descriptors..
- epage 6y ago
  Another problem: Windows support. Windows' file semantics are different and knowing when a file is closed becomes critical.
  - gruez 6y ago
bassman9000 6y ago
_The csv version returns an iterator over a closed file_ That's why method/function naming is critical and underestimated. Without having worked with python too much: _json.load()_ is a load: implies the action is done on return. I understand there's nothing to do after it, other than use the return value. _csv.reader()_ clearly returns an object that it's going to be used to traverse the underlying, so the underlying must remain open. Implies a stateful operation to me. Besides, examples are clear: https://docs.python.org/3.8/library/csv.html#examples https://docs.python.org/3.8/library/csv.html#examples Classical problem of wanting to use The Perfect Abstraction.
dragonwriter 6y ago
> Moreover, according to this best practice, it's impossible to write the csv version correctly. No, it's not: def load(filepath): """with a different file format""" with open(filepath) as file: return list(csv.reader(file)) This both uses csv.reader and fulfills the required contract correctly. Yes, it's tricky to both follow the “assure a resource is closed” pattern and return a lazy iterator over the external resource which you open. If you are going to lazily iterate over a resource and do the context manager pattern, you have to do the work within the context manager.
deathanatos 6y ago
IMO, the author is looking at the central problem with GC systems: non-deterministic destruction, and unclear ownership. > This trivial example has deeper implications. If one accepts this practice, then one must also accept that storing a file handle anywhere, such as on an instance, is also disallowed. Unless of course that object then virally implements it owns context manager, ad infinitum. Yes, and I just accept that. This happens in C#, too[1]: > If your class owns a field or property, and its type implements IDisposable, the containing class itself should also implement IDisposable. A class that instantiates an IDisposable implementation and storing it as an instance member, is also responsible for its cleanup. And for good reasons; in this regard, the languages are basically identical. I'd say any language that uses GC and instead has some sort of language-based guard system (Python's `with`, C#'s `using`) or even just through convention (e.g., Java, back before try-with-resources) will end up having that happen. Depending on the GC to collect these resources isn't wise, since no behavior on the GC is actually guaranteed. > No, it will not be fixed. This version only appears to work by not closing the file until the generator is exhausted or collected. I'd call that fixed; the file is properly and deterministically closed, when the generator is exhausted or manually closed. (By calling the generator's `close()` member function.) Now, it is now up to the calling code to remember at every use site to do that, and it should be made an explicit part of the documentation for the interface. But, the same for context managers, or things having a close: it's part of the contract that you will call that .close(), or wrap it in a with block, or the resource leaks until the GC gets to it. (I do wish generators were context managers, to make this easier.) IME, the bigger problem here is that GC forces, for resources that have a destructor beyond "free the memory" (that is, what the GC itself handles), for each call-site to remember to clean up the resource. Files are one thing, and generally, yes, you'll probably get lucky and not run into issues. Locks are more fun, and often leaving it to the GC to release a lock is a lot more painful. (Then there are threads, child processes, sockets, …) RAII fixes this for all types of resources by bounding the lifetime of the resource to the scope(s) that it lives in. (And, because it inevitably comes up, IME, people worry too much about cycles; cycles do happen, but are incredibly rare compared to non-memory resources (those that are more than just memory, e.g., files, locks, etc.), and vanishingly rare compared to all resources (those that are more than just memory and those that are just memory, like vectors or strings). Yes, they are an issue of sorts in RAII, but there are ways to handle them. I do not think the bad with cycles outweighs the good of RAII.) [1]: https://docs.microsoft.com/en-us/dotnet/standard/garbage-collection/implementing-dispose#cascade-dispose-calls https://docs.microsoft.com/en-us/dotnet/standard/garbage-col...
deleted 6y ago
[deleted]
dvt 6y ago
> The csv version returns an iterator over a closed file. It's a violation of procedural abstraction to know whether the result of load is lazily evaluated or not; it's just supposed to implement an interface. Moreover, according to this best practice, it's impossible to write the csv version correctly. As absurd as it sounds, it's just an abstraction that can't exist. This is simply not true. A load being lazy or not will often be a part of API documentation. In some cases, documentation will include that a .close() needs to be manually called. It's also trivial to read an entire CSV file, close it, and return an array of lines, or a dataframe, or whatever. Of course, you probably don't need to do it with a generator, and it won't be a two-liner (although @Izkata gives a two-liner example). In that implementation, you'd achieve behavior parity with json.loads(...). > No, it will not be fixed. This version only appears to work by not closing the file until the generator is exhausted or collected. I'm no fan of Python (even though I use it every day), but this is extremely well documented in PEP 342[1]. I'm honestly not sure what the complaint is here. I guess the standard Python CSV library is a bit wonky, but that's about it. [1] https://legacy.python.org/dev/peps/pep-0342/ https://legacy.python.org/dev/peps/pep-0342/
ehutch79 6y ago
This is like a worse version of javascript semicolons. Sure you dont HAVE to... but there's gotchas and corner cases to not doing it that you don't have to think about if you close your file desciptors when you're done with them
Camillo 6y ago
This is an actual drawback of not having deterministic destruction. This would not be a problem in C++ or Swift.
- The_rationalist 6y ago
  Exactly. Yet Java has autocloseable without having deterministic destruction
  - masklinn 6y ago
    AutoCloseable is the same as Python's __exit__, it exists as a hook to try-with-resource in the same way __exit__ is a hook for `with`.
Animats 6y ago
This is to some extent a Python-specific problem. In most languages with destructors, you're guaranteed a destructor call when a local variable's scope is exited. But not Python. Python isn't RAII, it's an explicit close language. That's what "with" is for. Python's "with" clauses, unusually, play well with exceptions at scope exit. Nested "with" clauses can fault out with a close exception and still have everything unwind properly. Very few languages get that right. Python probably has the most useful exception system around. Calling destructors from the garbage collector has a whole other set of problems. Suppose you're writing to a network and the network connection fails during close. Now you can be stuck in the garbage collector calling the destructor waiting for the network to time out. I'm not sure how garbage-collected Python implementations handle this. C# can potentially get into that state, too. Anybody know? Raising an exception in a C++ destructor is usually not happy-making. Microsoft tried to make that work in Managed C++, which has "re-animation". A destructor can pass a reference to its own object outside of the destructor, which "re-animates" the object. Turned out not to be a good idea. I would have liked to see exceptions in Rust, which has enough static ownership analysis to determine whether a destructor is trying to make something outlive its lifetime. Most of the problems with destructors vs exceptions involve scoping issues. If such errors are caught at compile time, exceptions are better behaved.
- tsimionescu 6y ago
  In most managed-memory languages, try...finally (and using/try-with-resources which are based on it) handle exceptions just fine. The only problem that sometimes occurs is that exceptions thrown by a finally handler override the exception being bubbled, which can be bad if you wanted to handle that exception later on, but otherwise the resources get cleared properly. D has better behavior here, where throwing an exception on this path will automatically keep track of many in-flight exceptions. Also, the problem of finalizers blocking is partly handled by the fact that finalizers are not run by the GC threads, but by a separate finalizer thread. Still, if a finalizer blocks, eventually you will run out memory, as all objects which require finalization will be pooling up. Also, the re-animation behavior you describe is common to finalizers in all GC languages which expose them, it's not in any way specific to Managed C++ (C++/CLR). It is simply a consequence of how finalization works: when an object which requires finalization is eligible for GC, it is not collected, but moved to the finalization queue. Objects in that queue are still live objects. When the finalizer thread successfully runs the finalizer of such an object, it is simply removed from the finalization queue. If it's finalizer has added back a reference from some other GC root to the object, then it will still stay live. The only special property is that an object can't be finalized twice, so even if its finalizer attaches it to the live object graph once, when it becomes unreachable again it will be collected, it will not be put back in the finalization queue, it will be directly collected. Basically, I don't think there is too much chance to find a valid use for this behavior, but it is very hard to prevent it without complicating the language significantly.
lmm 6y ago
> But in all seriousness, this advice is applying a notably higher standard of premature optimization to file descriptors than to any other kind of resource. Maybe the same standard should be applied to other kinds of resources. Maybe garbage collection was a mistake. > It's a violation of procedural abstraction to know whether the result of load is lazily evaluated or not; it's just supposed to implement an interface. Lazily-loaded things are different from eagerly loaded things in important ways. If it's not possible to make that distinction visible in your language, get a better language. > No, it will not be fixed. This version only appears to work by not closing the file until the generator is exhausted or collected. Which is exactly what we wanted. It is fixed. > Furthermore it demonstrates that often the context is not being managed locally. If a file object is passed another function, then it's being used outside of the context. Let's revisit the json version, which works because the file is fully read. Doesn't a json parser have some expensive parsing to do after it's read the file? It might even throw an error. And isn't it be desirable, trivial, and likely that the implementation releases interest in the file as soon as possible? Look at iteratees for the full solution to this problem. Managing context can be hard, but it doesn't have to be; you just have to use the right abstractions. > Those threads are essentially debating whether a resource pool is "leaking" resources. If you open a pool and never close it, of course you're leaking everything in that pool. > if all this orphaned file descriptor paranoia was well-founded, it would have been a problem back then too. It was. Long-running python programs broke because they ran out of file descriptors.
perfunctory 6y ago
> Your setup script is not going to run out of file descriptors because you wrote open("README.md").read(). How about even simpler (python3) Path("readme.md").read_text() Ta-da. I don't even have to worry about opening files, let alone closing.
- xenonite 6y ago
  To complete your comment, let us import pathlib. from pathlib import Path Path("readme.md").read_text() The functions read_text and the similar write_text, read_bytes, write_bytes are new in Python 3.5. https://docs.python.org/3/library/pathlib.html#pathlib.Path.read_text https://docs.python.org/3/library/pathlib.html#pathlib.Path....
  - pansa2 6y ago
    I wasn’t aware of these functions. At first glance, their addition to Python seems to be a significant violation of “one-- and preferably only one --obvious way to do it”. What are the advantages of `read_text` over the traditional `file` read methods?
patrec 6y ago
GC advocates say that managing memory lifetimes is an excessive burden to programmers. Anti-GC advocates say that memory is just one particular instance of management of life-timed resources, and that special casing it is an ugly hack and you should be using RAII. Rust probably is the leading exponent (C++'s way is more of a mess). The RAII critics point out that RAII also glosses over part of the problem because often de-allocating resources can itself fail (e.g. closing a file) and RAII (in Rust or C++) has no proper way to deal with that. Then, there is the Reference counting camp, the only people who can say that they solve the something close to the general automatic resource de-allocation problem -- because lifetimes are precisely tracked. But reference counting can't cope with cycles and generally has poor performance. Also, it does not directly help with de-allocation failures. One interesting recent variation are attempts at static reference counting: http://strlen.com/lobster/ http://strlen.com/lobster/. Last there are the old and the new muddle through camp. The old muddle through camp (e.g. java, python) adds finalizers with crap semantics (call-me-maybe). The new muddle-through camp uses GC for memory but adds some extra mechanism for other resources (defer in go, <close> in lua 5.4). Defer in go seems like an inferior version of rust-style RAII. Lua 5.4's <close> also has/had problems (see http://lua-users.org/lists/lua-l/2019-06/msg00179.html http://lua-users.org/lists/lua-l/2019-06/msg00179.html). The author of the article advocates old-style muddling through, but this feels unsatisfactory because it's not robust in general and likely worse than following current "best practice". But it still seems like there is still no really good general solution available. Rust-style ownership tracking would have solved the lifetime ambiguity of the file for the csv reader, but would not be able to deal with file close errors properly. These can be somewhat handled with exception based mechanisms, but not without problems either -- because multiple things can go out of scope at once, requiring cleanup and one or more of these cleanups may fail themselves but try catch statements (with stack unwinding) offer no natural way to deal with multiple failures being propagated up at once.

91 points by coady 6y ago

Hmm. I just ran out of file descriptors on a little personal project of mine just a few days ago. I kept calling pty.openpty() and I forgot to close them. My program was calling it three times (stdout, stderr, stdin) every 10 seconds :( I ended up writing a context manager for it and now it works well. Unfortunately contextlib.closing() wouldn't work as pty.openpty() returns two objects.

bjourne 6y ago

Pretty ballsy posting a link to your own blog post on HN on such a controversial topic. Prepare to get smashed in the comments!

Izkata 6y ago

> Moreover, according to this best practice, it's impossible to write the csv version correctly. Um, just consume the iterator immediately, then it's the same as the json one: def load(filepath): with open(filepath) as f: return list(csv.reader(f)) The equivalent for the latter json example that separates the read and parse would be: def load(filepath): with open(filepath) as f: contents = f.readlines() return csv.reader(contents)

panarky 6y ago
> Um, just consume the iterator immediately That's fine for some use cases, but TFA wanted a lazy iterator.
- Skunkleton 6y ago
  If you want a lazy iterator, then you will need to keep the file handle open. I don’t see how this one use case invalidates the general guideline of not sitting on file descriptors that you are finished with.
  - dmurray 6y ago
    How you feel about this is key to the article's argument. It makes the point that you shouldn't have to know if the iterator is lazy or not, that's an implementation detail. That does seem...generally desirable? But given Python already gives you easy and different syntax to create lazy vs greedy iterators, and you often can't use them interchangeably, it's clearly not a top design goal of the language.
    - joshuamorton 6y ago
      It sounds desirable, but isn't feasible unless the language is stateless. If you iterate over a thing that is stateful, you need to know whether the iterator is lazy or not to decide if you can modify the underlying thing.
      - dmurray 6y ago
        I don't think that's right. You can always turn it into a greedy iterable (e.g. convert it to a list) if you need the certainty that you can modify the underlying thing, while still reaping some performance benefits from the maybe-lazy iterator in the rest of your use cases. Assuming the iterator wasn't infinite, of course, but in that case you already knew it was lazy.
        joshuamorton 6y ago
        Right but you have to know the iterator was lazy to know that you need to convert it.
        hansvm 6y ago
        Is it really that arduous to have a high-level understanding of what the method you're calling does, at least at the level of knowing whether or not it returns the type of object you want and whether it takes ownership of its arguments? There's a separate question of whether the standard library should be inconsistent with itself, but it seems like a reasonable choice -- you can't efficiently use a json file without doing _something_ invasive to it like loading it all into ram, but a csv file isn't burdened the same way, and you probably don't want to be limited by available ram when all you need to do is enumerate a csv.
- Izkata 6y ago
  The article did not imply that: > """with a different file format""" The assumption behind the paragraph following the example does not imply lazy evaluation, but rather the opposite - the broken one is obvious if you have lazy evaluation in mind, whereas the docstrings imply the caller would be using these the same and not be expecting lazy evaluation.
compressedgas 6y ago
Or you can go the other way and force the caller of the load function to ensure that the iterator is closed Pass the responsibility as far up as required. Make the `load` function return a context manager. So any use of load would be like: with load(x) as it: pass `load` could be written with `contextlib.contextmanager` as: @contextlib.contextmanager def load(filepath): with open(filepath) as f: yield csv.reader(f)

BeefySwain 6y ago

Many good points are made here, but I would point out that there is a significant difference between working with JSON and CSV: data that can be operated on can typically be extracted from a CSV by working iteratively over each line. The same can not be said about JSON files. It is normal and (almost always ?) required to parse/de-serialize an entire JSON file at once and hold the contents of it in memory, while the same is often impractical and unnecessary for a CSV (when it is millions of lines). The standard methods and idioms of doing these things in Python reflects this.

dragonwriter 6y ago
> It is normal and (almost always ?) required to parse/de-serialize an entire JSON file at once and hold the contents of it in memory It may be normal, but there's no reason it is required. Streaming JSON processing makes sense just like streaming XML processing.
tomjakubowski 6y ago
It's not uncommon to see files or streams in "JSON lines" format. For files where each line is a record and the records are sorted or indexed, it's possible to seek to a record of interest just as in CSV. http://jsonlines.org/ http://jsonlines.org/
- coldtea 6y ago
  Those files are individual JSON entries though, as per the spec. They're not read as a JSON (since they aren't) they are read per line, and each line is parsed as a JSON.
  - orf 6y ago
    Same as CSV then?
    - coldtea 6y ago
      Yes, but not the same as JSON. Which was my point. In other words, collection of JSON strings in a file != a JSON file.

j88439h84 6y ago

If you're reading the whole file at once, I'd use `Path().read_text()`.

eximius 6y ago

The point of a file context manager is to close the file when you're done with it. The CSV reader isn't done with it. Given that the CSV reader does not control ownership of the file handle, it shouldn't return a lazy iterator when given a file handle. It should accept a path, then it could own the file handle until the iterator is complete or dropped. So, yea, the CSV reader does a bad. But so what? That is a deficiency of the CSV library, not context managers.

dmurray 6y ago
The CSV library is part of the standard library, though. Its way of doing things has been blessed by the Python maintainers. OK, it might still be correct to say the problem is with the CSV library. But if the context manager "best practices" lead to popular parts of the standard library being bad and unidiomatic, that's at best unpythonic. After all, "There should be one -- and preferably only one -- obvious way to do it".
- arugulum 6y ago
  By the Python standard library already does contain things that don't follow best practice. See: logging doesn't abide by PEP 8.
throwaway894345 6y ago
Libraries that accept filepaths (except perhaps as a convenience) are bad. The CSV library is well-designed. It isn’t responsible for closing files because it doesn’t take files, it takes TextIO. None of this is to say context managers are bad; I think they’re great but they only work as intended.
- solidasparagus 6y ago
  That's being a little pedantic IMO. From a practical, user-facing perspective, the CSV library Reader takes in files. And they know this, which is why the documentation specifically calls out gotchas related to passing in a file object and the example code shows how it should work with a file object [1]. [1] https://docs.python.org/3/library/csv.html#csv.reader https://docs.python.org/3/library/csv.html#csv.reader

nemetroid 6y ago

> But in all seriousness, this advice is applying a notably higher standard of premature optimization to file descriptors than to any other kind of resource. I don't think it's premature, but properly closing files is definitely not about optimization. Maybe "a notably higher standard of code quality"?

shawnz 6y ago
How does it positivity affect code quality? It seems to me like it would be the opposite, automatic closing means less boilerplate and more expressive code
- tcbasche 6y ago
  By automatic closing do you mean not at all and expecting the GC to do it? Or with a context manager. I think trusting an implementation detail of the underlying interpreter is lazy at best and an anti-pattern at worst. We should aim to show our intent as much as possible, and closing a file or using a context manager certainly expresses our intent.
  - shawnz 6y ago
    I mean imagine a hypothetical language where it was idiomatic to let the GC close files for you and the language was specified in such a way that it was guaranteed to work efficiently and wasn't just an implementation detail. Would code in that language have lower code quality than code in a language where files have to be closed manually? That's what the other poster seemed to be implying, but I think the opposite is true
    - nemetroid 6y ago
      I don’t quite see what you mean by ”efficiently”. A language where garbage collection of file objects is guaranteed to happen within N milliseconds? The issue is holding on to an external resource for longer than needed. and by definition, garbage collection is deferred cleanup.

wwright 6y ago

IMO part of what’s happening here is that Python’s GC model works well for memory and leaks for other resources. Rust’s ownership model works very well for many non-memory resources IMO, and even for memory it is very safe and efficient, with the only cost being mental overhead at times. Something like Rust’s ownership system but with an easy escape hatch for “loose” resource management (ie GC) could solve a lot of the cognitive load with resources, I think. Rc is helpful, but not nearly as ergonomic as a GC language. I think withoutboats had a similar blog post a year or two ago.

gizmo686 6y ago
I'm not sure it is fair to attribute this to Python's GC model, in that what is commonly thought of as garbage collection is not involved with the issue being discussed. However, in principle, this issue is very simmilar to that which garbage collectors set out to solve. In effect, python's context managers combined with the "with" syntax recreate the sementics of C's local variables in that: 1) They can be passed outside the lexical scope in which they were defined but 2) They're usage is illegal after execution has exited the "block" in which they are defined. 3) There is nothing which prevents a programmer from keeping and attempt to use a reference to them after execution has left the code block. The approach I take in python is the same I take in C. Most of the time, I only need to file for a short period, so I can "stack" allocate it with a context manager and let the runtime handle closing it for me. In cases where stack-like sementic do not fit, you need to use heap-like semantics and manually open/close it.
- wwright 6y ago
  If you’re not familiar with Rust, one of the core features is that you can “pass” or “borrow” that stack-like scoping in a safe way. IE, if you create a file handle, it gets automatically cleaned up when the current scope ends… unless you transfer ownership to other code, and then it is cleaned when THAT scope ends. It also prevents references from dangling after the resource has been cleaned, even in other classes or functions. This lets you model ownership rather than memory, and the compiler will translate that (for the most part) to memory management for you. What I was trying to say is that I think managing ownership as a first-class language feature is more generally useful than trying to abstract away the concern with something like GC.

Skunkleton 6y ago

There are lots of reasons to close a file when you are done with it, especially if you are writing to that file. These considerations are not about “pythonicity”, but rather about best practices for unix.

dragonwriter 6y ago

> Any resource may be limited or require exclusivity; that's why they're called resources. Similarly one should always explicitly call dict.clear when finished with a dict. I think there is an important distinction between resources for which there may be contention within the process, vs. external resources. It's tractable to provide sufficient certainty that in an internal resource isn't subject to problematic contention either analytically or by testing or a combination, but that's less tractable with external resources. So, use guaranteed freeing of resources (context managers, if possible, in Python) for external resources or for internal resources that you can't otherwise be reasonably sure are free of contention issues.

lsorber 6y ago

As a counterexample: we did actually run out of file descriptors on Lambda by not closing file descriptors.

viraptor 6y ago

I'm lost between the no-true-scotsmen and strawmen the post tries to construct. The advice about closing the descriptors is for practical reasons. There's no need for "true believers" taking that advice to extreme. Just like clearing the dict if you're dropping the reference anyway is useless. If you can consume the file immediately, you can close it. If you're doing lazy iteration you have to keep it open, and I don't understand why that's an issue.

Someone 6y ago
“If you're doing lazy iteration you have to keep it open, and I don't understand why that's an issue.” It isn’t an issue if you process a few files that way (unless you’re running your system very close to the limit), but it is an issue if you have a long running process that lazily iterates over files often. Examples are iterating over a file system (say a kind of grep) or a web server where an often called entry point does this.
- viraptor 6y ago
  Sure. But I was getting at something different. You can choose your open files limit on one side and the approach (lazy or not) on the other. If you choose lazy there's no way around having to leave the file open.

superbatfish 6y ago

I'd like to emphasize the point even further for CPython users. Here's the author's closing sentence: >Your setup script is not going to run out of file descriptors because you wrote open("README.md").read(). In CPython, files are closed (if necessary) in __del__, so the descriptor is released immediately after that line executes, thanks to CPython's reference-counting mechanism. That idiom wouldn't cause you to run out of descriptors even in a long program. (The author knows this. Their point is that even if you're NOT using CPython, you STILL shouldn't worry about it too much, and I think I agree.) I know, I know, CPython's reference-counting behavior is considered an "implementation detail", but for RAM-intensive workloads which are common in Scientific Computing (especially biological imaging), it's often a necessity. So if I were to force myself to use context managers in cases like this, I'd be losing the benefits of terse code without actually preserving portability. (I didn't have portability in the first place.) In other words, if you're dependent on ref-counting anyway, you may as well admit it and enjoy the benefits that come with it!

orf 6y ago
It won’t be released immediately, that’s the point. It will be “released at some point in the future and definitely not during interpreter shutdown”. A “with” statement ensures the release happens deterministically, which it turns out is a desirable trait to have.
- superbatfish 6y ago
  The Python standard makes no guarantee, but in CPython, the example above is deterministic. The cycle-detecting garbage collector is not involved in this case. Just refcounting.
  - james412 6y ago
    If open() throws an exception, perhaps after the FD is already open (Python does non-trivial IO to a file immediately after open), or read() throws an exception, the file object is at the mercy of any try/except higher in the stack, any bugs/"features" contained in those, and any sys.excepthook installed like Sentry The example above is deterministic only on current versions of CPython and only if it is guaranteed to succeed and if it is the only line of code in the program now and in perpetuity.

james412 6y ago

Horrendous advice on many levels - reference counting is an implementation detail, it is not part of the spec - that's because real implementations exist where refcounting does not exist (ironpython, pypy, jython) - even if it were part of the spec, the schedule for object destruction also is not part of the spec. Your file object was noticed as part of an unused graph at T+0, even mainline CPython can perpetually defer its destruction to T+infinity due to its handling of reference cycles - the random system resource will eventually be garbage collected, but perhaps not if your code throws the wrong exception and/or one of those cute syntax sugar libraries takes a reference to its frame object you know nothing about. Or how about pretty much any debugging support library like Sentry? Best to have audited them all.. - the random system resource was garbage a few tens of opcodes ago, but you left it locked, and some future change caused your code to immediately attempt to reopen and relock it. surprise, deadlock in production The comparison with 'dict.clear()' makes no sense whatsoever because explicit external resource control is precisely required because those external resources aren't automatically managed like normal in-memory objects. They have material external state beyond whether they are allocated or deallocated, and require explicit initialization and de-initialization in order to avoid leaving them in unspecified states. Python's memory allocator provides none of these behaviours as guarantees. It's not even guaranteed to always call __del__, this is most commonly observable during interpreter shutdown. Perhaps a better way of looking at it is this: try/finally and with: exist for you to demand stronger guarantees from the runtime than it normally offers you. "Hey Mr. Python, I know you'll /probably/ clean up this object eventually, but I also know there are 100 edge cases where you won't bother. For this object, you are on the hook for every case, including the cases where I wrote buggy code" Meanwhile, feel free to follow this advice, and good luck at 4am debugging your prod system that for some inexplicable reason recently began crashing due to running out of file descriptors..

epage 6y ago
Another problem: Windows support. Windows' file semantics are different and knowing when a file is closed becomes critical.
- gruez 6y ago

bassman9000 6y ago

_The csv version returns an iterator over a closed file_ That's why method/function naming is critical and underestimated. Without having worked with python too much: _json.load()_ is a load: implies the action is done on return. I understand there's nothing to do after it, other than use the return value. _csv.reader()_ clearly returns an object that it's going to be used to traverse the underlying, so the underlying must remain open. Implies a stateful operation to me. Besides, examples are clear: https://docs.python.org/3.8/library/csv.html#examples https://docs.python.org/3.8/library/csv.html#examples Classical problem of wanting to use The Perfect Abstraction.

dragonwriter 6y ago

> Moreover, according to this best practice, it's impossible to write the csv version correctly. No, it's not: def load(filepath): """with a different file format""" with open(filepath) as file: return list(csv.reader(file)) This both uses csv.reader and fulfills the required contract correctly. Yes, it's tricky to both follow the “assure a resource is closed” pattern and return a lazy iterator over the external resource which you open. If you are going to lazily iterate over a resource and do the context manager pattern, you have to do the work within the context manager.

deathanatos 6y ago

IMO, the author is looking at the central problem with GC systems: non-deterministic destruction, and unclear ownership. > This trivial example has deeper implications. If one accepts this practice, then one must also accept that storing a file handle anywhere, such as on an instance, is also disallowed. Unless of course that object then virally implements it owns context manager, ad infinitum. Yes, and I just accept that. This happens in C#, too[1]: > If your class owns a field or property, and its type implements IDisposable, the containing class itself should also implement IDisposable. A class that instantiates an IDisposable implementation and storing it as an instance member, is also responsible for its cleanup. And for good reasons; in this regard, the languages are basically identical. I'd say any language that uses GC and instead has some sort of language-based guard system (Python's `with`, C#'s `using`) or even just through convention (e.g., Java, back before try-with-resources) will end up having that happen. Depending on the GC to collect these resources isn't wise, since no behavior on the GC is actually guaranteed. > No, it will not be fixed. This version only appears to work by not closing the file until the generator is exhausted or collected. I'd call that fixed; the file is properly and deterministically closed, when the generator is exhausted or manually closed. (By calling the generator's `close()` member function.) Now, it is now up to the calling code to remember at every use site to do that, and it should be made an explicit part of the documentation for the interface. But, the same for context managers, or things having a close: it's part of the contract that you will call that .close(), or wrap it in a with block, or the resource leaks until the GC gets to it. (I do wish generators were context managers, to make this easier.) IME, the bigger problem here is that GC forces, for resources that have a destructor beyond "free the memory" (that is, what the GC itself handles), for each call-site to remember to clean up the resource. Files are one thing, and generally, yes, you'll probably get lucky and not run into issues. Locks are more fun, and often leaving it to the GC to release a lock is a lot more painful. (Then there are threads, child processes, sockets, …) RAII fixes this for all types of resources by bounding the lifetime of the resource to the scope(s) that it lives in. (And, because it inevitably comes up, IME, people worry too much about cycles; cycles do happen, but are incredibly rare compared to non-memory resources (those that are more than just memory, e.g., files, locks, etc.), and vanishingly rare compared to all resources (those that are more than just memory and those that are just memory, like vectors or strings). Yes, they are an issue of sorts in RAII, but there are ways to handle them. I do not think the bad with cycles outweighs the good of RAII.) [1]: https://docs.microsoft.com/en-us/dotnet/standard/garbage-collection/implementing-dispose#cascade-dispose-calls https://docs.microsoft.com/en-us/dotnet/standard/garbage-col...

deleted 6y ago

[deleted]

dvt 6y ago

> The csv version returns an iterator over a closed file. It's a violation of procedural abstraction to know whether the result of load is lazily evaluated or not; it's just supposed to implement an interface. Moreover, according to this best practice, it's impossible to write the csv version correctly. As absurd as it sounds, it's just an abstraction that can't exist. This is simply not true. A load being lazy or not will often be a part of API documentation. In some cases, documentation will include that a .close() needs to be manually called. It's also trivial to read an entire CSV file, close it, and return an array of lines, or a dataframe, or whatever. Of course, you probably don't need to do it with a generator, and it won't be a two-liner (although @Izkata gives a two-liner example). In that implementation, you'd achieve behavior parity with json.loads(...). > No, it will not be fixed. This version only appears to work by not closing the file until the generator is exhausted or collected. I'm no fan of Python (even though I use it every day), but this is extremely well documented in PEP 342[1]. I'm honestly not sure what the complaint is here. I guess the standard Python CSV library is a bit wonky, but that's about it. [1] https://legacy.python.org/dev/peps/pep-0342/ https://legacy.python.org/dev/peps/pep-0342/

ehutch79 6y ago

This is like a worse version of javascript semicolons. Sure you dont HAVE to... but there's gotchas and corner cases to not doing it that you don't have to think about if you close your file desciptors when you're done with them

Camillo 6y ago

This is an actual drawback of not having deterministic destruction. This would not be a problem in C++ or Swift.

The_rationalist 6y ago
Exactly. Yet Java has autocloseable without having deterministic destruction
- masklinn 6y ago
  AutoCloseable is the same as Python's __exit__, it exists as a hook to try-with-resource in the same way __exit__ is a hook for `with`.

Animats 6y ago

This is to some extent a Python-specific problem. In most languages with destructors, you're guaranteed a destructor call when a local variable's scope is exited. But not Python. Python isn't RAII, it's an explicit close language. That's what "with" is for. Python's "with" clauses, unusually, play well with exceptions at scope exit. Nested "with" clauses can fault out with a close exception and still have everything unwind properly. Very few languages get that right. Python probably has the most useful exception system around. Calling destructors from the garbage collector has a whole other set of problems. Suppose you're writing to a network and the network connection fails during close. Now you can be stuck in the garbage collector calling the destructor waiting for the network to time out. I'm not sure how garbage-collected Python implementations handle this. C# can potentially get into that state, too. Anybody know? Raising an exception in a C++ destructor is usually not happy-making. Microsoft tried to make that work in Managed C++, which has "re-animation". A destructor can pass a reference to its own object outside of the destructor, which "re-animates" the object. Turned out not to be a good idea. I would have liked to see exceptions in Rust, which has enough static ownership analysis to determine whether a destructor is trying to make something outlive its lifetime. Most of the problems with destructors vs exceptions involve scoping issues. If such errors are caught at compile time, exceptions are better behaved.

tsimionescu 6y ago
In most managed-memory languages, try...finally (and using/try-with-resources which are based on it) handle exceptions just fine. The only problem that sometimes occurs is that exceptions thrown by a finally handler override the exception being bubbled, which can be bad if you wanted to handle that exception later on, but otherwise the resources get cleared properly. D has better behavior here, where throwing an exception on this path will automatically keep track of many in-flight exceptions. Also, the problem of finalizers blocking is partly handled by the fact that finalizers are not run by the GC threads, but by a separate finalizer thread. Still, if a finalizer blocks, eventually you will run out memory, as all objects which require finalization will be pooling up. Also, the re-animation behavior you describe is common to finalizers in all GC languages which expose them, it's not in any way specific to Managed C++ (C++/CLR). It is simply a consequence of how finalization works: when an object which requires finalization is eligible for GC, it is not collected, but moved to the finalization queue. Objects in that queue are still live objects. When the finalizer thread successfully runs the finalizer of such an object, it is simply removed from the finalization queue. If it's finalizer has added back a reference from some other GC root to the object, then it will still stay live. The only special property is that an object can't be finalized twice, so even if its finalizer attaches it to the live object graph once, when it becomes unreachable again it will be collected, it will not be put back in the finalization queue, it will be directly collected. Basically, I don't think there is too much chance to find a valid use for this behavior, but it is very hard to prevent it without complicating the language significantly.

lmm 6y ago

> But in all seriousness, this advice is applying a notably higher standard of premature optimization to file descriptors than to any other kind of resource. Maybe the same standard should be applied to other kinds of resources. Maybe garbage collection was a mistake. > It's a violation of procedural abstraction to know whether the result of load is lazily evaluated or not; it's just supposed to implement an interface. Lazily-loaded things are different from eagerly loaded things in important ways. If it's not possible to make that distinction visible in your language, get a better language. > No, it will not be fixed. This version only appears to work by not closing the file until the generator is exhausted or collected. Which is exactly what we wanted. It is fixed. > Furthermore it demonstrates that often the context is not being managed locally. If a file object is passed another function, then it's being used outside of the context. Let's revisit the json version, which works because the file is fully read. Doesn't a json parser have some expensive parsing to do after it's read the file? It might even throw an error. And isn't it be desirable, trivial, and likely that the implementation releases interest in the file as soon as possible? Look at iteratees for the full solution to this problem. Managing context can be hard, but it doesn't have to be; you just have to use the right abstractions. > Those threads are essentially debating whether a resource pool is "leaking" resources. If you open a pool and never close it, of course you're leaking everything in that pool. > if all this orphaned file descriptor paranoia was well-founded, it would have been a problem back then too. It was. Long-running python programs broke because they ran out of file descriptors.

perfunctory 6y ago

> Your setup script is not going to run out of file descriptors because you wrote open("README.md").read(). How about even simpler (python3) Path("readme.md").read_text() Ta-da. I don't even have to worry about opening files, let alone closing.

xenonite 6y ago
To complete your comment, let us import pathlib. from pathlib import Path Path("readme.md").read_text() The functions read_text and the similar write_text, read_bytes, write_bytes are new in Python 3.5. https://docs.python.org/3/library/pathlib.html#pathlib.Path.read_text https://docs.python.org/3/library/pathlib.html#pathlib.Path....
- pansa2 6y ago
  I wasn’t aware of these functions. At first glance, their addition to Python seems to be a significant violation of “one-- and preferably only one --obvious way to do it”. What are the advantages of `read_text` over the traditional `file` read methods?

patrec 6y ago

GC advocates say that managing memory lifetimes is an excessive burden to programmers. Anti-GC advocates say that memory is just one particular instance of management of life-timed resources, and that special casing it is an ugly hack and you should be using RAII. Rust probably is the leading exponent (C++'s way is more of a mess). The RAII critics point out that RAII also glosses over part of the problem because often de-allocating resources can itself fail (e.g. closing a file) and RAII (in Rust or C++) has no proper way to deal with that. Then, there is the Reference counting camp, the only people who can say that they solve the something close to the general automatic resource de-allocation problem -- because lifetimes are precisely tracked. But reference counting can't cope with cycles and generally has poor performance. Also, it does not directly help with de-allocation failures. One interesting recent variation are attempts at static reference counting: http://strlen.com/lobster/ http://strlen.com/lobster/. Last there are the old and the new muddle through camp. The old muddle through camp (e.g. java, python) adds finalizers with crap semantics (call-me-maybe). The new muddle-through camp uses GC for memory but adds some extra mechanism for other resources (defer in go, <close> in lua 5.4). Defer in go seems like an inferior version of rust-style RAII. Lua 5.4's <close> also has/had problems (see http://lua-users.org/lists/lua-l/2019-06/msg00179.html http://lua-users.org/lists/lua-l/2019-06/msg00179.html). The author of the article advocates old-style muddling through, but this feels unsatisfactory because it's not robust in general and likely worse than following current "best practice". But it still seems like there is still no really good general solution available. Rust-style ownership tracking would have solved the lifetime ambiguity of the file for the csv reader, but would not be able to deal with file close errors properly. These can be somewhat handled with exception based mechanisms, but not without problems either -- because multiple things can go out of scope at once, requiring cleanup and one or more of these cleanups may fail themselves but try catch statements (with stack unwinding) offer no natural way to deal with multiple failures being propagated up at once.