6 ms·
On the Beauty of Python's ExitStack (2015)
- ctur 6y agoIt's beauty in the sense that it is addressing a language deficiency ("destructor" behavior, awkwardness of the "with" construct) with a library, sure, but this is ultimately working around a gap in Python itself.
- xapata 6y agoI'd argue the context manager concept is better than destructors, because it's more explicit. BTW, Python does have the `__del__` special method. It's just that almost all usage of it would be better implemented via context manager.
- bobbyi_settv 6y agoI think the idiomatic way is with weakref https://docs.python.org/3/library/weakref.html https://docs.python.org/3/library/weakref.html Particularly note "finalize provides a straight forward way to register a cleanup function to be called when an object is garbage collected"
- ctur 6y agoThat's a little snarkier than I intended. It is good to share this method of handling what is otherwise very awkward code. It just is unfortunate the language itself doesn't have constructs that lend to proper syntactically scalable resource handling other than nested "with" blocks.
- MereInterest 6y agoAgreed. I like Python, and I think ExitStack is a fantastic way to manage resources. However, that is because it most closely emulates RAII in C++. The limitation that I still run into is when a class needs to have some cleanup code added to it. In C++, you just add a destructor. In Python, you are forced to choose between non-deterministic release of resources through __del__, or changing the interface by requiring users to use a "with" block. Neither of those is a particularly satisfying option.
- xapata 6y agoThe need to add cleanup code is a change to the interface. If you don't change the interface, you've left yourself open to bugs around accidentally using an instance after it's been cleaned. If you really want to keep the same interface, add a layer of abstraction. The current interface becomes a wrapper around the underlying context manager.
- MereInterest 6y agoExactly. The language design of Python means that the addition of cleanup code is an API change. The language design of C++ means that the addition of cleanup code is not an API change.
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]
- xapata 6y agoI actually appreciate that API change, as it makes my use of resources more explicit. However, you might be interested in using ``weakref.finalize`` to keep that C++ style. https://docs.python.org/3.6/library/weakref.html#comparing-finalizers-with-del-methods https://docs.python.org/3.6/library/weakref.html#comparing-f...
- epage 6y agoimo the main benefit to context manager is reporting errors during cleanup while the main benefit of RAII/destructors is composability.
- xapata 6y agoI'd go a little further and say context managers are underused on the whole. They're a great tool and aren't used often enough. ExitStack enables using context managers in more situations.
- toxik 6y agoProblem with context managers is that they add up indentation really, really fast. I guess TFA is at least a partial remedy to this though!
- redrum 6y agoTFA?
- drdrey 6y agoThe Fine Article, as posted by OP
- lunixbochs 6y agoYou can do multiple in a single with statement these days.
- BerislavLopac 6y agoThis is an often overlooked feature of context managers; to clarify, this is a perfectly normal example: with open("foo.txt") as foo_file, open("bar.txt") as bar_file, open("baz.txt", "w") as baz_file: baz_file.write(foo_file.read() + bar_file.read())
- lunixbochs 6y agoI think you can also do something like this if you want it to be readable with (open("foo.txt") as foo_file, open("bar.txt") as bar_file, open("baz.txt", "w") as baz_file): baz_file.write(foo_file.read() + bar_file.read())
- davidwf 6y agoI write a lot of Python and didn't know about this despite it being a 2015 blog post! The real kicker for me is: * [ExitStack] easily scales up to many resources (including a dynamic number that's acquired in a loop)
- jnwatson 6y agoIn particular ExitStacks are useful when you have 0 or 1 resources. It allows a sort of null-context manager for when the number of resources you have to allocate might be 0.
- hchasestevens 6y agoDo be aware that https://docs.python.org/3.9/library/contextlib.html#contextlib.nullcontext https://docs.python.org/3.9/library/contextlib.html#contextl... also exists as of 3.7 -- you can do ctx_mgr = foo() if bar else nullcontext() with ctx_mgr: ... if you like.
- BerislavLopac 6y agoOr even, from Python 3.8: with (ctx_mgr := foo() if bar else nullcontext()): ...
- whateveracct 6y agoI never knew about ExitStack. It reminds me of Haskell's ResourceT.
- jnwatson 6y agoHere's a good example IRL. https://github.com/vertexproject/synapse/blob/master/synapse/tools/backup.py#L56 https://github.com/vertexproject/synapse/blob/master/synapse... It grabs a transaction on an arbitrary number of databases, then yields.
- Groxx 6y agoThough I very much like that this exists, and the documentation for contextlib describes its need and use very well[1], I do want to point out that this has always been possible with a tiny bit of recursive glue code like this[2], which works fine even in python 2: @contextmanager def multi(*cms): if len(cms) > 0: with cms[0]: with multi(*cms[1:]): yield else: yield # use with 0..N items, it behaves the same with multi(...): pass Prior to finding ExitStack, I've frequently seen people either give up and do something obviously wrong, or try to manually manage their own stack of __enter__ and __exit__ funcs (which works, but exception handling is non-obvious and often loses useful info). Personally I prefer the recursive route, since it very obviously behaves the same as if you wrote the nesting out by hand. It even retains accurate stack traces "naturally", unlike most non-ExitStack tactics I've seen through the years. ExitStack's stack-correcting tactic is... a bit more "interesting"[3], though AFAIK it achieves roughly the same stack trace in the end. Since ExitStack exists, and is likely more efficient (by how much, I have no idea), it's probably a better choice for most situations. And it's just broadly a bit nicer for things that yield resources. It's always worth browsing stdlib documentation to find gems like this, there are quite a few neat things in most languages that seem fairly unknown. [1]: https://docs.python.org/3/library/contextlib.html#supporting-a-variable-number-of-context-managers https://docs.python.org/3/library/contextlib.html#supporting... [2]: https://repl.it/@Groxx/TepidRosybrownLocatorprogram#main.py https://repl.it/@Groxx/TepidRosybrownLocatorprogram#main.py [3]: https://github.com/python/cpython/blob/master/Lib/contextlib.py#L564 https://github.com/python/cpython/blob/master/Lib/contextlib...
- hchasestevens 6y agoI always thought it was a shame that `nested`[0] (the stdlib equivalent to your `multi` above) was deprecated in 2.7 and, ultimately, removed in favor of `ExitStack` -- despite its flaws, I think it was more beginner-friendly. [0]: https://docs.python.org/2.7/library/contextlib.html#contextlib.nested https://docs.python.org/2.7/library/contextlib.html#contextl...
- Groxx 6y agoYeah, and the source for `nested` is a lot easier to follow than ExitStack. Conceptually it's the same as mine, but with a loop rather than recursion, and it'll yield values (not hard to do with a recursive one, I just didn't feel like it): https://github.com/python/cpython/blob/2.7/Lib/contextlib.py#L89 https://github.com/python/cpython/blob/2.7/Lib/contextlib.py... Seems like the __new__/__init__ issues exist with ExitStack as well, unlike with `with a, b, c`? So it's not really resolved / improved behaviorally, though ExitStack does have a few new uses (e.g. managing non-context-manager resources).
- roca 6y agoThe comparison with other languages seems to miss the fact that the idiomatic approach in Rust and C++ is RAII, which does let you put the cleanup code next to the allocation code.
- gregopet 6y ago..and that Java has, like the object oriented language that it is, an AutoCloseable interface which you can implement and then use in blocks, much like with this Python construct.
- cbsmith 6y agoIn fairness, Java's AutoCloseable interface is missing some of the niceties of ExitStack, as with C++'s destructors. People have made libraries for both that attempt to close the gap.
- tathougies 6y agoIn Haskell, this is done with the Managed monad or ResourceT if you're feeling impure.
- deleted 6y ago[deleted]
- nyanpasu64 6y agoIt's ExitStack.enter_context(), not enter(). https://docs.python.org/3/library/contextlib.html#contextlib.ExitStack.enter_context https://docs.python.org/3/library/contextlib.html#contextlib...