3 ms·

The "late‐binding closure" pitfall in Python

5 points by behnamoh 2y ago

rzzzwilson 2y ago
> the surprising part about closures is that they capture variables by reference, not by value Not too surprising when you realize python objects are accessed by reference. Zero, one or many names can be bound to that reference. The options when binding an environment to a closure are to copy the current environment with the current values, or to just save a reference to the current environment. That latter approach is used everywhere else in python. Assignment never copies, for example, so that's what happens with closures. The workaround that involves creating a helper function seems much too heavy an approach. Just force evaluation of the formal parameter in the environment: fns = [] for i in range(3): def f(i=i): # force evaluation of parameter print(i) fns.append(f) for f in fns: f()
- Terr_ 2y ago
  *squints* I assume that relies on a different Python newbie-pitfall, which is that default values for parameters all point to the same thing present at the time the def is processed? # Don't do this! The same initially-empty array is #shared across all no-arg calls. def foo(items = []): ... # Better, because even though None is still shared # it immutable so nobody cares. def foo(items = None): if x is None: x = [] ...
  - Doxin 2y ago
    yes.
  - zahlman 2y ago
    It does, and I've always thought it was a bad way to address the problem for that reason. Using `functools.partial` is far clearer and more explicit as a way to bind parameters to a call: https://stackoverflow.com/questions/277922 https://stackoverflow.com/questions/277922
    - Terr_ 2y ago
      Yeah: fns = [functools.partial(print, i) for i in range(3)] for f in fns: f()
  - rzzzwilson 2y ago
    My suggested workaround has nothing to do with mutable default parameters (the "items = []" code). It is true that a default parameter value is evaluated at "def" evaluation time, but that's how python works for all default parameter values. It's a trap only if that default value is mutable and the programmer thinks that the default value is created at function call time. Your code examples, though I agree with what you are saying, aren't really anything to do with what I posted. The effect of what I wrote is to bind the parameter to the value of the argument at "def" evaluation time, which is what most people want. The advice "Don't do this!" is fine for beginners, but you need further explanation: with more experience a programmer might actually use the mutable default parameter behaviour to advantage.
niemandhier 2y ago
This is a pitfall for many people when starting to work with async code. Pep 567 introduced contextvars to address it in this scenario.
zahlman 2y ago
Of course, it's always worked this way, and has been discussed on Stack Overflow more or less the whole time - e.g. https://stackoverflow.com/questions/3431676 https://stackoverflow.com/questions/3431676 , https://stackoverflow.com/questions/2295290 https://stackoverflow.com/questions/2295290 , https://stackoverflow.com/questions/233673 https://stackoverflow.com/questions/233673 .

5 points by behnamoh 2y ago

> the surprising part about closures is that they capture variables by reference, not by value Not too surprising when you realize python objects are accessed by reference. Zero, one or many names can be bound to that reference. The options when binding an environment to a closure are to copy the current environment with the current values, or to just save a reference to the current environment. That latter approach is used everywhere else in python. Assignment never copies, for example, so that's what happens with closures. The workaround that involves creating a helper function seems much too heavy an approach. Just force evaluation of the formal parameter in the environment: fns = [] for i in range(3): def f(i=i): # force evaluation of parameter print(i) fns.append(f) for f in fns: f()

Terr_ 2y ago
*squints* I assume that relies on a different Python newbie-pitfall, which is that default values for parameters all point to the same thing present at the time the def is processed? # Don't do this! The same initially-empty array is #shared across all no-arg calls. def foo(items = []): ... # Better, because even though None is still shared # it immutable so nobody cares. def foo(items = None): if x is None: x = [] ...
- Doxin 2y ago
  yes.
- zahlman 2y ago
  It does, and I've always thought it was a bad way to address the problem for that reason. Using `functools.partial` is far clearer and more explicit as a way to bind parameters to a call: https://stackoverflow.com/questions/277922 https://stackoverflow.com/questions/277922
  - Terr_ 2y ago
    Yeah: fns = [functools.partial(print, i) for i in range(3)] for f in fns: f()
- rzzzwilson 2y ago
  My suggested workaround has nothing to do with mutable default parameters (the "items = []" code). It is true that a default parameter value is evaluated at "def" evaluation time, but that's how python works for all default parameter values. It's a trap only if that default value is mutable and the programmer thinks that the default value is created at function call time. Your code examples, though I agree with what you are saying, aren't really anything to do with what I posted. The effect of what I wrote is to bind the parameter to the value of the argument at "def" evaluation time, which is what most people want. The advice "Don't do this!" is fine for beginners, but you need further explanation: with more experience a programmer might actually use the mutable default parameter behaviour to advantage.

niemandhier 2y ago

This is a pitfall for many people when starting to work with async code. Pep 567 introduced contextvars to address it in this scenario.

zahlman 2y ago

Of course, it's always worked this way, and has been discussed on Stack Overflow more or less the whole time - e.g. https://stackoverflow.com/questions/3431676 https://stackoverflow.com/questions/3431676 , https://stackoverflow.com/questions/2295290 https://stackoverflow.com/questions/2295290 , https://stackoverflow.com/questions/233673 https://stackoverflow.com/questions/233673 .