48 ms·

Understanding a Python Closure Oddity

67 points by ingve 2y ago

hanglong10 2y ago
Further proof that Python is not fit for functional programming and that Python's namespaces and default lookup by reference are confusing. In safe languages like ML you would have to create an explicit reference to get that effect, and then it is obvious to the reader. Python is not built for complex abstractions, but unfortunately it is (ab)used for that, particularly in machine learning.
- mharig 2y ago
  No. The code is completely consistent: the closures are different instances of a function that returns the value of 'number' at the moment it gets called. Not the value of 'number' when it gets created. And it gets called differently when a list comprehension calls it each step of the loop or when the list function loops completely through and then the list comprehension calls the instances. Nothing to see here.
  - akfred 2y ago
    > No. The code is completely consistent: the closures are different instances of a function that returns the value of 'number' at the moment it gets called. Not the value of 'number' when it gets created. Isn't that exactly what GP said? And gave an example how to create that effect explicitly in functional languages, using refs?
    - mharig 2y ago
      Yes. But I found the explanations of the GP a little misleading, especially the eager/lazy part.
  - PurpleRamen 2y ago
    It's consistent, but unexpected. Especially if you don't know python well enough. And it leads to ugly effect sometimes, even for experienced pythonistas.
    - topspin 2y ago
      Yes, this bit me last week. I stumble over it about once a year. Recently encountered it in an open source project and filed an issue. I don't know where I fall on the "experienced" spectrum, but I've written a lot of python in the last 10 years.
  - bobbylarrybobby 2y ago
    Naively I would expect that each iteration of a for loop creates a new loop variable (all with the same name, but effectively each in their own scope) and so each closure holds a reference to a variable named number whose value never changes, and there is a distinct one of these number variables per closure .
    - mharig 2y ago
      That is not how variables in Python work. Variable assignment is just name binding. This is clearly mentioned in every Python beginner tutorial I know of. Python code is a script. Follow the script and you know what Python does (with only a few exceptions, see e.g. below). About the scope: Python knows in principal only 2 scopes: local and global. If you define a function (or a class) Python creates a new emtpy/filled with the function arguments local scope (system table). That gets filled successively by local variable assignments etc. There happens a little bit of magic: if you use a variable of the next higher scope, it gets copied (!) to the local scope. You can follow that easily by print(locals()). (You can also use the global scope, but that is another topic). Edit: had to clarify some points
  - Y_Y 2y ago
    No.
- geysersam 2y ago
  If you want to "close over" the value in the closure then there's a simple trick you can use: Pass a keyword argument with the default value equal to the value you want to close over. Voila!
  - zbentley 2y ago
    Functions.partial can also supply captures in that way, and is slightly more obvious (and less likely to generate linter complaints if the kwarg is mutable).
    - zbentley 2y ago
      er, functools.partial, too late to edit.
- deleted 2y ago
  [deleted]
- d0mine 2y ago
  all one needs to understand is that there are names that refer to objects in Python. Nothing is passed by value. The closure returns whatever object the name refers at the moment. That is all. def loop(): for i in range(10): yield lambda j=i: j `j` refers to the same object (int) `i` refers to at the time of the lambda creation (the desired result). `j` is local to the lambda (each lambda has its own local namespace). Just `lambda: i` code would result in an object that `i` refers to at the time of calling the lambda (`i` name is not local to the lambda). https://docs.python.org/3/faq/programming.html#why-do-lambdas-defined-in-a-loop-with-different-values-all-return-the-same-result https://docs.python.org/3/faq/programming.html#why-do-lambda...
Spivak 2y ago
This is less wat. when you write it like eager: for item in loop(): item() lazy: items = [] for item in loop(): items.append(item) for item in items: item() Each next() into loop increments a shared counter so when you print it out as you go it 1, 2, 3 but when you print it out once you've called it a bunch is 9, 9, 9.
riffraff 2y ago
but is it a closure oddity? Looks to me like it's a generator/comprehension oddity :)
- amluto 2y ago
  It’s a closure oddity. The closure captures a reference to the variable, not the value of the variable at the time of its creation. Given that, the results are entirely predictable.
  - abecedarius 2y ago
    If we're really going to argue, I'd argue for a 'for' oddity. The for statement could've been defined with a fresh binding per iteration instead of a mutation.
    - Izkata 2y ago
      That would make it work differently than the entire rest of the language. Python is function-scoped, not block-scoped.
      - abecedarius 2y ago
        That ship already sailed when Python 3 made comprehension expressions (like list comprehensions) introduce a nested scope. They're even a kind of 'for' too. (Previously, [x for x in y] would clobber any x in the same function. Fixing this meant implicit nested scopes.) It's true that this would've been a further change (every loop entering the implicit nest, rather than entering one nest which then mutates per loop). But this design was a decision, not a necessity. JS's "for let" shows this by example.
    - b3orn 2y ago
      It's unrelated to the for loop, the for only changes the variable, there are other ways you can do that and you'll run into the same "oddity" def test(): n = 0 def a(): return n n = 1 def b(): return n return a, b a, b = test() a() == 1 b() == 1
sakex 2y ago
That's why Javascript introduced the "let" keyword years ago This logs from 0 to 9: for(let i = 0; i < 10; ++i;) setTimeout(() => console.log(i)); This logs from 10, 10 times: for(var i = 0; i < 10; ++i;) setTimeout(() => console.log(i));
- NwtnsMthd 2y ago
  Do you think you could elaborate on why the results differ? I'm somewhat unfamiliar with how JavaScript works.
  - wging 2y ago
    'var' is JavaScript's older variable declaration construct. Variables created this way are live from the beginning of the function that contains them (or globally if there isn't one). So a block with braces (such as you'd use for a for or while loop body) doesn't actually restrict the scope of var `v` below: console.log(v); // <-- v is a variable here, we can access its value even though it is only declared below // prints 'undefined' { var v = 1; console.log(v); // prints 1 } console.log(v); // prints 1 You used to (and might still) see a workaround to recover more restrictive scoping, known as the "IIFE" (Immediately Evaluated Function Expression): (function () { var v; ... code using v here ... })() creates a function (and thus a more restrictive scope for v to live in) and evaluates it once; this is a sort of poor man's block scoping. `let` and `const` were created to fill this gap. They have block scope and are special-cased in for loops (well, `let` is; you can't reassign to a `const` variable so nontrivial for loops won't work): console.log(l); // <-- throws ReferenceError: l is not defined { // pretend the console.log line above is commented out, so we can reach this line let l = 1; console.log(l); // prints 1, as expected } console.log(l); // throws ReferenceError: l is not defined // ^^ l was only in scope for the block above The interaction with `for` is explained well on MDN, including the special casing: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for#lexical_declarations_in_the_initialization_block https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe... "More precisely, let declarations are special-cased by for loops..." (followed by a more detailed explanation of why) See also https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/let https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe... and https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/var https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
peterhull90 2y ago
You can put an extra 'level' in, to make both options return the same 0,1...9 def loop(): for number in range(10): def outer(n): def inner(): return n return inner yield outer(number) Is there a neater way?
- nbadg 2y ago
  Yes. The usual way to do this is to bind them as defaults to an argument, for example: def loop(): for number in range(10): def func_w_closure(_num=number): return _num yield func_w_closure This works because default arguments in python are evaluated exactly once, at function definition time. So it's a way of effectively copying the ``number`` out of the closure[1] and into the function definition. [1] side note, closures in python are always late-binding, which is what causes the behavior in OP
  - NwtnsMthd 2y ago
    Does this work for user defined classes (objects) as well?
    - Jtsummers 2y ago
      Yes, and by the same mechanism and for the same reason.
  - amluto 2y ago
    I’ve never thought that leaking this type of implementation detail into the return value (and return type!) was a nice solution. I like the double closure better, and one can shorten it a bit with a lambda. For those who prefer a functional style, functools.partial can also solve this problem. (I use Python, and I like a lot of things about Python, but I don’t like its scoping rules at all, nor do I like the way that Python’s closures work. I would use a double lambda.)
  - deleted 2y ago
    [deleted]
crabbone 2y ago
I'll give you one better: >>> strings = tuple((lambda: f'{i}') for i in range(10)) >>> strings[1]() '9' The f-strings are supposed to be evaluated at... what time exactly? And some more stuff: >>> class Elephant: ... def __init__(self, i): ... self.i = i ... def __call__(self): ... return self.i ... >>> elephants = tuple(Elephant(i) for i in range(10)) >>> elephants[0]() 0 >>> elephants[5]() 5
- deleted 2y ago
  [deleted]
- postexitus 2y ago
  The answer is simple. f strings are evaluated at run time. The behaviour is correct. Up until you call the lambda, they are just a function definition. What's so interesting about the Elephant case? You store the value of i at run time, so it can be accessed later. lambda doesn't have that capability, as it's inheriting i from the block.
  - crabbone 2y ago
    > The answer is simple. f strings are evaluated at run time. You didn't understand the question. Everything in Python is evaluated at runtime. Bytecode is just a form of caching. > The behaviour is correct. Python doesn't have a standard. Nobody knows what behavior is correct. That's the only behavior Python has, so, again, calling it correct is just not saying anything. > What's so interesting about the Elephant case? This class, in principle, shouldn't be different from lambda from the previous example, yet it acts differently. You don't need to lecture me on why things behave the way they are. Since I found these examples, you can trust me, I know why they behave like this. The idea is to drive your attention to the inconsistencies that a "naive" reader would discover.
    - BerislavLopac 2y ago
      > Everything in Python is evaluated at runtime That is not correct. Python generally has two "runtimes": one is on import, when all of the module level code is evaluated; and the runtime, when the bodies of various functions and methods are evaluated. The two can be combined, e.g. by importing a module from within a function, but generally they are separate. > Bytecode is just a form of caching Not quite. It is perfectly possible to deploy just the bytecode and disable recompiling on import. > Python doesn't have a standard Of course it does -- CPython is the reference implementation, so its behaviour is "correct".
gpderetta 2y ago
Typical workaround: def loop(): for number in range(10): def closure(number=number): return number yield closure Many languages have surprising behaviour when closing over iteration variables.

67 points by ingve 2y ago

Further proof that Python is not fit for functional programming and that Python's namespaces and default lookup by reference are confusing. In safe languages like ML you would have to create an explicit reference to get that effect, and then it is obvious to the reader. Python is not built for complex abstractions, but unfortunately it is (ab)used for that, particularly in machine learning.

mharig 2y ago
No. The code is completely consistent: the closures are different instances of a function that returns the value of 'number' at the moment it gets called. Not the value of 'number' when it gets created. And it gets called differently when a list comprehension calls it each step of the loop or when the list function loops completely through and then the list comprehension calls the instances. Nothing to see here.
- akfred 2y ago
  > No. The code is completely consistent: the closures are different instances of a function that returns the value of 'number' at the moment it gets called. Not the value of 'number' when it gets created. Isn't that exactly what GP said? And gave an example how to create that effect explicitly in functional languages, using refs?
  - mharig 2y ago
    Yes. But I found the explanations of the GP a little misleading, especially the eager/lazy part.
- PurpleRamen 2y ago
  It's consistent, but unexpected. Especially if you don't know python well enough. And it leads to ugly effect sometimes, even for experienced pythonistas.
  - topspin 2y ago
    Yes, this bit me last week. I stumble over it about once a year. Recently encountered it in an open source project and filed an issue. I don't know where I fall on the "experienced" spectrum, but I've written a lot of python in the last 10 years.
- bobbylarrybobby 2y ago
  Naively I would expect that each iteration of a for loop creates a new loop variable (all with the same name, but effectively each in their own scope) and so each closure holds a reference to a variable named number whose value never changes, and there is a distinct one of these number variables per closure .
  - mharig 2y ago
    That is not how variables in Python work. Variable assignment is just name binding. This is clearly mentioned in every Python beginner tutorial I know of. Python code is a script. Follow the script and you know what Python does (with only a few exceptions, see e.g. below). About the scope: Python knows in principal only 2 scopes: local and global. If you define a function (or a class) Python creates a new emtpy/filled with the function arguments local scope (system table). That gets filled successively by local variable assignments etc. There happens a little bit of magic: if you use a variable of the next higher scope, it gets copied (!) to the local scope. You can follow that easily by print(locals()). (You can also use the global scope, but that is another topic). Edit: had to clarify some points
- Y_Y 2y ago
  No.
geysersam 2y ago
If you want to "close over" the value in the closure then there's a simple trick you can use: Pass a keyword argument with the default value equal to the value you want to close over. Voila!
- zbentley 2y ago
  Functions.partial can also supply captures in that way, and is slightly more obvious (and less likely to generate linter complaints if the kwarg is mutable).
  - zbentley 2y ago
    er, functools.partial, too late to edit.
deleted 2y ago
[deleted]
d0mine 2y ago
all one needs to understand is that there are names that refer to objects in Python. Nothing is passed by value. The closure returns whatever object the name refers at the moment. That is all. def loop(): for i in range(10): yield lambda j=i: j `j` refers to the same object (int) `i` refers to at the time of the lambda creation (the desired result). `j` is local to the lambda (each lambda has its own local namespace). Just `lambda: i` code would result in an object that `i` refers to at the time of calling the lambda (`i` name is not local to the lambda). https://docs.python.org/3/faq/programming.html#why-do-lambdas-defined-in-a-loop-with-different-values-all-return-the-same-result https://docs.python.org/3/faq/programming.html#why-do-lambda...

Spivak 2y ago

This is less wat. when you write it like eager: for item in loop(): item() lazy: items = [] for item in loop(): items.append(item) for item in items: item() Each next() into loop increments a shared counter so when you print it out as you go it 1, 2, 3 but when you print it out once you've called it a bunch is 9, 9, 9.

riffraff 2y ago

but is it a closure oddity? Looks to me like it's a generator/comprehension oddity :)

amluto 2y ago
It’s a closure oddity. The closure captures a reference to the variable, not the value of the variable at the time of its creation. Given that, the results are entirely predictable.
- abecedarius 2y ago
  If we're really going to argue, I'd argue for a 'for' oddity. The for statement could've been defined with a fresh binding per iteration instead of a mutation.
  - Izkata 2y ago
    That would make it work differently than the entire rest of the language. Python is function-scoped, not block-scoped.
    - abecedarius 2y ago
      That ship already sailed when Python 3 made comprehension expressions (like list comprehensions) introduce a nested scope. They're even a kind of 'for' too. (Previously, [x for x in y] would clobber any x in the same function. Fixing this meant implicit nested scopes.) It's true that this would've been a further change (every loop entering the implicit nest, rather than entering one nest which then mutates per loop). But this design was a decision, not a necessity. JS's "for let" shows this by example.
  - b3orn 2y ago
    It's unrelated to the for loop, the for only changes the variable, there are other ways you can do that and you'll run into the same "oddity" def test(): n = 0 def a(): return n n = 1 def b(): return n return a, b a, b = test() a() == 1 b() == 1

sakex 2y ago

That's why Javascript introduced the "let" keyword years ago This logs from 0 to 9: for(let i = 0; i < 10; ++i;) setTimeout(() => console.log(i)); This logs from 10, 10 times: for(var i = 0; i < 10; ++i;) setTimeout(() => console.log(i));

NwtnsMthd 2y ago
Do you think you could elaborate on why the results differ? I'm somewhat unfamiliar with how JavaScript works.
- wging 2y ago
  'var' is JavaScript's older variable declaration construct. Variables created this way are live from the beginning of the function that contains them (or globally if there isn't one). So a block with braces (such as you'd use for a for or while loop body) doesn't actually restrict the scope of var `v` below: console.log(v); // <-- v is a variable here, we can access its value even though it is only declared below // prints 'undefined' { var v = 1; console.log(v); // prints 1 } console.log(v); // prints 1 You used to (and might still) see a workaround to recover more restrictive scoping, known as the "IIFE" (Immediately Evaluated Function Expression): (function () { var v; ... code using v here ... })() creates a function (and thus a more restrictive scope for v to live in) and evaluates it once; this is a sort of poor man's block scoping. `let` and `const` were created to fill this gap. They have block scope and are special-cased in for loops (well, `let` is; you can't reassign to a `const` variable so nontrivial for loops won't work): console.log(l); // <-- throws ReferenceError: l is not defined { // pretend the console.log line above is commented out, so we can reach this line let l = 1; console.log(l); // prints 1, as expected } console.log(l); // throws ReferenceError: l is not defined // ^^ l was only in scope for the block above The interaction with `for` is explained well on MDN, including the special casing: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for#lexical_declarations_in_the_initialization_block https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe... "More precisely, let declarations are special-cased by for loops..." (followed by a more detailed explanation of why) See also https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/let https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe... and https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/var https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

peterhull90 2y ago

You can put an extra 'level' in, to make both options return the same 0,1...9 def loop(): for number in range(10): def outer(n): def inner(): return n return inner yield outer(number) Is there a neater way?

nbadg 2y ago
Yes. The usual way to do this is to bind them as defaults to an argument, for example: def loop(): for number in range(10): def func_w_closure(_num=number): return _num yield func_w_closure This works because default arguments in python are evaluated exactly once, at function definition time. So it's a way of effectively copying the ``number`` out of the closure[1] and into the function definition. [1] side note, closures in python are always late-binding, which is what causes the behavior in OP
- NwtnsMthd 2y ago
  Does this work for user defined classes (objects) as well?
  - Jtsummers 2y ago
    Yes, and by the same mechanism and for the same reason.
- amluto 2y ago
  I’ve never thought that leaking this type of implementation detail into the return value (and return type!) was a nice solution. I like the double closure better, and one can shorten it a bit with a lambda. For those who prefer a functional style, functools.partial can also solve this problem. (I use Python, and I like a lot of things about Python, but I don’t like its scoping rules at all, nor do I like the way that Python’s closures work. I would use a double lambda.)
- deleted 2y ago
  [deleted]

crabbone 2y ago

I'll give you one better: >>> strings = tuple((lambda: f'{i}') for i in range(10)) >>> strings[1]() '9' The f-strings are supposed to be evaluated at... what time exactly? And some more stuff: >>> class Elephant: ... def __init__(self, i): ... self.i = i ... def __call__(self): ... return self.i ... >>> elephants = tuple(Elephant(i) for i in range(10)) >>> elephants[0]() 0 >>> elephants[5]() 5

deleted 2y ago
[deleted]
postexitus 2y ago
The answer is simple. f strings are evaluated at run time. The behaviour is correct. Up until you call the lambda, they are just a function definition. What's so interesting about the Elephant case? You store the value of i at run time, so it can be accessed later. lambda doesn't have that capability, as it's inheriting i from the block.
- crabbone 2y ago
  > The answer is simple. f strings are evaluated at run time. You didn't understand the question. Everything in Python is evaluated at runtime. Bytecode is just a form of caching. > The behaviour is correct. Python doesn't have a standard. Nobody knows what behavior is correct. That's the only behavior Python has, so, again, calling it correct is just not saying anything. > What's so interesting about the Elephant case? This class, in principle, shouldn't be different from lambda from the previous example, yet it acts differently. You don't need to lecture me on why things behave the way they are. Since I found these examples, you can trust me, I know why they behave like this. The idea is to drive your attention to the inconsistencies that a "naive" reader would discover.
  - BerislavLopac 2y ago
    > Everything in Python is evaluated at runtime That is not correct. Python generally has two "runtimes": one is on import, when all of the module level code is evaluated; and the runtime, when the bodies of various functions and methods are evaluated. The two can be combined, e.g. by importing a module from within a function, but generally they are separate. > Bytecode is just a form of caching Not quite. It is perfectly possible to deploy just the bytecode and disable recompiling on import. > Python doesn't have a standard Of course it does -- CPython is the reference implementation, so its behaviour is "correct".

gpderetta 2y ago

Typical workaround: def loop(): for number in range(10): def closure(number=number): return number yield closure Many languages have surprising behaviour when closing over iteration variables.