6 ms·

The same problem frequently bites me in Python too: functions = [] for i in range(10): functions.append(lambda: print(f'Hello {i}'))

The same problem frequently bites me in Python too: functions = [] for i in range(10): functions.append(lambda: print(f'Hello {i}')) for fn in functions: fn() # Hello 9 # Hello 9 # Hello 9 # Hello 9 # Hello 9 # Hello 9 # Hello 9 # Hello 9 # Hello 9 # Hello 9 I find it one of Python's biggest warts because it's silent, hard to troubleshoot (especially the first time!), and like in Go the most straightforward fix looks like a mistake (i=i): for i in range(10): functions.append(lambda i=i: print(f'Hello {i}')) I guess this is a lesson in designing language semantics that match people's intuitions, and learning from previous languages' mistakes.

Simplicitas 4y ago
Yup. "Match [human] intuition" is the operative phrase in all of this. Thanks. Hope Go addresses this soon.
Semaphor 4y ago
If I do the same with for instead of foreach in C#, my IDE gives me a warning: "Captured variable is modified in the outer scope". Isn’t that the case for Python?
- ketzu 4y ago
  Copy pasting OPs code to pycharm: No warning.
  - Semaphor 4y ago
    Probably worthy of a YouTrack report ;)
  - jstx1 4y ago
    Pylint catches it - https://pylint.pycqa.org/en/latest/user_guide/messages/warning/cell-var-from-loop.html https://pylint.pycqa.org/en/latest/user_guide/messages/warni...
- maxloh 4y ago
  I don't think it is a good practice to capture it in lint tooling, instead of by the language itself.
  - Semaphor 4y ago
    It’s what linters are great at, catch things that work the way you don’t expect.
throw_m239339 4y ago
This is mostly why Ecmascript introduced lexical scoping (a bit similar to "block" scoping) with "let" back int ES 5 I believe, because var used the function scope and developers would run into issues whenever they did this: for (var i = 0 ; i < 10 ; i ++ ){ someElement.addEventListener('click',function(){ console.log(i); }) } i would always be equal to 9. With let instead of var, i is properly scoped and the executed script display each increment correctly. The solution before let was to introduce a closure in the for statement body for (var i = 0 ; i < 10 ; i ++ ){ (function(i){ someElement.addEventListener('click',function(){ console.log(i); }))(i) } in order to capture i value. let can also easily isolate the scope of a variable so that it doesn't popule the global scope { let foo = "bar"; var baz = "qix"; } // foo is undefined here, while baz is defined. which removes the need for self invoking functions.
gshulegaard 4y ago
I found this interesting because I have never run into this before...and I thought that was rare (in relation to how comfortable I am with Python). That said, I personally think this is the right and intuitive outcome. If we expand out the loop to be manual we would have a script like: functions = [] # expanded loop i = 1 functions.append(lambda: print(f"Hello {i}")) i = 2 functions.append(lambda: print(f"Hello {i}")) ... i = 9 functions.append(lambda: print(f"Hello {i}")) Now at this point if we were to: print(f"Hello {i}") What would the expected output be? I would posit that anything other than "Hello 9" would be wrong, both logically and intuitively. So by extension a loop of effectively "print(f"Hello {i}")" 9 times should just print "Hello 9" 9 times IMO. Anything else is counter-intuitive and definitely surprising.
- _flux 4y ago
  I think the reason this seems surprising is the value/reference semantics of values. For example: def foo(i, j): i[0] += 1 j += 1 a = [0] b = 0 foo(a, b) print(a) # [1] print(b) # [0] So people think the body of the loop like a function call. I don't think it's a bad expectation, in fact I think it's quite a natural expectation—in particular if you've programmed functional languages where modifying values is the exception, not the rule—which is why it surprises people. It's just not the one way Python chose.
- tmazeika 4y ago
  The example makes sense when you think of `i` getting reassigned, but in other languages with variable shadowing (think Rust, but Go too when you're in a nested scope) you might replace `i = 2` with `let i = 2` (Rust), in which case the closure closes over precisely the `let i = 2` variable and the expected output differs.
- jrumbut 4y ago
  I feel that the original Python example is suffering from a little too much lambda. It seems like we're iterating through something to build up a computation we may execute later. I can conceive of a situation where you might want to do that, or at least consider it, but in general I say just do the work now and build a list of results.
jmconfuzeus 4y ago
Technically, it makes sense for Python to work like this because your functions are being bound to a variable i which is located in the global scope. I wouldn't call this a design mistake but more of a misunderstanding of how scope works.
- remram 4y ago
  If other languages get this right then it's not a fundamental misunderstanding but a quirk of that specific language.
  - jmconfuzeus 4y ago
    Dunno what you mean by getting it right tbh. for i in range(10) The statement literally declares or sets a variable named i in that scope. When the loop exits, i still exists in the scope with the value 9. If you call a function that was given a reference to i, the value will be 9 as expected because the function was called after the loop exited. Don't see any quirk or mistake here.
    - masklinn 4y ago
      An equivalent construct works differently (and one would say in a way that is less error-prone) in other languages. For instance the behaviour of a Python loop varies drastically depending on the size of the iteration: def loop(n): for i in range(n): pass print(i) loop(10) # 9 loop(0) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 4, in loop UnboundLocalError: local variable 'i' referenced before assignment That Python works this way is specific to Python. And a language which doesn't have the issues this implies would be "getting it right", in the sense of avoiding sharp corners and edge cases.
      - jmconfuzeus 4y ago
        I see, it makes sense to force people to use different variables for iteration in that case because it can easily trip inexperienced developers up.
Epokhe 4y ago
This seems like a cleverly crafted satire reply, rather than an actual complaint about python.
- iudqnolq 4y ago
  If so, it's a bit misleading. The go bug just requires a reference, rather than a lambda. I bucket lambdas into a category of language features I expect to have more sharp edges around capture semantics than references. Bug: var all []*Item for _, item := range items { all = append(all, &item) } Fix: var all []*Item for _, item := range items { item := item all = append(all, &item) }
- BoppreH 4y ago
  I assure you it's not satire, otherwise it wouldn't be such a popular question on StackOverflow[1] and an item in the Python FAQ[2] [1]: https://stackoverflow.com/questions/2295290/what-do-lambda-function-closures-capture https://stackoverflow.com/questions/2295290/what-do-lambda-f... [2]: https://docs.python.org/3/faq/programming.html#why-do-lambdas-defined-in-a-loop-with-different-values-all-return-the-same-result https://docs.python.org/3/faq/programming.html#why-do-lambda...
  - Epokhe 4y ago
    Yeah, makes sense that it's popular. I just thought you were making a joke by creating a similar looking issue with a different underlying reason. In Go example the issue happens because item variable is per-loop, in your python example the issue is not related to loops at all, it's just because functions capture the value of global variables at execution time. And the cherry on top is that the solution is also similar looking(i=i), but working with a different mechanic underneath(default argument assignment). Anyway, this was my perspective that led me to interpret this as satire. A bit disappointed haha
dekhn 4y ago
I guess this is confusing but i just think of it as capturing the reference to i, not the value of i. It would be nice if python had a nice way to deal with this as many people also trip over: def fun(initial_empty_list=[]): where initial_empty_list is a reference captured at function definition time, not a new value initialized on each call to the function.
littlestymaar 4y ago
Wow, Python and Go have it too? It was the biggest wart of pre-ES6 JavaScript, I never imagined other languages had it too (and it's honestly very disappointing for Go, since it's much more recent and we had plenty of hindsight when it was created…) Edit: looks like Rust does it right: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=aec1261e40a7880e6882f6502a34abde https://play.rust-lang.org/?version=stable&mode=debug&editio...
- kevincox 4y ago
  Not quite how you tested it. The `move` is actually copying the variable: fn main() { let mut functions = vec![]; let mut i = 0; functions.push(Box::new(move || println!("Hello {i}")) as Box<dyn Fn()>); i = 1; functions.push(Box::new(move || println!("Hello {i}"))); i = 2; functions.push(Box::new(move || println!("Hello {i}"))); for func in functions { func(); } } But the most equivalent code would be the following which doesn't compile: fn main() { let mut functions = vec![]; for i in 0..10 { functions.push(Box::new(|| println!("Hello {i}"))) } for func in functions { func(); } }
  - littlestymaar 4y ago
    That's the point: the misleading code doesn't compile because the `i` variable scope in limited to the current execution of the loop (and that's why you can't borrow it for `functions`'s lifetime), which is exactly how ES6 fixed this in JavaScript.
maxloh 4y ago
It is a scoping problem IMO. In JavaScript variables are block-scoped, i will be out of scope after the loop. It would be better if Python could support block-scoping too. (Maybe it is time for inventing a "strict mode" for Python)
kjeetgill 4y ago
I've always referenced this as one of the subtle points of genius for how Java did lambdas: by requiring captured variables to be final it removes a lot of ambiguity around what a variable name refers to. I like that local variables can only be changed locally. If you do want crosstalk between the inner and outer scopes you have to be more explicit and introduce a reference to talk through. I love python, but I basically avoid this construction and use a single element list if I need it. I can never remember exactly how it works.
tomtung 4y ago
Ah I have "fond" memories about once spending a good week debugging this, because I was also spawning threads/sub-processes in the loop, making this a Heisenbug that caused the program to crash every hour or so...
deleted 4y ago
[deleted]

Simplicitas 4y ago

Yup. "Match [human] intuition" is the operative phrase in all of this. Thanks. Hope Go addresses this soon.

Semaphor 4y ago

If I do the same with for instead of foreach in C#, my IDE gives me a warning: "Captured variable is modified in the outer scope". Isn’t that the case for Python?

ketzu 4y ago
Copy pasting OPs code to pycharm: No warning.
- Semaphor 4y ago
  Probably worthy of a YouTrack report ;)
- jstx1 4y ago
  Pylint catches it - https://pylint.pycqa.org/en/latest/user_guide/messages/warning/cell-var-from-loop.html https://pylint.pycqa.org/en/latest/user_guide/messages/warni...
maxloh 4y ago
I don't think it is a good practice to capture it in lint tooling, instead of by the language itself.
- Semaphor 4y ago
  It’s what linters are great at, catch things that work the way you don’t expect.

throw_m239339 4y ago

This is mostly why Ecmascript introduced lexical scoping (a bit similar to "block" scoping) with "let" back int ES 5 I believe, because var used the function scope and developers would run into issues whenever they did this: for (var i = 0 ; i < 10 ; i ++ ){ someElement.addEventListener('click',function(){ console.log(i); }) } i would always be equal to 9. With let instead of var, i is properly scoped and the executed script display each increment correctly. The solution before let was to introduce a closure in the for statement body for (var i = 0 ; i < 10 ; i ++ ){ (function(i){ someElement.addEventListener('click',function(){ console.log(i); }))(i) } in order to capture i value. let can also easily isolate the scope of a variable so that it doesn't popule the global scope { let foo = "bar"; var baz = "qix"; } // foo is undefined here, while baz is defined. which removes the need for self invoking functions.

gshulegaard 4y ago

I found this interesting because I have never run into this before...and I thought that was rare (in relation to how comfortable I am with Python). That said, I personally think this is the right and intuitive outcome. If we expand out the loop to be manual we would have a script like: functions = [] # expanded loop i = 1 functions.append(lambda: print(f"Hello {i}")) i = 2 functions.append(lambda: print(f"Hello {i}")) ... i = 9 functions.append(lambda: print(f"Hello {i}")) Now at this point if we were to: print(f"Hello {i}") What would the expected output be? I would posit that anything other than "Hello 9" would be wrong, both logically and intuitively. So by extension a loop of effectively "print(f"Hello {i}")" 9 times should just print "Hello 9" 9 times IMO. Anything else is counter-intuitive and definitely surprising.

_flux 4y ago
I think the reason this seems surprising is the value/reference semantics of values. For example: def foo(i, j): i[0] += 1 j += 1 a = [0] b = 0 foo(a, b) print(a) # [1] print(b) # [0] So people think the body of the loop like a function call. I don't think it's a bad expectation, in fact I think it's quite a natural expectation—in particular if you've programmed functional languages where modifying values is the exception, not the rule—which is why it surprises people. It's just not the one way Python chose.
tmazeika 4y ago
The example makes sense when you think of `i` getting reassigned, but in other languages with variable shadowing (think Rust, but Go too when you're in a nested scope) you might replace `i = 2` with `let i = 2` (Rust), in which case the closure closes over precisely the `let i = 2` variable and the expected output differs.
jrumbut 4y ago
I feel that the original Python example is suffering from a little too much lambda. It seems like we're iterating through something to build up a computation we may execute later. I can conceive of a situation where you might want to do that, or at least consider it, but in general I say just do the work now and build a list of results.

jmconfuzeus 4y ago

Technically, it makes sense for Python to work like this because your functions are being bound to a variable i which is located in the global scope. I wouldn't call this a design mistake but more of a misunderstanding of how scope works.

remram 4y ago
If other languages get this right then it's not a fundamental misunderstanding but a quirk of that specific language.
- jmconfuzeus 4y ago
  Dunno what you mean by getting it right tbh. for i in range(10) The statement literally declares or sets a variable named i in that scope. When the loop exits, i still exists in the scope with the value 9. If you call a function that was given a reference to i, the value will be 9 as expected because the function was called after the loop exited. Don't see any quirk or mistake here.
  - masklinn 4y ago
    An equivalent construct works differently (and one would say in a way that is less error-prone) in other languages. For instance the behaviour of a Python loop varies drastically depending on the size of the iteration: def loop(n): for i in range(n): pass print(i) loop(10) # 9 loop(0) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 4, in loop UnboundLocalError: local variable 'i' referenced before assignment That Python works this way is specific to Python. And a language which doesn't have the issues this implies would be "getting it right", in the sense of avoiding sharp corners and edge cases.
    - jmconfuzeus 4y ago
      I see, it makes sense to force people to use different variables for iteration in that case because it can easily trip inexperienced developers up.

Epokhe 4y ago

This seems like a cleverly crafted satire reply, rather than an actual complaint about python.

iudqnolq 4y ago
If so, it's a bit misleading. The go bug just requires a reference, rather than a lambda. I bucket lambdas into a category of language features I expect to have more sharp edges around capture semantics than references. Bug: var all []*Item for _, item := range items { all = append(all, &item) } Fix: var all []*Item for _, item := range items { item := item all = append(all, &item) }
BoppreH 4y ago
I assure you it's not satire, otherwise it wouldn't be such a popular question on StackOverflow[1] and an item in the Python FAQ[2] [1]: https://stackoverflow.com/questions/2295290/what-do-lambda-function-closures-capture https://stackoverflow.com/questions/2295290/what-do-lambda-f... [2]: https://docs.python.org/3/faq/programming.html#why-do-lambdas-defined-in-a-loop-with-different-values-all-return-the-same-result https://docs.python.org/3/faq/programming.html#why-do-lambda...
- Epokhe 4y ago
  Yeah, makes sense that it's popular. I just thought you were making a joke by creating a similar looking issue with a different underlying reason. In Go example the issue happens because item variable is per-loop, in your python example the issue is not related to loops at all, it's just because functions capture the value of global variables at execution time. And the cherry on top is that the solution is also similar looking(i=i), but working with a different mechanic underneath(default argument assignment). Anyway, this was my perspective that led me to interpret this as satire. A bit disappointed haha

dekhn 4y ago

I guess this is confusing but i just think of it as capturing the reference to i, not the value of i. It would be nice if python had a nice way to deal with this as many people also trip over: def fun(initial_empty_list=[]): where initial_empty_list is a reference captured at function definition time, not a new value initialized on each call to the function.

littlestymaar 4y ago

Wow, Python and Go have it too? It was the biggest wart of pre-ES6 JavaScript, I never imagined other languages had it too (and it's honestly very disappointing for Go, since it's much more recent and we had plenty of hindsight when it was created…) Edit: looks like Rust does it right: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=aec1261e40a7880e6882f6502a34abde https://play.rust-lang.org/?version=stable&mode=debug&editio...

kevincox 4y ago
Not quite how you tested it. The `move` is actually copying the variable: fn main() { let mut functions = vec![]; let mut i = 0; functions.push(Box::new(move || println!("Hello {i}")) as Box<dyn Fn()>); i = 1; functions.push(Box::new(move || println!("Hello {i}"))); i = 2; functions.push(Box::new(move || println!("Hello {i}"))); for func in functions { func(); } } But the most equivalent code would be the following which doesn't compile: fn main() { let mut functions = vec![]; for i in 0..10 { functions.push(Box::new(|| println!("Hello {i}"))) } for func in functions { func(); } }
- littlestymaar 4y ago
  That's the point: the misleading code doesn't compile because the `i` variable scope in limited to the current execution of the loop (and that's why you can't borrow it for `functions`'s lifetime), which is exactly how ES6 fixed this in JavaScript.

maxloh 4y ago

It is a scoping problem IMO. In JavaScript variables are block-scoped, i will be out of scope after the loop. It would be better if Python could support block-scoping too. (Maybe it is time for inventing a "strict mode" for Python)

kjeetgill 4y ago

I've always referenced this as one of the subtle points of genius for how Java did lambdas: by requiring captured variables to be final it removes a lot of ambiguity around what a variable name refers to. I like that local variables can only be changed locally. If you do want crosstalk between the inner and outer scopes you have to be more explicit and introduce a reference to talk through. I love python, but I basically avoid this construction and use a single element list if I need it. I can never remember exactly how it works.

tomtung 4y ago

Ah I have "fond" memories about once spending a good week debugging this, because I was also spawning threads/sub-processes in the loop, making this a Heisenbug that caused the program to crash every hour or so...

deleted 4y ago

[deleted]