6 ms·

I found this interesting because I have never run into this before...and I thought that was rare (in relation to how comfortable I am with Python). That said,

I found this interesting because I have never run into this before...and I thought that was rare (in relation to how comfortable I am with Python). That said, I personally think this is the right and intuitive outcome. If we expand out the loop to be manual we would have a script like: functions = [] # expanded loop i = 1 functions.append(lambda: print(f"Hello {i}")) i = 2 functions.append(lambda: print(f"Hello {i}")) ... i = 9 functions.append(lambda: print(f"Hello {i}")) Now at this point if we were to: print(f"Hello {i}") What would the expected output be? I would posit that anything other than "Hello 9" would be wrong, both logically and intuitively. So by extension a loop of effectively "print(f"Hello {i}")" 9 times should just print "Hello 9" 9 times IMO. Anything else is counter-intuitive and definitely surprising.

_flux 4y ago
I think the reason this seems surprising is the value/reference semantics of values. For example: def foo(i, j): i[0] += 1 j += 1 a = [0] b = 0 foo(a, b) print(a) # [1] print(b) # [0] So people think the body of the loop like a function call. I don't think it's a bad expectation, in fact I think it's quite a natural expectation—in particular if you've programmed functional languages where modifying values is the exception, not the rule—which is why it surprises people. It's just not the one way Python chose.
tmazeika 4y ago
The example makes sense when you think of `i` getting reassigned, but in other languages with variable shadowing (think Rust, but Go too when you're in a nested scope) you might replace `i = 2` with `let i = 2` (Rust), in which case the closure closes over precisely the `let i = 2` variable and the expected output differs.
jrumbut 4y ago
I feel that the original Python example is suffering from a little too much lambda. It seems like we're iterating through something to build up a computation we may execute later. I can conceive of a situation where you might want to do that, or at least consider it, but in general I say just do the work now and build a list of results.
- deleted 4y ago
  [deleted]
- BoppreH 4y ago
  It looks like that because it's a minimal example. Real cases usually involve callbacks or expensive computations. How often it happens depends a lot on what you're doing.
duped 4y ago
I think it comes down to if you think it makes sense for variables to be captured by value or by reference in closures. Which ever makes the most sense depends on the context of a program which is why in languages that have both reference and closure semantics you can choose how the variables are captured. When that isn't the case you need to pick for someone, and it gets weird.
- dragonwriter 4y ago
  > I think it comes down to if you think it makes sense for variables to be captured by value or by reference in closures. No, this comes down to “should a loop control variable be scoped to the block—or in python’s case function—the loop is in and updated with each iteration or a fresh variable scoped to each loop iteration that happens to share the same name.”
  - randomswede 4y ago
    In the "the variable lives in the function scope", the answer is unequivocally "it should be bound once and updated", if it only exists in the scope of the loop function, both "it is bound on each iteration" (and thus safe to close over without surprise) and "it is bound once and updated" are valid answers, but I have a preference for the first, but many languages actually choose the second.
masklinn 4y ago
> I thought that was rare It is rare, if only because defining functions (lambdas or otherwise) in a loop is somewhat uncommon. That's not very helpful when you hit that issue though.
BoppreH 4y ago
That's one way of thinking about it, and it works well for Python. Here's another one, and the way I think about it: def _loop(i): functions.append(lambda: print(f"Hello {i}")) _loop(0) _loop(1) _loop(2) print(i) # Error! I highly prefer lexical scoping, where the variables are bound to the block they were declared in, like Javascript's `let` vs the old `var`. This avoids shadowing and general namespace pollution. I know it's not how Python operates, but I think it's how it should. Though I'd argue the syntactical similarities between functions and loops nudge users towards this second model.
- pdonis 4y ago
  Note that the way you write this is the best way to form closures in Python: form them inside a function with the closure variables as function parameters. This forces the closure to capture the current value of the closure variables when the closure is formed. Note that you could put the calls to the _loop function in a for loop and things would still work: >>> functions = [] >>> def _loop(i): ... functions.append(lambda: print(f"Hello {i}")) ... >>> for i in range(10): ... _loop(i) ... >>> for f in functions: ... f() ... Hello 0 Hello 1 Hello 2 Hello 3 Hello 4 Hello 5 Hello 6 Hello 7 Hello 8 Hello 9 > lexical scoping, where the variables are bound to the block they were declared in In Python that would only work if a "block" included comprehensions. For example: >>> functions = [lambda: print(f"Hello {i}") for i in range(10)] >>> for f in functions: ... f() ... Hello 9 Hello 9 Hello 9 Hello 9 Hello 9 Hello 9 Hello 9 Hello 9 Hello 9 Hello 9 You could fix this by defining a _loop function as above and forming the closure inside it; but changing variable scoping to be lexical in "blocks" wouldn't fix this case unless the list comprehension itself counted as a "block", which is not how Python defines blocks.
  - dkmar 4y ago
    Wow. When I saw the original for-loop example I was like "ok not a huge deal, doesn't surprise me too much really". But rewritten in the comprehension form I'm like wtf
- 4y ago
OkayPhysicist 4y ago
The problem is the intersection of people's intuition about closures from languages without mutable state bumping into the imperative world built of mutable state. For example, equivalent code in Elixir: i = 0 list = [] list = list ++ [fn -> i end] i = i + 1 list = list ++ [fn -> i end] i = i + 1 halfway = list list = list ++ [fn -> i end] i = i + 1 list = list ++ [fn -> i end] IO.inspect Enum.map(halfway, fn f -> f.() end) IO.inspect Enum.map(list, fn f -> f.() end) Would produce the functional-intuitive result of [0,1] [0,1,2,3] Because there is no mutable state. Those repeated assignments to i and list are exactly equivalent to the scenario where each i was actually i1, i2, i3, etc.
alkonaut 4y ago
I expect that to write the obvious result (the last value assigned) and I also expect a capture in a loop to write the value as it was at the time of iteration. They are both the behavior that’s intuitively obvious. And that’s despite them both being the same behavior. As we can see, it’s important to not use the simple/obvious implementation because it’s so unintuitive it’ll need to be changed even if breaking (as in C#)
asddubs 4y ago
seems like the non magic solution here would be block scope. I guess it would still be slightly magic in that each iteration gets its own scope, but at least that's easier to wrap your head around than just a special case

_flux 4y ago

I think the reason this seems surprising is the value/reference semantics of values. For example: def foo(i, j): i[0] += 1 j += 1 a = [0] b = 0 foo(a, b) print(a) # [1] print(b) # [0] So people think the body of the loop like a function call. I don't think it's a bad expectation, in fact I think it's quite a natural expectation—in particular if you've programmed functional languages where modifying values is the exception, not the rule—which is why it surprises people. It's just not the one way Python chose.

tmazeika 4y ago

The example makes sense when you think of `i` getting reassigned, but in other languages with variable shadowing (think Rust, but Go too when you're in a nested scope) you might replace `i = 2` with `let i = 2` (Rust), in which case the closure closes over precisely the `let i = 2` variable and the expected output differs.

jrumbut 4y ago

I feel that the original Python example is suffering from a little too much lambda. It seems like we're iterating through something to build up a computation we may execute later. I can conceive of a situation where you might want to do that, or at least consider it, but in general I say just do the work now and build a list of results.

deleted 4y ago
[deleted]
BoppreH 4y ago
It looks like that because it's a minimal example. Real cases usually involve callbacks or expensive computations. How often it happens depends a lot on what you're doing.

duped 4y ago

I think it comes down to if you think it makes sense for variables to be captured by value or by reference in closures. Which ever makes the most sense depends on the context of a program which is why in languages that have both reference and closure semantics you can choose how the variables are captured. When that isn't the case you need to pick for someone, and it gets weird.

dragonwriter 4y ago
> I think it comes down to if you think it makes sense for variables to be captured by value or by reference in closures. No, this comes down to “should a loop control variable be scoped to the block—or in python’s case function—the loop is in and updated with each iteration or a fresh variable scoped to each loop iteration that happens to share the same name.”
- randomswede 4y ago
  In the "the variable lives in the function scope", the answer is unequivocally "it should be bound once and updated", if it only exists in the scope of the loop function, both "it is bound on each iteration" (and thus safe to close over without surprise) and "it is bound once and updated" are valid answers, but I have a preference for the first, but many languages actually choose the second.

masklinn 4y ago

> I thought that was rare It is rare, if only because defining functions (lambdas or otherwise) in a loop is somewhat uncommon. That's not very helpful when you hit that issue though.

BoppreH 4y ago

That's one way of thinking about it, and it works well for Python. Here's another one, and the way I think about it: def _loop(i): functions.append(lambda: print(f"Hello {i}")) _loop(0) _loop(1) _loop(2) print(i) # Error! I highly prefer lexical scoping, where the variables are bound to the block they were declared in, like Javascript's `let` vs the old `var`. This avoids shadowing and general namespace pollution. I know it's not how Python operates, but I think it's how it should. Though I'd argue the syntactical similarities between functions and loops nudge users towards this second model.

pdonis 4y ago
Note that the way you write this is the best way to form closures in Python: form them inside a function with the closure variables as function parameters. This forces the closure to capture the current value of the closure variables when the closure is formed. Note that you could put the calls to the _loop function in a for loop and things would still work: >>> functions = [] >>> def _loop(i): ... functions.append(lambda: print(f"Hello {i}")) ... >>> for i in range(10): ... _loop(i) ... >>> for f in functions: ... f() ... Hello 0 Hello 1 Hello 2 Hello 3 Hello 4 Hello 5 Hello 6 Hello 7 Hello 8 Hello 9 > lexical scoping, where the variables are bound to the block they were declared in In Python that would only work if a "block" included comprehensions. For example: >>> functions = [lambda: print(f"Hello {i}") for i in range(10)] >>> for f in functions: ... f() ... Hello 9 Hello 9 Hello 9 Hello 9 Hello 9 Hello 9 Hello 9 Hello 9 Hello 9 Hello 9 You could fix this by defining a _loop function as above and forming the closure inside it; but changing variable scoping to be lexical in "blocks" wouldn't fix this case unless the list comprehension itself counted as a "block", which is not how Python defines blocks.
- dkmar 4y ago
  Wow. When I saw the original for-loop example I was like "ok not a huge deal, doesn't surprise me too much really". But rewritten in the comprehension form I'm like wtf
4y ago

OkayPhysicist 4y ago

The problem is the intersection of people's intuition about closures from languages without mutable state bumping into the imperative world built of mutable state. For example, equivalent code in Elixir: i = 0 list = [] list = list ++ [fn -> i end] i = i + 1 list = list ++ [fn -> i end] i = i + 1 halfway = list list = list ++ [fn -> i end] i = i + 1 list = list ++ [fn -> i end] IO.inspect Enum.map(halfway, fn f -> f.() end) IO.inspect Enum.map(list, fn f -> f.() end) Would produce the functional-intuitive result of [0,1] [0,1,2,3] Because there is no mutable state. Those repeated assignments to i and list are exactly equivalent to the scenario where each i was actually i1, i2, i3, etc.

alkonaut 4y ago

I expect that to write the obvious result (the last value assigned) and I also expect a capture in a loop to write the value as it was at the time of iteration. They are both the behavior that’s intuitively obvious. And that’s despite them both being the same behavior. As we can see, it’s important to not use the simple/obvious implementation because it’s so unintuitive it’ll need to be changed even if breaking (as in C#)

asddubs 4y ago

seems like the non magic solution here would be block scope. I guess it would still be slightly magic in that each iteration gets its own scope, but at least that's easier to wrap your head around than just a special case