6 ms·

Python’s lambda is broken

31 points by dangoldin 18y ago

enum 18y ago
Lambda is fine. The loops are mutating the variable i, so all the lambda share the same i. Apparently, list comprehensions also use mutation--I suppose that's fine.
- scott_s 18y ago
  Indeed, Python is using lexical scoping. This behavior is well known.
  - jrockway 18y ago
    While true, that is not really the issue. The issue is which lexical scope the loop iterator is defined in. In python, it is apparently like this: i = 0 increment i <do loop body with i> It could have been: i = 0 increment 1 create new environment bind i here <do loop body> In the second case, each loop body would have its own "i". In the first case, they all share the same "i".
shiro 18y ago
Same in Common Lisp. cl-user(1): (defvar fns (loop for i from 0 below 10 collect (lambda (n) (+ i n)))) fns cl-user(2): (funcall (elt fns 3) 4) 14 Personally I don't like this behavior (I've been bitten before in more subtle cases), but you can't blame lambda. It is consistent if one understands the same i is shared by all lambdas and loop mutates the i.
- dangoldin 18y ago
  Well it makes sense given the language - I just found it interesting that Haskell acts differently.
  - jrockway 18y ago
    Haskell doesn't act differently. If you translate the Haskell to Lisp: (funcall (elt (mapcar (lambda (x) (lambda (y) (+ x y))) '(1 2 3)) 0) 2) ==> 3 Then the Lisp works the same as the Haskell. The difference is that "loop" mutates the loop variable, whereas mapping makes a copy. In Haskell, there is no way to mutate a variable (that's not even what variables are), so that's the only way you can express the concept. In Lisp, you can either copy or mutate, and you get a different result depending on which you choose. Closures always work this way. Otherwise, you couldn't write things like: (defun make-incrementers () (let ((i 0)) (cons (lambda () (+ i 1)) (lambda () (- i 1)))))
    - trjordan 18y ago
      Similarly, if you use map instead, you get the right answer: >>> map(lambda x: lambda i: x + i, range(0,10))[3](4) 7
      - gamache 18y ago
        Perl 5.10.0 gives three different sets of answers. Map works as the original author expected: @fs = map {sub {$_ + shift}} (0..9); print $fs[$_](3), " " for (0..9); >>> 3 4 5 6 7 8 9 10 11 12 A Perlish for loop, or rather a statement with a "for" modifier: push @fs, sub {$_ + shift} for (0..9); print $fs[$_](3), " " for (0..9); >>> 3 4 5 6 7 8 9 10 11 12 The same loop, unpacked into a more universal syntax: for (0..9) {push @fs, sub {$_ + shift}} print $fs[$_](3), " " for (0..9); >>> 3 4 5 6 7 8 9 10 11 12 Here's where things start getting awesome. Let's name our loop variable $i, rather than using the Perlish default $_. for $i (0..9) {push @fs, sub {$i + shift}} print $fs[$_](3), " " for (0..9); >>> 3 3 3 3 3 3 3 3 3 3 And now let's write a C-style for loop. for ($i=0; $i<10; $i++) {push @fs, sub {$i + shift}} print $fs[$_](3), " " for (0..9); >>> 13 13 13 13 13 13 13 13 13 13 I am not sure why the last two versions work differently; I'd halfway expect the bottom version, but the "for $i (0..9)" loop leaves me scratching my head.
        jrockway 18y ago
lbolognini 18y ago
And your site doesn't feel that good either...
pavelludiq 18y ago
heres what i did: >>> [i(4) for i in [(lambda n: i + n) for i in range(10)]] and i got this: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 1, in <lambda> TypeError: unsupported operand type(s) for +: 'function' and 'int' Apparently the outer and inter "i"s got confused. when i change the outer 'i' to 'f': [f(4) for f in [(lambda n: i + n) for i in range(10)]] i get this: [13, 13, 13, 13, 13, 13, 13, 13, 13, 13] With generator expressions it worked as expected: >>> [i(4) for i in ((lambda n: i + n) for i in range(10))] [4, 5, 6, 7, 8, 9, 10, 11, 12, 13] Meanwhile: [i(4) for i in map(lambda i: lambda n:i+n, range(10))] Also works as expected. This obviously has nothing to do with the lambda in python and everything to do with the way loops work. Python has a few quirks of this kind, this one i didn't know about. I didn't know that list comprehensions and generator expressions differ in any other way than the thing they return. I should have known better, they are very different things. edit: looks like this also works: [i(4) for i in [(lambda i: lambda n:i+n)(i) for i in range(10)]] Its all about scope I guess.
- abdulhaq 18y ago
  This issue (list comprehensions vs generator expressions) becomes clearer after reading PEP 289 (http://www.python.org/dev/peps/pep-0289/ http://www.python.org/dev/peps/pep-0289/) which includes: List comprehensions also "leak" their loop variable into the surrounding scope. This will also change in Python 3.0, so that the semantic definition of a list comprehension in Python 3.0 will be equivalent to list(<generator expression>). Python 2.4 and beyond should issue a deprecation warning if a list comprehension's loop variable has the same name as a variable used in the immediately surrounding scope. and After exploring many possibilities, a consensus emerged that binding issues were hard to understand and that users should be strongly encouraged to use generator expressions inside functions that consume their arguments immediately. For more complex applications, full generator definitions are always superior in terms of being obvious about scope, lifetime, and binding.
  - pavelludiq 18y ago
    Thanks for the clarification. I didn't know that until i started messing with them just now for my post. Heres a a clearer example(you could deduce this from my previous post): >>> x=((lambda n: i + n) for i in range(10)) >>> y=((lambda i: lambda n: i + n)(i) for i in range(10)) >>> [f for f in x]==[i for i in y] True this is with generators. with list comprehensions it becomes: >>> x=[(lambda n: i + n) for i in range(10)] >>> y=[(lambda i: lambda n: i + n)(i) for i in range(10)] >>> [f for f in x]==[i for i in y] False
jrockway 18y ago
The deeper issue here is "mutation is hard to reason about". Well, yeah.
- miloshh 18y ago
  Well, I would disagree - the problem is that there is no apparent mutation in the given example. There is hidden mutation, in that the list comprehension mutates its variable. This is probably done for efficiency, but I would claim it's a bad design decision.
- dreish 18y ago
  Since that's Clojure's motto, let's see how it does: user=> (def fs (vec (for [i (range 10)] #(+ i %)))) #'user/fs user=> ((fs 3) 4) 7 Looks good.
juanpablo 18y ago
Same thing happens with closures in JavaScript
yummyfajitas 18y ago
Yes, Python closes over references to a mutable variable, and list comprehensions work the same way. It doesn't close over the value. I'd make the suggestion that closures are not very "Pythonic". Use objects instead: class add_n_func: def __init__(self, n): self.n = n def __call__(self, x): return x+self.n return [add_n_func(n) for n in range(10)]
- pavelludiq 18y ago
  How is a 5 line class with 2 underscore methods more pythonic than a nested lambda?
  - yummyfajitas 18y ago
    While python supports some basic functional programming, it's primarily a procedural/object oriented language. And it shows. As the article demonstrates, using nested lambda's doesn't work the way you expect for closures. Using an object does. edit: update. I found the presentation where guido talks about how he dislikes all the functional stuff, including lambda. http://74.125.93.104/search?q=cache:2QztD1KncJgJ:www.python.org/doc/essays/ppt/regrets/PythonRegrets.ppt+OSCON+%22python+regrets%22&cd=1&hl=en&ct=clnk&gl=us* http://74.125.93.104/search?q=cache:2QztD1KncJgJ:www.python....
    - tptacek 18y ago
      Does it make you sad that this is the same answer you'd have given if the language in question was Java? Just rubbing it in.
- deleted 18y ago
  [deleted]
- ii 18y ago
  I would suggest that the most "Pythonic" way would be to use functools.partial and operator.add from the standard library. As I already wrote it would look "fs = [partial(add, i) for i in range(10)]" and it works as expected.
d0mine 18y ago
Ordinary Python function have the same behaviour due to Python has lexical scopes but dynamic namespaces. >>> def add(n): ... return i + n ... >>> fs = [add for i in range(10)] >>> fs[3](4) 13 This example works due to list-comprehension leaks `i` binding. http://stackoverflow.com/questions/139819/why-results-of-map-and-list-comprehension-are-different http://stackoverflow.com/questions/139819/why-results-of-map...
senko 18y ago
The difference in number of comments here (17 at the time of writing this) and on the article itself (0) shows a nice example of (lack of) usability. I'm guessing a number of people thought about commenting on the post, then found out they had to log in, and just gave up there (at least that was my reaction). The author missed out on some interesting feedback he might've got otherwise. [apologies for being slightly offtopic for the discussion topic, but it's related to the article]
ii 18y ago
Workarounds: fs = [(lambda n, i=i: i+n) for i in range(10)] or from operator import add from functools import partial fs = [partial(add, i) for i in range(10)]
glomek 18y ago
Seems fine to me. If you want the captured variable to be different each time, you need to create a new environment each time. How about this? >>> fs = [(lambda x: (lambda n: x + n))(i) for i in range(10)] >>> [f(4) for f in fs] [4, 5, 6, 7, 8, 9, 10, 11, 12, 13] >>>
vineetk 18y ago
fs = map(lambda i: lambda n: i + n, range(10)) I actually don't think this is all that difficult to explain to newcomers. It may not be the expected behavior the first time they run into it, but it's an instructive example about python scopes and name spaces.
alecco 18y ago
This is rubbish. From comments right there on that site, Python and many other languages "leak" the iterated result, and combined with closures this doesn't work. The right way to do it is there: >>> fs = [(lambda n, j=i: j + n) for i in range(10)] >>> fs[3](4) 7 Anyway, Python 3.0 got rid of lambda. It complicates things and always there's a way to do it more pythonic. Note for lisp knee-jerkers: I don't have anything against anonymous functions. In Python, 3rd party code is harder to understand if it has lambdas all over the place. At least for me. HN: This is yet another example of front page trash. EDIT: Seems this teacher guy is not even recognizing his mistake. Now it's a "design error." While I could accept the explanation about how closures capture the location and not the value, this really is a design error. The loop variable ough to be local to the loop. But I suppose I get what I deserve for using a dynamic language. From http://www.python.org/doc/2.5.2/ref/naming.html http://www.python.org/doc/2.5.2/ref/naming.html, it is either a module, function, class definition, script (file|command|eval), or the expression fed to input() (rare.) 4.1 Naming and binding Names refer to objects. Names are introduced by name binding operations. Each occurrence of a name in the program text refers to the binding of that name established in the innermost function block containing the use. A block is a piece of Python program text that is executed as a unit. The following are blocks: a module, a function body, and a class definition. A script file [...] is a code block. A script command [...] is a code block. The file read by the built-in function execfile() is a code block. The string argument passed to the built-in function eval() and to the exec statement is a code block. The expression read and evaluated by the built-in function input() is a code block.

31 points by dangoldin 18y ago

enum 18y ago

Lambda is fine. The loops are mutating the variable i, so all the lambda share the same i. Apparently, list comprehensions also use mutation--I suppose that's fine.

scott_s 18y ago
Indeed, Python is using lexical scoping. This behavior is well known.
- jrockway 18y ago
  While true, that is not really the issue. The issue is which lexical scope the loop iterator is defined in. In python, it is apparently like this: i = 0 increment i <do loop body with i> It could have been: i = 0 increment 1 create new environment bind i here <do loop body> In the second case, each loop body would have its own "i". In the first case, they all share the same "i".

shiro 18y ago

Same in Common Lisp. cl-user(1): (defvar fns (loop for i from 0 below 10 collect (lambda (n) (+ i n)))) fns cl-user(2): (funcall (elt fns 3) 4) 14 Personally I don't like this behavior (I've been bitten before in more subtle cases), but you can't blame lambda. It is consistent if one understands the same i is shared by all lambdas and loop mutates the i.

dangoldin 18y ago
Well it makes sense given the language - I just found it interesting that Haskell acts differently.
- jrockway 18y ago
  Haskell doesn't act differently. If you translate the Haskell to Lisp: (funcall (elt (mapcar (lambda (x) (lambda (y) (+ x y))) '(1 2 3)) 0) 2) ==> 3 Then the Lisp works the same as the Haskell. The difference is that "loop" mutates the loop variable, whereas mapping makes a copy. In Haskell, there is no way to mutate a variable (that's not even what variables are), so that's the only way you can express the concept. In Lisp, you can either copy or mutate, and you get a different result depending on which you choose. Closures always work this way. Otherwise, you couldn't write things like: (defun make-incrementers () (let ((i 0)) (cons (lambda () (+ i 1)) (lambda () (- i 1)))))
  - trjordan 18y ago
    Similarly, if you use map instead, you get the right answer: >>> map(lambda x: lambda i: x + i, range(0,10))[3](4) 7
    - gamache 18y ago
      Perl 5.10.0 gives three different sets of answers. Map works as the original author expected: @fs = map {sub {$_ + shift}} (0..9); print $fs[$_](3), " " for (0..9); >>> 3 4 5 6 7 8 9 10 11 12 A Perlish for loop, or rather a statement with a "for" modifier: push @fs, sub {$_ + shift} for (0..9); print $fs[$_](3), " " for (0..9); >>> 3 4 5 6 7 8 9 10 11 12 The same loop, unpacked into a more universal syntax: for (0..9) {push @fs, sub {$_ + shift}} print $fs[$_](3), " " for (0..9); >>> 3 4 5 6 7 8 9 10 11 12 Here's where things start getting awesome. Let's name our loop variable $i, rather than using the Perlish default $_. for $i (0..9) {push @fs, sub {$i + shift}} print $fs[$_](3), " " for (0..9); >>> 3 3 3 3 3 3 3 3 3 3 And now let's write a C-style for loop. for ($i=0; $i<10; $i++) {push @fs, sub {$i + shift}} print $fs[$_](3), " " for (0..9); >>> 13 13 13 13 13 13 13 13 13 13 I am not sure why the last two versions work differently; I'd halfway expect the bottom version, but the "for $i (0..9)" loop leaves me scratching my head.
      - jrockway 18y ago

lbolognini 18y ago

And your site doesn't feel that good either...

pavelludiq 18y ago

heres what i did: >>> [i(4) for i in [(lambda n: i + n) for i in range(10)]] and i got this: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 1, in <lambda> TypeError: unsupported operand type(s) for +: 'function' and 'int' Apparently the outer and inter "i"s got confused. when i change the outer 'i' to 'f': [f(4) for f in [(lambda n: i + n) for i in range(10)]] i get this: [13, 13, 13, 13, 13, 13, 13, 13, 13, 13] With generator expressions it worked as expected: >>> [i(4) for i in ((lambda n: i + n) for i in range(10))] [4, 5, 6, 7, 8, 9, 10, 11, 12, 13] Meanwhile: [i(4) for i in map(lambda i: lambda n:i+n, range(10))] Also works as expected. This obviously has nothing to do with the lambda in python and everything to do with the way loops work. Python has a few quirks of this kind, this one i didn't know about. I didn't know that list comprehensions and generator expressions differ in any other way than the thing they return. I should have known better, they are very different things. edit: looks like this also works: [i(4) for i in [(lambda i: lambda n:i+n)(i) for i in range(10)]] Its all about scope I guess.

abdulhaq 18y ago
This issue (list comprehensions vs generator expressions) becomes clearer after reading PEP 289 (http://www.python.org/dev/peps/pep-0289/ http://www.python.org/dev/peps/pep-0289/) which includes: List comprehensions also "leak" their loop variable into the surrounding scope. This will also change in Python 3.0, so that the semantic definition of a list comprehension in Python 3.0 will be equivalent to list(<generator expression>). Python 2.4 and beyond should issue a deprecation warning if a list comprehension's loop variable has the same name as a variable used in the immediately surrounding scope. and After exploring many possibilities, a consensus emerged that binding issues were hard to understand and that users should be strongly encouraged to use generator expressions inside functions that consume their arguments immediately. For more complex applications, full generator definitions are always superior in terms of being obvious about scope, lifetime, and binding.
- pavelludiq 18y ago
  Thanks for the clarification. I didn't know that until i started messing with them just now for my post. Heres a a clearer example(you could deduce this from my previous post): >>> x=((lambda n: i + n) for i in range(10)) >>> y=((lambda i: lambda n: i + n)(i) for i in range(10)) >>> [f for f in x]==[i for i in y] True this is with generators. with list comprehensions it becomes: >>> x=[(lambda n: i + n) for i in range(10)] >>> y=[(lambda i: lambda n: i + n)(i) for i in range(10)] >>> [f for f in x]==[i for i in y] False

jrockway 18y ago

The deeper issue here is "mutation is hard to reason about". Well, yeah.

miloshh 18y ago
Well, I would disagree - the problem is that there is no apparent mutation in the given example. There is hidden mutation, in that the list comprehension mutates its variable. This is probably done for efficiency, but I would claim it's a bad design decision.
dreish 18y ago
Since that's Clojure's motto, let's see how it does: user=> (def fs (vec (for [i (range 10)] #(+ i %)))) #'user/fs user=> ((fs 3) 4) 7 Looks good.

juanpablo 18y ago

Same thing happens with closures in JavaScript

yummyfajitas 18y ago

Yes, Python closes over references to a mutable variable, and list comprehensions work the same way. It doesn't close over the value. I'd make the suggestion that closures are not very "Pythonic". Use objects instead: class add_n_func: def __init__(self, n): self.n = n def __call__(self, x): return x+self.n return [add_n_func(n) for n in range(10)]

pavelludiq 18y ago
How is a 5 line class with 2 underscore methods more pythonic than a nested lambda?
- yummyfajitas 18y ago
  While python supports some basic functional programming, it's primarily a procedural/object oriented language. And it shows. As the article demonstrates, using nested lambda's doesn't work the way you expect for closures. Using an object does. edit: update. I found the presentation where guido talks about how he dislikes all the functional stuff, including lambda. http://74.125.93.104/search?q=cache:2QztD1KncJgJ:www.python.org/doc/essays/ppt/regrets/PythonRegrets.ppt+OSCON+%22python+regrets%22&cd=1&hl=en&ct=clnk&gl=us* http://74.125.93.104/search?q=cache:2QztD1KncJgJ:www.python....
  - tptacek 18y ago
    Does it make you sad that this is the same answer you'd have given if the language in question was Java? Just rubbing it in.
deleted 18y ago
[deleted]
ii 18y ago
I would suggest that the most "Pythonic" way would be to use functools.partial and operator.add from the standard library. As I already wrote it would look "fs = [partial(add, i) for i in range(10)]" and it works as expected.

d0mine 18y ago

Ordinary Python function have the same behaviour due to Python has lexical scopes but dynamic namespaces. >>> def add(n): ... return i + n ... >>> fs = [add for i in range(10)] >>> fs[3](4) 13 This example works due to list-comprehension leaks `i` binding. http://stackoverflow.com/questions/139819/why-results-of-map-and-list-comprehension-are-different http://stackoverflow.com/questions/139819/why-results-of-map...

senko 18y ago

The difference in number of comments here (17 at the time of writing this) and on the article itself (0) shows a nice example of (lack of) usability. I'm guessing a number of people thought about commenting on the post, then found out they had to log in, and just gave up there (at least that was my reaction). The author missed out on some interesting feedback he might've got otherwise. [apologies for being slightly offtopic for the discussion topic, but it's related to the article]

ii 18y ago

Workarounds: fs = [(lambda n, i=i: i+n) for i in range(10)] or from operator import add from functools import partial fs = [partial(add, i) for i in range(10)]

glomek 18y ago

Seems fine to me. If you want the captured variable to be different each time, you need to create a new environment each time. How about this? >>> fs = [(lambda x: (lambda n: x + n))(i) for i in range(10)] >>> [f(4) for f in fs] [4, 5, 6, 7, 8, 9, 10, 11, 12, 13] >>>

vineetk 18y ago

fs = map(lambda i: lambda n: i + n, range(10)) I actually don't think this is all that difficult to explain to newcomers. It may not be the expected behavior the first time they run into it, but it's an instructive example about python scopes and name spaces.

alecco 18y ago

This is rubbish. From comments right there on that site, Python and many other languages "leak" the iterated result, and combined with closures this doesn't work. The right way to do it is there: >>> fs = [(lambda n, j=i: j + n) for i in range(10)] >>> fs[3](4) 7 Anyway, Python 3.0 got rid of lambda. It complicates things and always there's a way to do it more pythonic. Note for lisp knee-jerkers: I don't have anything against anonymous functions. In Python, 3rd party code is harder to understand if it has lambdas all over the place. At least for me. HN: This is yet another example of front page trash. EDIT: Seems this teacher guy is not even recognizing his mistake. Now it's a "design error." While I could accept the explanation about how closures capture the location and not the value, this really is a design error. The loop variable ough to be local to the loop. But I suppose I get what I deserve for using a dynamic language. From http://www.python.org/doc/2.5.2/ref/naming.html http://www.python.org/doc/2.5.2/ref/naming.html, it is either a module, function, class definition, script (file|command|eval), or the expression fed to input() (rare.) 4.1 Naming and binding Names refer to objects. Names are introduced by name binding operations. Each occurrence of a name in the program text refers to the binding of that name established in the innermost function block containing the use. A block is a piece of Python program text that is executed as a unit. The following are blocks: a module, a function body, and a class definition. A script file [...] is a code block. A script command [...] is a code block. The file read by the built-in function execfile() is a code block. The string argument passed to the built-in function eval() and to the exec statement is a code block. The expression read and evaluated by the built-in function input() is a code block.