5 ms·

One interesting side-thing with list comprehensions in Python 2 vs. Python 3: Python 2.7: >list_of_numbers = [1,2,3] >[x/2 for x in list_of_numbers] >print(

One interesting side-thing with list comprehensions in Python 2 vs. Python 3: Python 2.7: >list_of_numbers = [1,2,3] >[x/2 for x in list_of_numbers] >print(x) 3 Python 3: >list_of_numbers = [1,2,3] >[x/2 for x in list_of_numbers] >print(x) NameError: name 'x' is not defined They "leak" their variables in Python 2, if you're someone who reuses variables this can lead to an enormous headache!

Cyph0n 11y ago
Wow, I never noticed that! Who the hell thought that was a good idea?
- bpicolo 11y ago
  Pretty sure it was a bug that they kept for backwards compat, not intentional, though I may be wrong. Py3 let them shed it off though.
  - masklinn 11y ago
    > Pretty sure it was a bug that they kept for backwards compat, not intentional Yep, as demonstrated by only list comprehensions exposing this behaviour (on python 2) whereas none of generator comprehensions, dictionary comprehensions or set comprehensions do: >>> [x for x in range(5)] [0, 1, 2, 3, 4] >>> x 4 >>> {y for y in range(5)} set([0, 1, 2, 3, 4]) >>> y Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'y' is not defined >>> {y:y for y in range(5)} {0: 0, 1: 1, 2: 2, 3: 3, 4: 4} >>> y Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'y' is not defined >>> list(y for y in range(5)) [0, 1, 2, 3, 4] >>> y Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'y' is not defined >>>
    - bpicolo 11y ago
      To be fair, generators are lazily evaluated so I would sure hope they wouldn't expose scope! That would show up in truly unique places haha
- maxerickson 11y ago
  It was largely an implementation detail. This mentions it some: http://python-history.blogspot.com/2010/06/from-list-comprehensions-to-generator.html http://python-history.blogspot.com/2010/06/from-list-compreh... The PEP is silent on the issue: https://www.python.org/dev/peps/pep-0202/ https://www.python.org/dev/peps/pep-0202/ I guess mail list spelunking would probably reveal some conversations about how to handle it.
- raymondh 11y ago
  Barry Warsaw proprosed List Comprehensions 16 years ago in PEP 202. The original concept was that it provided a more concise way to write loops so that: t = [expr(x, y) for x in s1 for y in s2] was just a short way to write: _t = [] for x in s1: for y in s2 _t.append(expr(x, y)) t = _t del _t The original implementation reflected that design. I later added the LIST_APPEND opcode to give the list comprehensions a speed advantage over the unrolled code. The question of whether to expose the loop induction variable didn't get much discussion until I proposed Generator Expressions in PEP 279 and decided to give them the behavior of hiding the loop induction variables so that the behavior would match that of a normal unrolled generator function. When set and dict comprehensions (displays) came along afterwards, they were given the latter behavior because there was precedent, because there was a mechanism to implement that precedent, and to provide a short-cut for then common practice of creating dicts and sets with generator comprehensions: s = {expr(x) for x in t} was a short-form for: s = set(expr(x) for x in t) The advent of Python 3 gave us an opportunity to make a four forms (listcomps, genexps, set and dict displays) consistent about hiding the loop induction variable. The current state in Python 3 has the advantage of being consistent between all four variants and matches how mathematicians treat bound and free variables. There are some disadvantages as well. List comprehensions can no longer be cleanly explained as being equivalent to the unrolled version. The disassembly is harder to explore be cause you need to drill into the internal code object. Tracing the execution with PDB is no fun because you go up and down the stack. It is more difficult to explain scoping -- formerly, all you had was locals/globals/builtins, but now we have locals/nonlocals/globals/builtins plus variables bound in list comps, genexps, set/dict displays plus exception instances that are only visible inside the except-block.
  - bearfrieze 11y ago
    Thanks for the excellent explanation and your great work on Python.
  - Cyph0n 11y ago
    Thanks for the detailed reply, Raymond! I never expected a core developer who worked on implementing list comprehensions to respond haha! That last point about traceability is something I never thought about, but I think the trade-off is worth it. Keep up the awesome work man. For anyone interested, you can follow him here: https://twitter.com/raymondh https://twitter.com/raymondh
deleted 11y ago
[deleted]
chishaku 11y ago
> One interesting side-thing with list comprehensions Just to clarify, this is the behavior of py2 for loops in general, whether in a comprehension or not.
- prlin 11y ago
  Except dictionary comprehensions don't leak their variables.
  - sophacles 11y ago
    Probably because dict comprehensions were introduced as a 3.0 feature, and put into 2.7 since that version is designed to be a bridge to 3.0. (They were actually proposed a long time ago, it was decided not to add them, but the decision changed to add them for 3.0)
bearfrieze 11y ago
Nice bonus info. I didn't know about this (haven't spent much time with Python 2.x).

Cyph0n 11y ago

Wow, I never noticed that! Who the hell thought that was a good idea?

bpicolo 11y ago
Pretty sure it was a bug that they kept for backwards compat, not intentional, though I may be wrong. Py3 let them shed it off though.
- masklinn 11y ago
  > Pretty sure it was a bug that they kept for backwards compat, not intentional Yep, as demonstrated by only list comprehensions exposing this behaviour (on python 2) whereas none of generator comprehensions, dictionary comprehensions or set comprehensions do: >>> [x for x in range(5)] [0, 1, 2, 3, 4] >>> x 4 >>> {y for y in range(5)} set([0, 1, 2, 3, 4]) >>> y Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'y' is not defined >>> {y:y for y in range(5)} {0: 0, 1: 1, 2: 2, 3: 3, 4: 4} >>> y Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'y' is not defined >>> list(y for y in range(5)) [0, 1, 2, 3, 4] >>> y Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'y' is not defined >>>
  - bpicolo 11y ago
    To be fair, generators are lazily evaluated so I would sure hope they wouldn't expose scope! That would show up in truly unique places haha
maxerickson 11y ago
It was largely an implementation detail. This mentions it some: http://python-history.blogspot.com/2010/06/from-list-comprehensions-to-generator.html http://python-history.blogspot.com/2010/06/from-list-compreh... The PEP is silent on the issue: https://www.python.org/dev/peps/pep-0202/ https://www.python.org/dev/peps/pep-0202/ I guess mail list spelunking would probably reveal some conversations about how to handle it.
raymondh 11y ago
Barry Warsaw proprosed List Comprehensions 16 years ago in PEP 202. The original concept was that it provided a more concise way to write loops so that: t = [expr(x, y) for x in s1 for y in s2] was just a short way to write: _t = [] for x in s1: for y in s2 _t.append(expr(x, y)) t = _t del _t The original implementation reflected that design. I later added the LIST_APPEND opcode to give the list comprehensions a speed advantage over the unrolled code. The question of whether to expose the loop induction variable didn't get much discussion until I proposed Generator Expressions in PEP 279 and decided to give them the behavior of hiding the loop induction variables so that the behavior would match that of a normal unrolled generator function. When set and dict comprehensions (displays) came along afterwards, they were given the latter behavior because there was precedent, because there was a mechanism to implement that precedent, and to provide a short-cut for then common practice of creating dicts and sets with generator comprehensions: s = {expr(x) for x in t} was a short-form for: s = set(expr(x) for x in t) The advent of Python 3 gave us an opportunity to make a four forms (listcomps, genexps, set and dict displays) consistent about hiding the loop induction variable. The current state in Python 3 has the advantage of being consistent between all four variants and matches how mathematicians treat bound and free variables. There are some disadvantages as well. List comprehensions can no longer be cleanly explained as being equivalent to the unrolled version. The disassembly is harder to explore be cause you need to drill into the internal code object. Tracing the execution with PDB is no fun because you go up and down the stack. It is more difficult to explain scoping -- formerly, all you had was locals/globals/builtins, but now we have locals/nonlocals/globals/builtins plus variables bound in list comps, genexps, set/dict displays plus exception instances that are only visible inside the except-block.
- bearfrieze 11y ago
  Thanks for the excellent explanation and your great work on Python.
- Cyph0n 11y ago
  Thanks for the detailed reply, Raymond! I never expected a core developer who worked on implementing list comprehensions to respond haha! That last point about traceability is something I never thought about, but I think the trade-off is worth it. Keep up the awesome work man. For anyone interested, you can follow him here: https://twitter.com/raymondh https://twitter.com/raymondh

deleted 11y ago

[deleted]

chishaku 11y ago

> One interesting side-thing with list comprehensions Just to clarify, this is the behavior of py2 for loops in general, whether in a comprehension or not.

prlin 11y ago
Except dictionary comprehensions don't leak their variables.
- sophacles 11y ago
  Probably because dict comprehensions were introduced as a 3.0 feature, and put into 2.7 since that version is designed to be a bridge to 3.0. (They were actually proposed a long time ago, it was decided not to add them, but the decision changed to add them for 3.0)

bearfrieze 11y ago

Nice bonus info. I didn't know about this (haven't spent much time with Python 2.x).