8 ms·

PEP 709 – Inlined Comprehensions

126 points by BiteCode_dev 4y ago

BiteCode_dev 4y ago
Not only this will speed up comprehensions (which are already faster than map/filter), but it will likely solve this annoying weird edge case with the scoping of lambdas in comprehensions. Last but not least, since it removes the additional frame that the comprensions used be wrapped in, it will fix the way you interact with them using pdb, which has always been rough thanks to this invisible indirection layer. I assume JIT will also appreciate the change, so let's see if pypy moves on this. Migrating to a PEG parser opened so many opportunities for Python. I never expected we would see so many good things come out of it, after years of defending LL1 to keep the reference implementation simple. E.G: the new error messages are awesome.
- nneonneo 4y ago
  So, out of curiosity: what does the PEG parser have to do with this particular PEP? It seems like this was just a somewhat inefficient (but simple) implementation choice from the start that is now being improved.
  - nerdponx 4y ago
    I don't know that it matters here specifically. But I think it was a kind of watershed moment in terms of how the development approach to CPython has changed.
- ghostwriter 4y ago
  > Not only this will speed up comprehensions (which are already faster than map/filter) As someone coming from Haskell I find it odd that comprehensions are prioritized over generic maps (fmaps) and traversals in Python. Not only they are more verbose syntax-wise and don't allow for partial application, they also discourage people from discovering generators, as most of the tutorials go with [] and {} brackets instead of the lazy (). At least with map() people could get acquainted with generators earlier. > I assume JIT will also appreciate the change, so let's see if pypy moves on this. I wonder if there's a real difference between map and comprehensions performance when it comes to JIT inlining.
  - zarzavat 4y ago
    Guido hates higher-order functions. Once upon a time there was map, filter and reduce as builtins. Then for Python 3, they tried to remove all three, but failed due to community pushback. reduce() did get banished however. The argument is that making it hard to use higher-order functions discourages point-free style. "Explicit is better than implicit" is a design principle of the language.
    - weatherlight 4y ago
      "Explicit is better than implicit" seems vacuous in the context of Higher Order Functions and it's applied functions. It's pretty explicit. Nothing is implied, it only can work one way.
      - roflyear 4y ago
        You don't explicitly pass the arguments to the function, is the point. I am not sure I agree with it but it's a thing to consider.
      - btilly 4y ago
        Well you can see Guido in his own words at https://www.artima.com/weblogs/viewpost.jsp?thread=98196 https://www.artima.com/weblogs/viewpost.jsp?thread=98196. The simple version is that higher order functions do not correspond to anything that is part of our common sense thinking. As a result people have created arbitrary terminology for it like "lambda" and "reduce". When you've mastered the terminology it may seem like a good way of doing things. But there are easy alternatives for doing the same things that DON'T require mastering that terminology. Python has a number of guiding principles, https://en.wikipedia.org/wiki/Zen_of_Python https://en.wikipedia.org/wiki/Zen_of_Python has a list, but one of them is, "There should be one– and preferably only one –obvious way to do it." And higher order functions do not win in Guido's mind. Separately, while the operation of higher order functions is explicit, code that makes heavy use of them tends to make behavior implicit and dependent on setup elsewhere. I have heard Guido complain about that exact style of coding. I have grudging sympathy for his view. Functional code can easily be a joy to write but a confusing nightmare to debug. And we spent more time on debugging than writing.
- Pinus 4y ago
  Are you sure about the lambda scope bit? It seems to me there will still be only one loop variable, which gets re-used throughout the loop (unlike, for example, the 'do' loop in Scheme, where each turn through the loop explicitly gets new locations for all variables).
frognumber 4y ago
The semantics are much nicer after than before.

126 points by BiteCode_dev 4y ago

Not only this will speed up comprehensions (which are already faster than map/filter), but it will likely solve this annoying weird edge case with the scoping of lambdas in comprehensions. Last but not least, since it removes the additional frame that the comprensions used be wrapped in, it will fix the way you interact with them using pdb, which has always been rough thanks to this invisible indirection layer. I assume JIT will also appreciate the change, so let's see if pypy moves on this. Migrating to a PEG parser opened so many opportunities for Python. I never expected we would see so many good things come out of it, after years of defending LL1 to keep the reference implementation simple. E.G: the new error messages are awesome.

nneonneo 4y ago
So, out of curiosity: what does the PEG parser have to do with this particular PEP? It seems like this was just a somewhat inefficient (but simple) implementation choice from the start that is now being improved.
- nerdponx 4y ago
  I don't know that it matters here specifically. But I think it was a kind of watershed moment in terms of how the development approach to CPython has changed.
ghostwriter 4y ago
> Not only this will speed up comprehensions (which are already faster than map/filter) As someone coming from Haskell I find it odd that comprehensions are prioritized over generic maps (fmaps) and traversals in Python. Not only they are more verbose syntax-wise and don't allow for partial application, they also discourage people from discovering generators, as most of the tutorials go with [] and {} brackets instead of the lazy (). At least with map() people could get acquainted with generators earlier. > I assume JIT will also appreciate the change, so let's see if pypy moves on this. I wonder if there's a real difference between map and comprehensions performance when it comes to JIT inlining.
- zarzavat 4y ago
  Guido hates higher-order functions. Once upon a time there was map, filter and reduce as builtins. Then for Python 3, they tried to remove all three, but failed due to community pushback. reduce() did get banished however. The argument is that making it hard to use higher-order functions discourages point-free style. "Explicit is better than implicit" is a design principle of the language.
  - weatherlight 4y ago
    "Explicit is better than implicit" seems vacuous in the context of Higher Order Functions and it's applied functions. It's pretty explicit. Nothing is implied, it only can work one way.
    - roflyear 4y ago
      You don't explicitly pass the arguments to the function, is the point. I am not sure I agree with it but it's a thing to consider.
    - btilly 4y ago
      Well you can see Guido in his own words at https://www.artima.com/weblogs/viewpost.jsp?thread=98196 https://www.artima.com/weblogs/viewpost.jsp?thread=98196. The simple version is that higher order functions do not correspond to anything that is part of our common sense thinking. As a result people have created arbitrary terminology for it like "lambda" and "reduce". When you've mastered the terminology it may seem like a good way of doing things. But there are easy alternatives for doing the same things that DON'T require mastering that terminology. Python has a number of guiding principles, https://en.wikipedia.org/wiki/Zen_of_Python https://en.wikipedia.org/wiki/Zen_of_Python has a list, but one of them is, "There should be one– and preferably only one –obvious way to do it." And higher order functions do not win in Guido's mind. Separately, while the operation of higher order functions is explicit, code that makes heavy use of them tends to make behavior implicit and dependent on setup elsewhere. I have heard Guido complain about that exact style of coding. I have grudging sympathy for his view. Functional code can easily be a joy to write but a confusing nightmare to debug. And we spent more time on debugging than writing.
Pinus 4y ago
Are you sure about the lambda scope bit? It seems to me there will still be only one loop variable, which gets re-used throughout the loop (unlike, for example, the 'do' loop in Scheme, where each turn through the loop explicitly gets new locations for all variables).

frognumber 4y ago

The semantics are much nicer after than before.