6 ms·

> Not only this will speed up comprehensions (which are already faster than map/filter) As someone coming from Haskell I find it odd that comprehensions are pr

> Not only this will speed up comprehensions (which are already faster than map/filter) As someone coming from Haskell I find it odd that comprehensions are prioritized over generic maps (fmaps) and traversals in Python. Not only they are more verbose syntax-wise and don't allow for partial application, they also discourage people from discovering generators, as most of the tutorials go with [] and {} brackets instead of the lazy (). At least with map() people could get acquainted with generators earlier. > I assume JIT will also appreciate the change, so let's see if pypy moves on this. I wonder if there's a real difference between map and comprehensions performance when it comes to JIT inlining.

zarzavat 4y ago
Guido hates higher-order functions. Once upon a time there was map, filter and reduce as builtins. Then for Python 3, they tried to remove all three, but failed due to community pushback. reduce() did get banished however. The argument is that making it hard to use higher-order functions discourages point-free style. "Explicit is better than implicit" is a design principle of the language.
- weatherlight 4y ago
  "Explicit is better than implicit" seems vacuous in the context of Higher Order Functions and it's applied functions. It's pretty explicit. Nothing is implied, it only can work one way.
  - roflyear 4y ago
    You don't explicitly pass the arguments to the function, is the point. I am not sure I agree with it but it's a thing to consider.
  - btilly 4y ago
    Well you can see Guido in his own words at https://www.artima.com/weblogs/viewpost.jsp?thread=98196 https://www.artima.com/weblogs/viewpost.jsp?thread=98196. The simple version is that higher order functions do not correspond to anything that is part of our common sense thinking. As a result people have created arbitrary terminology for it like "lambda" and "reduce". When you've mastered the terminology it may seem like a good way of doing things. But there are easy alternatives for doing the same things that DON'T require mastering that terminology. Python has a number of guiding principles, https://en.wikipedia.org/wiki/Zen_of_Python https://en.wikipedia.org/wiki/Zen_of_Python has a list, but one of them is, "There should be one– and preferably only one –obvious way to do it." And higher order functions do not win in Guido's mind. Separately, while the operation of higher order functions is explicit, code that makes heavy use of them tends to make behavior implicit and dependent on setup elsewhere. I have heard Guido complain about that exact style of coding. I have grudging sympathy for his view. Functional code can easily be a joy to write but a confusing nightmare to debug. And we spent more time on debugging than writing.
    - weatherlight 4y ago
      > The simple version is that higher order functions do not correspond to anything that is part of our common sense thinking. and what is common sense thinking here? are you talking about python or just programming in general? > Functional code can easily be a joy to write but a confusing nightmare to debug. And we spent more time on debugging than writing Citations needed. I'm not sure who WE is in this context. Python is an imperative lang and doesn't even do OO that well, let alone FP. but, if you are talking about programming in general, I do find it to be much easier to hunt down bugs in languages where the design philosophy of the language favors functional programing ergonomics. There's even been studies on this. https://cacm.acm.org/magazines/2017/10/221326-a-large-scale-study-of-programming-languages-and-code-quality-in-github/fulltext https://cacm.acm.org/magazines/2017/10/221326-a-large-scale-... tl;dr: "For those with positive coefficients we can expect that the language is associated with a greater number of defect fixes. These languages include C, C++, Objective-C, Php, and Python. The languages Clojure, Haskell, Ruby, and Scala, all have negative coefficients implying that these languages are less likely than average to result in defect fixing commits." > As a result people have created arbitrary terminology for it like "lambda" and "reduce". But Python does exactly that (I'm not knocking it, terminology is important, but its all arbitrary) In Python, we have "for" loops, we have "while" loops for lists and tuples(+ special syntax for "for" loops, when looping over a list of tuples!). Any sufficient digging into "for loops" and their implementation, and we begin to realize it's all just iterators under the hood completely hidden from the developer, but I digress. We have a special syntax for list comprehensions, (special syntax for nested comprehensions) that extend to ranges. There's special syntax in the form of specific statements that exist only in the context of a loop; "continue", "break", "pass", etc. There's all the implicit state passing into the bodies of these loops, and there's no referential transparency. "map", "reduce", and "filter" are specific and have no special constructs, no magic, it does what it says on the box. I'd argue it has a very similar mental overhead to a for loop. This is my experience, YMMV.
      - btilly 4y ago
        About common sense thinking. The phrase "for x in y" has an immediate common sense meaning in English. What it means in Python works out to that. Sure, you find a lot of complexity under the hood when you dive into the implementation. But the whole point of programming languages is to allow the programmer to think in terms of human concepts rather than machine implementations. And the concept maps directly to one that beginning programmers already have. By contrast I've yet to meet any non-programmer who has an intuition about what "map", "reduce", and "lambda" should mean. As great as they are as organizing concepts, they are something that programmers need to learn and gain experience with before it makes sense. For a programmer who has completely internalized the concepts, it may well be similar mental overhead to a for loop. But most programmers have NOT done so. And beginning programmers find for loops a heck of a lot easier to learn - because it corresponds to something that already makes sense to them. And the implementation is less "completely hidden" than you claim. Heck, the official tutorial at https://docs.python.org/3/tutorial/ https://docs.python.org/3/tutorial/ includes https://docs.python.org/3/tutorial/classes.html#iterators https://docs.python.org/3/tutorial/classes.html#iterators which shows you exactly how to poke under the hood and see how it works. As for my comment about spending more time debugging than writing, that should be obvious. As https://www.scirp.org/journal/paperinformation.aspx?paperid=51631 https://www.scirp.org/journal/paperinformation.aspx?paperid=... notes, about 70% of total software costs come in the maintenance phase. Therefore as developers we spend over 2x the time maintaining software rather than developing it. The actual ratio varies by project size, industry, language, tooling, developer experience and so on. But that's a pretty good average. As your link notes, the contribution due to language is real, but modest. My comment about functional programming being hard to maintain is based on anecdotal experience. I find my own functional code easy to maintain. But others have not. And my experience picking up other people's code has been at best mixed. Oh finally, about Python, comments like "doesn't do OO very well" usually mean "doesn't do OO like I wish it did". I find them to be highly individual opinions, and not apples to apples when you compare people.
- nerdponx 4y ago
  AFAICT Guido has softened his stance on this somewhat. The main impediment to `map()` and friends at this point is the clunky `lambda` syntax. However here we see another benefit of syntactic comprehensions over higher-order functions: comprehensions can be inspected deeply and inlined or otherwise optimized as a special case, while a stack of higher-order functions might need a more advanced bytecode compiler or JIT to apply "deforesting" and inlining.
gnulinux 4y ago
> don't allow for partial application Unfortunately, partial application doesn't work in Python, the same way it works in Haskell. If a function has 3 args, calling `f(x)` is a guaranteed runtime `TypeError`. There is functools.partial to hack it into the language for cases where you absolutely need it but overall it's bad developer experience. In fact, when I do need to partially apply `f(x, y, z)` I much prefer doing `g = lambda y, z: f(fixed_x, y, z)` because it's clearer than using functools.partial.
- nerdponx 4y ago
  I don't agree that the `lambda` is clearer than `partial`. This is a matter of personal style and preference, and IMO the top priority in cases like this is consistency within the codebase. You can also refactor partial function applications into callable classes depending on your needs, which is basically what `functools.partial` does internally anyway.
MathMonkeyMan 4y ago
Comprehensions vs. map/filter/etc. is a style thing. If you think in terms of an expression language that composes functions, then map and filter are more natural. If you think in terms of a statement language with special syntax for computation expressions, then comprehensions are more natural. Even in Racket (a scheme), idiomatic code seems to prefer the `for/thing` family of macros, which have a variety of flavors and keyword arguments to support map, filter, reduce, mapcar, etc. Whatever floats your boat.

zarzavat 4y ago

Guido hates higher-order functions. Once upon a time there was map, filter and reduce as builtins. Then for Python 3, they tried to remove all three, but failed due to community pushback. reduce() did get banished however. The argument is that making it hard to use higher-order functions discourages point-free style. "Explicit is better than implicit" is a design principle of the language.

weatherlight 4y ago
"Explicit is better than implicit" seems vacuous in the context of Higher Order Functions and it's applied functions. It's pretty explicit. Nothing is implied, it only can work one way.
- roflyear 4y ago
  You don't explicitly pass the arguments to the function, is the point. I am not sure I agree with it but it's a thing to consider.
- btilly 4y ago
  Well you can see Guido in his own words at https://www.artima.com/weblogs/viewpost.jsp?thread=98196 https://www.artima.com/weblogs/viewpost.jsp?thread=98196. The simple version is that higher order functions do not correspond to anything that is part of our common sense thinking. As a result people have created arbitrary terminology for it like "lambda" and "reduce". When you've mastered the terminology it may seem like a good way of doing things. But there are easy alternatives for doing the same things that DON'T require mastering that terminology. Python has a number of guiding principles, https://en.wikipedia.org/wiki/Zen_of_Python https://en.wikipedia.org/wiki/Zen_of_Python has a list, but one of them is, "There should be one– and preferably only one –obvious way to do it." And higher order functions do not win in Guido's mind. Separately, while the operation of higher order functions is explicit, code that makes heavy use of them tends to make behavior implicit and dependent on setup elsewhere. I have heard Guido complain about that exact style of coding. I have grudging sympathy for his view. Functional code can easily be a joy to write but a confusing nightmare to debug. And we spent more time on debugging than writing.
  - weatherlight 4y ago
    > The simple version is that higher order functions do not correspond to anything that is part of our common sense thinking. and what is common sense thinking here? are you talking about python or just programming in general? > Functional code can easily be a joy to write but a confusing nightmare to debug. And we spent more time on debugging than writing Citations needed. I'm not sure who WE is in this context. Python is an imperative lang and doesn't even do OO that well, let alone FP. but, if you are talking about programming in general, I do find it to be much easier to hunt down bugs in languages where the design philosophy of the language favors functional programing ergonomics. There's even been studies on this. https://cacm.acm.org/magazines/2017/10/221326-a-large-scale-study-of-programming-languages-and-code-quality-in-github/fulltext https://cacm.acm.org/magazines/2017/10/221326-a-large-scale-... tl;dr: "For those with positive coefficients we can expect that the language is associated with a greater number of defect fixes. These languages include C, C++, Objective-C, Php, and Python. The languages Clojure, Haskell, Ruby, and Scala, all have negative coefficients implying that these languages are less likely than average to result in defect fixing commits." > As a result people have created arbitrary terminology for it like "lambda" and "reduce". But Python does exactly that (I'm not knocking it, terminology is important, but its all arbitrary) In Python, we have "for" loops, we have "while" loops for lists and tuples(+ special syntax for "for" loops, when looping over a list of tuples!). Any sufficient digging into "for loops" and their implementation, and we begin to realize it's all just iterators under the hood completely hidden from the developer, but I digress. We have a special syntax for list comprehensions, (special syntax for nested comprehensions) that extend to ranges. There's special syntax in the form of specific statements that exist only in the context of a loop; "continue", "break", "pass", etc. There's all the implicit state passing into the bodies of these loops, and there's no referential transparency. "map", "reduce", and "filter" are specific and have no special constructs, no magic, it does what it says on the box. I'd argue it has a very similar mental overhead to a for loop. This is my experience, YMMV.
    - btilly 4y ago
      About common sense thinking. The phrase "for x in y" has an immediate common sense meaning in English. What it means in Python works out to that. Sure, you find a lot of complexity under the hood when you dive into the implementation. But the whole point of programming languages is to allow the programmer to think in terms of human concepts rather than machine implementations. And the concept maps directly to one that beginning programmers already have. By contrast I've yet to meet any non-programmer who has an intuition about what "map", "reduce", and "lambda" should mean. As great as they are as organizing concepts, they are something that programmers need to learn and gain experience with before it makes sense. For a programmer who has completely internalized the concepts, it may well be similar mental overhead to a for loop. But most programmers have NOT done so. And beginning programmers find for loops a heck of a lot easier to learn - because it corresponds to something that already makes sense to them. And the implementation is less "completely hidden" than you claim. Heck, the official tutorial at https://docs.python.org/3/tutorial/ https://docs.python.org/3/tutorial/ includes https://docs.python.org/3/tutorial/classes.html#iterators https://docs.python.org/3/tutorial/classes.html#iterators which shows you exactly how to poke under the hood and see how it works. As for my comment about spending more time debugging than writing, that should be obvious. As https://www.scirp.org/journal/paperinformation.aspx?paperid=51631 https://www.scirp.org/journal/paperinformation.aspx?paperid=... notes, about 70% of total software costs come in the maintenance phase. Therefore as developers we spend over 2x the time maintaining software rather than developing it. The actual ratio varies by project size, industry, language, tooling, developer experience and so on. But that's a pretty good average. As your link notes, the contribution due to language is real, but modest. My comment about functional programming being hard to maintain is based on anecdotal experience. I find my own functional code easy to maintain. But others have not. And my experience picking up other people's code has been at best mixed. Oh finally, about Python, comments like "doesn't do OO very well" usually mean "doesn't do OO like I wish it did". I find them to be highly individual opinions, and not apples to apples when you compare people.
nerdponx 4y ago
AFAICT Guido has softened his stance on this somewhat. The main impediment to `map()` and friends at this point is the clunky `lambda` syntax. However here we see another benefit of syntactic comprehensions over higher-order functions: comprehensions can be inspected deeply and inlined or otherwise optimized as a special case, while a stack of higher-order functions might need a more advanced bytecode compiler or JIT to apply "deforesting" and inlining.

gnulinux 4y ago

> don't allow for partial application Unfortunately, partial application doesn't work in Python, the same way it works in Haskell. If a function has 3 args, calling `f(x)` is a guaranteed runtime `TypeError`. There is functools.partial to hack it into the language for cases where you absolutely need it but overall it's bad developer experience. In fact, when I do need to partially apply `f(x, y, z)` I much prefer doing `g = lambda y, z: f(fixed_x, y, z)` because it's clearer than using functools.partial.

nerdponx 4y ago
I don't agree that the `lambda` is clearer than `partial`. This is a matter of personal style and preference, and IMO the top priority in cases like this is consistency within the codebase. You can also refactor partial function applications into callable classes depending on your needs, which is basically what `functools.partial` does internally anyway.

MathMonkeyMan 4y ago

Comprehensions vs. map/filter/etc. is a style thing. If you think in terms of an expression language that composes functions, then map and filter are more natural. If you think in terms of a statement language with special syntax for computation expressions, then comprehensions are more natural. Even in Racket (a scheme), idiomatic code seems to prefer the `for/thing` family of macros, which have a variety of flavors and keyword arguments to support map, filter, reduce, mapcar, etc. Whatever floats your boat.