7 ms·

Guido on the fate of Lambda, map() and filter(), and reduce() in Python 3000

28 points by smanek 18y ago

apathy 18y ago
wait a minute, you can't just expect python primitives to vectorize? And people live without any(), all(), etc? Its been a long time since I primarily wrote python, and I guess R has made me lazy, but seriously: WTF? Maybe hold off on Py3K until this is done?
- ivank 18y ago
  any() and all() are available in Python 2.5.
  - apathy 18y ago
    cool, I can't believe I haven't had occasion to use them yet. thank you
- kurtosis 18y ago
  numpy supports vectorization of most elementary functions
pg 18y ago
"filter(P, S) is almost always written clearer as [x for x in S if P(x)]" Is this widely believed by Python hackers?
- PieSquared 18y ago
  In some cases, it might be, such as when you're combining map and filter. Then the list comprehension just looks like psuedocode: include this in the result if these conditions are met. In that example. though, it isn't much clearer. And the 'x for x' seems sort of repetitive.
  - mariorz 18y ago
    I don't think the "x for x" is repetitive. It's there for the cases you mention. e.g. [str(x) for x in range(100) if x%2 is 0] VS map(lambda x: str(x), filter(lambda x: x%2 is 0, range(100)))
    - ken 18y ago
      Instead of lambda x: str(x), I'd just write str. I think this, and the P/S example, are both symptomatic of how Python (anthropomorphized) wants you to think about functions: they're things you stick parens after and call, not things you pass around on their own.
      - mariorz 18y ago
        I still think the list comprehension is clearer than: map(str, filter(lambda x: x%2 is 0, range(100)))
    - apu 18y ago
      I find comprehensions to be especially useful in two situations: 1. Transforming a list using some non-standard function, often with some basic input checking: [int(x)*2+1 for x in seq if x] (Here, we're assuming x is a string which can possibly be empty or only blanks) 2. Transforming elements from multiple lists together: ['Iter %d took %0.2f secs' % (i, t) for i, t in zip(iters, times) if t > 1.5] (Where we want output strings only if some iteration took a long time) Although I'm not too fluent in lisp, I think the equivalent formulations might be longer (since you'd have to write lambda, map, filter and many parens), and perhaps harder to decipher.
      - PieSquared 18y ago
        Beat this short Arc snippet using map with a list comprehension: (def vector-dot-product (vector-1 vector-2) (map [* _1 _2] vector-1 vector-2)) That's from Light Makes Right the raytracer, I think. Equivalent python: def vectorDotProduct(vector1, vector2): return [x * y for x, y in zip(vector1, vector2)] I'm pretty sure that the Lisp snippet above is neither harder to decipher nor longer. Though maybe that's just because Arc is nice.
- kurtosis 18y ago
  With filter you have to remember that the predicate is the first argument and the list is the second. With the list comprehension the syntactic cues make it obvious.
- newt0311 18y ago
  yes.
- jrockway 18y ago
  I think the idea is that [x for x in S if P(x)] is more general; it can do more than just filter. Knowing about filter and how it works will take up a small part of your mental capacity (that could be used for something else, perhaps), and all it can do is filter. Why learn two ways to do the same thing, when one way is more general? I think that's what Python hackers think, anyway. I use Perl :P
- ynd 18y ago
  It's a question of habit. If you're experienced with Python List comprehensions, they become clearer than anything else. "The main problem I see with Ruby is that the Principle of Least Surprise can lead you astray, as it did with implicit lexical scoping. The question is, whose surprise are you pessimizing? Experts are surprised by different things than beginners. People who are trying to grow small programs into large programs are surprised by different things than people who design their programs large to begin with. For instance, I think it's a violation of the Beginner's Principle of Least Surprise to make everything an object. To a beginner, a number is just a number. A string is a string. They may well be objects as far as the computer is concerned, and it's even fine for experts to treat them as objects. But premature OO is a speed bump in the novice's onramp. " Larry Wall, http://interviews.slashdot.org/article.pl?sid=02/09/06/1343222 http://interviews.slashdot.org/article.pl?sid=02/09/06/13432...
  - zenspider 18y ago
    You're quoting Larry "I've done more than anyone to fuck up programming languages" Wall as an authority on programer clarity?!? Um... no? here are some examples from real python (twisted)... let's see how much clearer they are than anything else: ''.join([''.join(['\x1b[' + str(n) + ch for n in ('', 2, 20, 200)]) for ch in 'BACD']) ''.join([''.join(['\x1b' + g + n for n in 'AB012']) for g in '()']) msgs = [os.path.join(b, mail.maildir._generateMaildirName()) for b in ('cur', 'new') for x in range(5)] It is code like this that makes me run for the hills... Even ignoring the list comprehensions, I use join as an example of how python gets everything backwards... ary.join(', ') # ruby - oop: tell the array to join its elements with a str join(', ', ary) # perl - fun: join ary with a str ', '.join(ary) # python - wtf: tell the str to join the elements of some other ary?!?
    - mleonhard 18y ago
      string.join(ary, ', ') # python - ok: join ary with a str The parameters to Python's join function are in a better order than Perl's. This is expected since Perl perverts the concept of function arity. Also, twisted is not an example of the best Python code.
- etal 18y ago
  If P is a built-in function taking a single argument, or one that's already been defined, no. That's often not the case, though, and using a simple lambda is annoying and usually less efficient the rest of the time. For example, filter(isfile, os.listdir(some_dir)) is cleaner than: (f for f in os.listdir(some_dir) if isfile(f)) But filter(lambda f: f.endswith('.py'), os.listdir(some_dir)) is worse than: (f for f in os.listdir(some_dir) if f.endswith('.py')) Ruby's blocks are something to be jealous of, in this case. List comprehensions and generator expressions have other merits, though.
  - shiro 18y ago
    I think this is the case that the choice of the syntax (and a bit of semantics) did affect the choice of API. I'm not a Python programmer, but I can see in your second example that they would prefer comprehensions to filter. If there were curring or some convenient partial application syntax in the language, using 'filter' can be more natural choice. In Gauche Scheme, pa$ is a built-in partial applicaiton and I'd write: (filter (pa$ string-suffix? ".py") (sys-readdir some-dir)) Or in Haskell: getDirectoryContents somedir >>= return . filter (isSuffixOf ".py") In general I'd rather write this in Gauche, for a regexp object is applicable: (filter #/\.py$/ (sys-readdir some-dir)) So, if you have more variations than the bare lambda to create a higher order function (either in currying, partial application, or promoting objects to applicable something, etc), the more the functions such as filter, fold etc become useful. If you need to fall back to lambda for most of times, it would be natural that using those fns becomes annoying.
  - pg 18y ago
    Arc's [] notation would help in the second case. filter([_.endswith('.py')], os.listdir(some_dir)) I've found that one of the most important things to eliminate from a program is unnecessary names. Introducing a variable is more than 1 token's worth of conceptual load.
  - eru 18y ago
    I wish there was an easy syntax (in Python) for not only map and filter - but for something like reduce/foldr/foldl as well. You still have to write loops to do this (or use reduce.)
- aaronsw 18y ago
  I've often wished that the "x for" could be optional, but yes. It's like bar charts and tables. Tables communicate more information, but we're visual creatures and thus prefer bar charts.
deleted 18y ago
[deleted]
riobard 18y ago
I think the problem of list comprehension is that it is not as flexible as combination of map() and filter(). For example, [F(x) for x if P(x)] is equivalent to map(F, filter(P, S)) but what about the equivalence of filter(P, map(F, S))? Without using any map() and filter(), this is the solution came up to my mind [x for x in [F(y) for y in S] if P(x)] Pretty ugly, huh?
- bamsjakalaka 18y ago
  [F(x) for x in S if P(F(x))]
  - riobard 18y ago
    That means you have to do F(x) TWICE, which is sometimes ineffective (when F(x) costs too much)

28 points by smanek 18y ago

wait a minute, you can't just expect python primitives to vectorize? And people live without any(), all(), etc? Its been a long time since I primarily wrote python, and I guess R has made me lazy, but seriously: WTF? Maybe hold off on Py3K until this is done?

ivank 18y ago
any() and all() are available in Python 2.5.
- apathy 18y ago
  cool, I can't believe I haven't had occasion to use them yet. thank you
kurtosis 18y ago
numpy supports vectorization of most elementary functions

pg 18y ago

"filter(P, S) is almost always written clearer as [x for x in S if P(x)]" Is this widely believed by Python hackers?

PieSquared 18y ago
In some cases, it might be, such as when you're combining map and filter. Then the list comprehension just looks like psuedocode: include this in the result if these conditions are met. In that example. though, it isn't much clearer. And the 'x for x' seems sort of repetitive.
- mariorz 18y ago
  I don't think the "x for x" is repetitive. It's there for the cases you mention. e.g. [str(x) for x in range(100) if x%2 is 0] VS map(lambda x: str(x), filter(lambda x: x%2 is 0, range(100)))
  - ken 18y ago
    Instead of lambda x: str(x), I'd just write str. I think this, and the P/S example, are both symptomatic of how Python (anthropomorphized) wants you to think about functions: they're things you stick parens after and call, not things you pass around on their own.
    - mariorz 18y ago
      I still think the list comprehension is clearer than: map(str, filter(lambda x: x%2 is 0, range(100)))
  - apu 18y ago
    I find comprehensions to be especially useful in two situations: 1. Transforming a list using some non-standard function, often with some basic input checking: [int(x)*2+1 for x in seq if x] (Here, we're assuming x is a string which can possibly be empty or only blanks) 2. Transforming elements from multiple lists together: ['Iter %d took %0.2f secs' % (i, t) for i, t in zip(iters, times) if t > 1.5] (Where we want output strings only if some iteration took a long time) Although I'm not too fluent in lisp, I think the equivalent formulations might be longer (since you'd have to write lambda, map, filter and many parens), and perhaps harder to decipher.
    - PieSquared 18y ago
      Beat this short Arc snippet using map with a list comprehension: (def vector-dot-product (vector-1 vector-2) (map [* _1 _2] vector-1 vector-2)) That's from Light Makes Right the raytracer, I think. Equivalent python: def vectorDotProduct(vector1, vector2): return [x * y for x, y in zip(vector1, vector2)] I'm pretty sure that the Lisp snippet above is neither harder to decipher nor longer. Though maybe that's just because Arc is nice.
kurtosis 18y ago
With filter you have to remember that the predicate is the first argument and the list is the second. With the list comprehension the syntactic cues make it obvious.
newt0311 18y ago
yes.
jrockway 18y ago
I think the idea is that [x for x in S if P(x)] is more general; it can do more than just filter. Knowing about filter and how it works will take up a small part of your mental capacity (that could be used for something else, perhaps), and all it can do is filter. Why learn two ways to do the same thing, when one way is more general? I think that's what Python hackers think, anyway. I use Perl :P
ynd 18y ago
It's a question of habit. If you're experienced with Python List comprehensions, they become clearer than anything else. "The main problem I see with Ruby is that the Principle of Least Surprise can lead you astray, as it did with implicit lexical scoping. The question is, whose surprise are you pessimizing? Experts are surprised by different things than beginners. People who are trying to grow small programs into large programs are surprised by different things than people who design their programs large to begin with. For instance, I think it's a violation of the Beginner's Principle of Least Surprise to make everything an object. To a beginner, a number is just a number. A string is a string. They may well be objects as far as the computer is concerned, and it's even fine for experts to treat them as objects. But premature OO is a speed bump in the novice's onramp. " Larry Wall, http://interviews.slashdot.org/article.pl?sid=02/09/06/1343222 http://interviews.slashdot.org/article.pl?sid=02/09/06/13432...
- zenspider 18y ago
  You're quoting Larry "I've done more than anyone to fuck up programming languages" Wall as an authority on programer clarity?!? Um... no? here are some examples from real python (twisted)... let's see how much clearer they are than anything else: ''.join([''.join(['\x1b[' + str(n) + ch for n in ('', 2, 20, 200)]) for ch in 'BACD']) ''.join([''.join(['\x1b' + g + n for n in 'AB012']) for g in '()']) msgs = [os.path.join(b, mail.maildir._generateMaildirName()) for b in ('cur', 'new') for x in range(5)] It is code like this that makes me run for the hills... Even ignoring the list comprehensions, I use join as an example of how python gets everything backwards... ary.join(', ') # ruby - oop: tell the array to join its elements with a str join(', ', ary) # perl - fun: join ary with a str ', '.join(ary) # python - wtf: tell the str to join the elements of some other ary?!?
  - mleonhard 18y ago
    string.join(ary, ', ') # python - ok: join ary with a str The parameters to Python's join function are in a better order than Perl's. This is expected since Perl perverts the concept of function arity. Also, twisted is not an example of the best Python code.
etal 18y ago
If P is a built-in function taking a single argument, or one that's already been defined, no. That's often not the case, though, and using a simple lambda is annoying and usually less efficient the rest of the time. For example, filter(isfile, os.listdir(some_dir)) is cleaner than: (f for f in os.listdir(some_dir) if isfile(f)) But filter(lambda f: f.endswith('.py'), os.listdir(some_dir)) is worse than: (f for f in os.listdir(some_dir) if f.endswith('.py')) Ruby's blocks are something to be jealous of, in this case. List comprehensions and generator expressions have other merits, though.
- shiro 18y ago
  I think this is the case that the choice of the syntax (and a bit of semantics) did affect the choice of API. I'm not a Python programmer, but I can see in your second example that they would prefer comprehensions to filter. If there were curring or some convenient partial application syntax in the language, using 'filter' can be more natural choice. In Gauche Scheme, pa$ is a built-in partial applicaiton and I'd write: (filter (pa$ string-suffix? ".py") (sys-readdir some-dir)) Or in Haskell: getDirectoryContents somedir >>= return . filter (isSuffixOf ".py") In general I'd rather write this in Gauche, for a regexp object is applicable: (filter #/\.py$/ (sys-readdir some-dir)) So, if you have more variations than the bare lambda to create a higher order function (either in currying, partial application, or promoting objects to applicable something, etc), the more the functions such as filter, fold etc become useful. If you need to fall back to lambda for most of times, it would be natural that using those fns becomes annoying.
- pg 18y ago
  Arc's [] notation would help in the second case. filter([_.endswith('.py')], os.listdir(some_dir)) I've found that one of the most important things to eliminate from a program is unnecessary names. Introducing a variable is more than 1 token's worth of conceptual load.
- eru 18y ago
  I wish there was an easy syntax (in Python) for not only map and filter - but for something like reduce/foldr/foldl as well. You still have to write loops to do this (or use reduce.)
aaronsw 18y ago
I've often wished that the "x for" could be optional, but yes. It's like bar charts and tables. Tables communicate more information, but we're visual creatures and thus prefer bar charts.

deleted 18y ago

[deleted]

riobard 18y ago

I think the problem of list comprehension is that it is not as flexible as combination of map() and filter(). For example, [F(x) for x if P(x)] is equivalent to map(F, filter(P, S)) but what about the equivalence of filter(P, map(F, S))? Without using any map() and filter(), this is the solution came up to my mind [x for x in [F(y) for y in S] if P(x)] Pretty ugly, huh?

bamsjakalaka 18y ago
[F(x) for x in S if P(F(x))]
- riobard 18y ago
  That means you have to do F(x) TWICE, which is sometimes ineffective (when F(x) costs too much)