5 ms·

while > for > foreach - increasingly higher level abstractions. article stops at iteration (foreach) but misses an even higher level abstraction: filter, map,

while > for > foreach - increasingly higher level abstractions. article stops at iteration (foreach) but misses an even higher level abstraction: filter, map, reduce. each of these can be implemented on top of foreach, and i haven't yet seen a loop that can't be decomposed into a combination of map, filter, reduce. also note that filter/map/reduce are expressions, not statements, so they are more "mathy" and thus easier to reason about. its interesting that these are not necessarily idiomatic python despite being a slightly higher level abstraction. note they are idiomatic in all "real" functional languages, which python is decidedly not. also list comprehensions are faster than map/filter in python. (this is probably python's fault and is really dumb because you could just implement map/filter as a list comprehension. some people have criticized Guido for some of his design choices[1].) [1] http://www.quora.com/What-are-the-main-weaknesses-of-Python-as-a-programming-language http://www.quora.com/What-are-the-main-weaknesses-of-Python-... also on topic: http://docs.python.org/library/itertools.html http://docs.python.org/library/itertools.html

ggchappell 15y ago
He does do a "map", although without using the function of that name: answers = [f(x) for x in iterable] Also, the "zip" operation gives loops that seem to me to be unable to be decomposed into map, filter, reduce. E.g., how would you print a file with line numbers using only map, filter, reduce? Regardless, I agree with your basic point: that thinking of loops in terms of high-level loop primitives, is a Good Thing.
- settrans 15y ago
  Line numbering a file with reduce is straightforward: def line_number_catamorphism(fp): def accumulate(a, line): line_number, lines = a return line_number + 1, lines + [(line_number + 1, line)] return reduce(accumulate, fp, (0, []))[1] It may be worth nothing that zip can be straightforwardly expressed in terms of map as well, i.e.: zip = lambda *a: map(None, *a)
  - ned_batchelder 15y ago
    I'm sorry, is this a joke? "straightforward"? Why would you use that code when you could implement the same thing more simply like this? def line_number_mogrifier(f): num = 0 for line in f: yield (num, line) num += 1 The functional proponents say that eventually everything will be functional, I guess I'll wait and see how that goes over. People think procedurally. Functional constructs may have some technical advantages, but if adopting them shrinks the pool of effective developers, it won't catch on.
- dustingetz 15y ago
  def zip(*seqs): return map(lambda *items: items, *seqs) assert [(1, 'a'), (2, 'b'), (3, 'c')] == zip([1,2,3],['a','b','c']) def count(start=0): while True: yield start start += 1 from itertools import izip, islice # lazy versions with open('/etc/passwd', 'r') as f: zipped = izip(count(), f.readlines()) for numbered in zipped: print numbered izip and islice are trivial to implement: https://github.com/dustingetz/sandbox/blob/master/etc/lazy.py https://github.com/dustingetz/sandbox/blob/master/etc/lazy.p...
  - ggchappell 15y ago
    Well, strictly speaking, you have answered my objection. But I guess I was thinking in Haskell. If Python's map takes multiple iterables, then, yes, zip is a special case. But writing zip by mapping a function without side effects, using a "map" that works only on a single iterable (as in Haskell) strikes me as a rather different thing. So I suppose that the truth of "this loop can be written using map" depends on exactly what one means by "map".
    - dustingetz 15y ago
      interesting. how about def head(seq): return seq[0] def rest(seq): return seq[1:] def transpose(mtx): a = map(head, mtx) b = map(rest, mtx) return [a] + ([transpose(b)] if b[0] else []) def zip(xs, ys): return transpose((xs, ys)) def zip2(*seqs): return transpose(seqs) this works but is sort of icky in python, any ideas? this is also where functional python starts to fall apart - no tail call recursion, and default data structures aren't persistent. based on a haskell solution which is nicer. http://stackoverflow.com/questions/2578930/understanding-this-matrix-transposition-function-in-haskell http://stackoverflow.com/questions/2578930/understanding-thi...
deleted 15y ago
[deleted]
MaxGabriel 15y ago
Can you control the ordering of map/filter/reduce? As in, I want to iterate through a matrix by iterating through the columns and rows in reverse order? As in: [9,6,3] [8,5,2] [7,4,1] I've just started using map/filter/reduce so I'm not sure.
- kisielk 15y ago
  What data structure are you using to represent your matrix? A list of lists? If so, it's not terribly difficult: >>> a = [[1,2,3], [4,5,6], [7,8,9]] >>> b = zip(*map(reversed, reversed(a)) >>> b [(9, 6, 3), (8, 5, 2), (7, 4, 1)] >>> reduce(operator.add, b) # could also use a lambda with + instead of operator.add... (9, 6, 3, 8, 5, 2, 7, 4, 1)
- dalke 15y ago
  No. They start from the beginning and go to the end. Conceptually it could work on any iterable, and not just a list container. You can, however, use the 'reversed()' function, as in "for row in reversed(rows): for col in reversed(row): print col" There is a special double-underscore method, which lists and other data structures support, to allow iteration in reverse with the normal performance of forward iteration.
lloeki 15y ago
> its interesting that these are not necessarily idiomatic python despite being a slightly higher level abstraction. note they are idiomatic in all "real" functional languages, which python is decidedly not. also list comprehensions are faster than map/filter in python. (this is probably python's fault and is really dumb because you could just implement map/filter as a list comprehension. some people have criticized Guido for some of his design choices.) I don't miss map/reduce syntax the slightest. List comprehensions (for when I need to read multiple times a list result) and generators expressions (for lazy evaluation and genericity) are incredibly readable and efficient . Here's an example script[0] I threw in a cron to report on some daily FTP transfers from a legacy system. Those list comprehensions are very descriptive, like a mathematical set syntax. In comparison e.g Ruby's map/inject feels terribly awkward and alien to me. [1] (and [2], although a bit academic) is an invaluable resource on the subject. [0] https://gist.github.com/2492951 https://gist.github.com/2492951 [1] http://www.dabeaz.com/generators/ http://www.dabeaz.com/generators/ [2] http://www.dabeaz.com/coroutines/ http://www.dabeaz.com/coroutines/

ggchappell 15y ago

He does do a "map", although without using the function of that name: answers = [f(x) for x in iterable] Also, the "zip" operation gives loops that seem to me to be unable to be decomposed into map, filter, reduce. E.g., how would you print a file with line numbers using only map, filter, reduce? Regardless, I agree with your basic point: that thinking of loops in terms of high-level loop primitives, is a Good Thing.

settrans 15y ago
Line numbering a file with reduce is straightforward: def line_number_catamorphism(fp): def accumulate(a, line): line_number, lines = a return line_number + 1, lines + [(line_number + 1, line)] return reduce(accumulate, fp, (0, []))[1] It may be worth nothing that zip can be straightforwardly expressed in terms of map as well, i.e.: zip = lambda *a: map(None, *a)
- ned_batchelder 15y ago
  I'm sorry, is this a joke? "straightforward"? Why would you use that code when you could implement the same thing more simply like this? def line_number_mogrifier(f): num = 0 for line in f: yield (num, line) num += 1 The functional proponents say that eventually everything will be functional, I guess I'll wait and see how that goes over. People think procedurally. Functional constructs may have some technical advantages, but if adopting them shrinks the pool of effective developers, it won't catch on.
dustingetz 15y ago
def zip(*seqs): return map(lambda *items: items, *seqs) assert [(1, 'a'), (2, 'b'), (3, 'c')] == zip([1,2,3],['a','b','c']) def count(start=0): while True: yield start start += 1 from itertools import izip, islice # lazy versions with open('/etc/passwd', 'r') as f: zipped = izip(count(), f.readlines()) for numbered in zipped: print numbered izip and islice are trivial to implement: https://github.com/dustingetz/sandbox/blob/master/etc/lazy.py https://github.com/dustingetz/sandbox/blob/master/etc/lazy.p...
- ggchappell 15y ago
  Well, strictly speaking, you have answered my objection. But I guess I was thinking in Haskell. If Python's map takes multiple iterables, then, yes, zip is a special case. But writing zip by mapping a function without side effects, using a "map" that works only on a single iterable (as in Haskell) strikes me as a rather different thing. So I suppose that the truth of "this loop can be written using map" depends on exactly what one means by "map".
  - dustingetz 15y ago
    interesting. how about def head(seq): return seq[0] def rest(seq): return seq[1:] def transpose(mtx): a = map(head, mtx) b = map(rest, mtx) return [a] + ([transpose(b)] if b[0] else []) def zip(xs, ys): return transpose((xs, ys)) def zip2(*seqs): return transpose(seqs) this works but is sort of icky in python, any ideas? this is also where functional python starts to fall apart - no tail call recursion, and default data structures aren't persistent. based on a haskell solution which is nicer. http://stackoverflow.com/questions/2578930/understanding-this-matrix-transposition-function-in-haskell http://stackoverflow.com/questions/2578930/understanding-thi...

deleted 15y ago

[deleted]

MaxGabriel 15y ago

Can you control the ordering of map/filter/reduce? As in, I want to iterate through a matrix by iterating through the columns and rows in reverse order? As in: [9,6,3] [8,5,2] [7,4,1] I've just started using map/filter/reduce so I'm not sure.

kisielk 15y ago
What data structure are you using to represent your matrix? A list of lists? If so, it's not terribly difficult: >>> a = [[1,2,3], [4,5,6], [7,8,9]] >>> b = zip(*map(reversed, reversed(a)) >>> b [(9, 6, 3), (8, 5, 2), (7, 4, 1)] >>> reduce(operator.add, b) # could also use a lambda with + instead of operator.add... (9, 6, 3, 8, 5, 2, 7, 4, 1)
dalke 15y ago
No. They start from the beginning and go to the end. Conceptually it could work on any iterable, and not just a list container. You can, however, use the 'reversed()' function, as in "for row in reversed(rows): for col in reversed(row): print col" There is a special double-underscore method, which lists and other data structures support, to allow iteration in reverse with the normal performance of forward iteration.

lloeki 15y ago

> its interesting that these are not necessarily idiomatic python despite being a slightly higher level abstraction. note they are idiomatic in all "real" functional languages, which python is decidedly not. also list comprehensions are faster than map/filter in python. (this is probably python's fault and is really dumb because you could just implement map/filter as a list comprehension. some people have criticized Guido for some of his design choices.) I don't miss map/reduce syntax the slightest. List comprehensions (for when I need to read multiple times a list result) and generators expressions (for lazy evaluation and genericity) are incredibly readable and efficient . Here's an example script[0] I threw in a cron to report on some daily FTP transfers from a legacy system. Those list comprehensions are very descriptive, like a mathematical set syntax. In comparison e.g Ruby's map/inject feels terribly awkward and alien to me. [1] (and [2], although a bit academic) is an invaluable resource on the subject. [0] https://gist.github.com/2492951 https://gist.github.com/2492951 [1] http://www.dabeaz.com/generators/ http://www.dabeaz.com/generators/ [2] http://www.dabeaz.com/coroutines/ http://www.dabeaz.com/coroutines/