7 ms·

Python iteration

113 points by joshbaptiste 15y ago

benhoyt 15y ago
Simple and useful overview. I find enumerate() quite useful. I should probably define generators more often. One built-in function I basically never use is zip(). I know how it's used and what it does, and I grok the names/ages types of examples that are always used to explain it. But I've simply never needed it in real code. Anyone else use it often, and in what contexts?
- grifaton 15y ago
  I sometimes use zip as a poor man's matrix transposition operator: >>> m = [(1,2,3), (4,5,6), (7,8,9)] >>> zip(*m) [(1, 4, 7), (2, 5, 8), (3, 6, 9)]
  - christianmann 15y ago
    That's pretty.
  - mickeyp 15y ago
    If you want to transpose a matrix-like list of lists, but the sizes don't all match up, you can use izip_longest from itertools and give it a fillvalue to "fill in" for the missing elements: >>> m = [(1,2,3), (4,5,6), (7, 8)] >>> zip(*m) [(1, 4, 7), (2, 5, 8)] # Wrong >>> from itertools import izip_longest >>> list(izip_longest(*m, fillvalue=None)) [(1, 4, 7), (2, 5, 8), (3, 6, None)]
- samdk 15y ago
  I use it rarely in Python, although it's sometimes useful. I use it more often in other languages to replicate Python's enumerate, since zip(range(len(seq)),seq) is basically enumerate(seq) (although with this implementation it's not a generator).
  - koenigdavidmj 15y ago
    Ah, but this is: from itertools import izip izip(xrange(len(seq)), seq)
    - kbd 15y ago
      And in Python 3 it's a generator with zip(range(len(seq)), seq)
- andylei 15y ago
  i use it all the time. mostly to loop over n things at the same time: for a, b in zip(list_a, list_b): print a + b
- riobard 15y ago
  Quickly make a dict: >>> keys = ["foo", "bar", "spam", "egg"] >>> values = [1, 2, 3, 4] >>> dict(zip(keys, values)) {'egg': 4, 'foo': 1, 'bar': 2, 'spam': 3} Quite useful when you MGET from Redis.
- AncientPC 15y ago
  For example, trying to solve this problem (CodeJam 2008 minimum scalar product): You are given two vectors v1=(x1,x2,...,xn) and v2=(y1,y2,...,yn). The scalar product of these vectors is a single number, calculated as x1y1+x2y2+...+xnyn. Haskell: sum(zipWith (*) [1,2,3][3,2,1]) Python is not inherently functional, just supports a few convenience idioms. It has zip, but no zipWith. As a result you have to use list comprehension to perform the same: sum([x*y for (x,y) in zip([1,2,3][3,2,1])])
  - Lycanthrope 15y ago
    You can use map with several iterables: sum(map(operator.mul, [1,2,3], [3,2,1]))
- charliesome 15y ago
  I've used zip before to find an LCA: def lca(a,b) a.ancestors.zip(b.ancestors).take_while { |a,b| a == b }.last end
dustingetz 15y ago
while > for > foreach - increasingly higher level abstractions. article stops at iteration (foreach) but misses an even higher level abstraction: filter, map, reduce. each of these can be implemented on top of foreach, and i haven't yet seen a loop that can't be decomposed into a combination of map, filter, reduce. also note that filter/map/reduce are expressions, not statements, so they are more "mathy" and thus easier to reason about. its interesting that these are not necessarily idiomatic python despite being a slightly higher level abstraction. note they are idiomatic in all "real" functional languages, which python is decidedly not. also list comprehensions are faster than map/filter in python. (this is probably python's fault and is really dumb because you could just implement map/filter as a list comprehension. some people have criticized Guido for some of his design choices[1].) [1] http://www.quora.com/What-are-the-main-weaknesses-of-Python-as-a-programming-language http://www.quora.com/What-are-the-main-weaknesses-of-Python-... also on topic: http://docs.python.org/library/itertools.html http://docs.python.org/library/itertools.html
- ggchappell 15y ago
  He does do a "map", although without using the function of that name: answers = [f(x) for x in iterable] Also, the "zip" operation gives loops that seem to me to be unable to be decomposed into map, filter, reduce. E.g., how would you print a file with line numbers using only map, filter, reduce? Regardless, I agree with your basic point: that thinking of loops in terms of high-level loop primitives, is a Good Thing.
  - settrans 15y ago
    Line numbering a file with reduce is straightforward: def line_number_catamorphism(fp): def accumulate(a, line): line_number, lines = a return line_number + 1, lines + [(line_number + 1, line)] return reduce(accumulate, fp, (0, []))[1] It may be worth nothing that zip can be straightforwardly expressed in terms of map as well, i.e.: zip = lambda *a: map(None, *a)
    - ned_batchelder 15y ago
krupan 15y ago
First examples of generators that I've seen that made them look useful. I'd never thought of using a generator as a filter, or for turning nested for-loops into one for-loop. I have some code I can go clean up now.
- ff0066mote 15y ago
  Here's a pile more to blow your mind with: http://www.dabeaz.com/coroutines/ http://www.dabeaz.com/coroutines/ [2009]
ff0066mote 15y ago
I would have liked to have seen mention of this while loop instead of the index-based one. while my_list: v = my_list.pop(0) print v Although it doesn't fit the theme of the others because it consumes the list, it is certainly a more natural use of "while" to loop over a list.

113 points by joshbaptiste 15y ago

Simple and useful overview. I find enumerate() quite useful. I should probably define generators more often. One built-in function I basically never use is zip(). I know how it's used and what it does, and I grok the names/ages types of examples that are always used to explain it. But I've simply never needed it in real code. Anyone else use it often, and in what contexts?

grifaton 15y ago
I sometimes use zip as a poor man's matrix transposition operator: >>> m = [(1,2,3), (4,5,6), (7,8,9)] >>> zip(*m) [(1, 4, 7), (2, 5, 8), (3, 6, 9)]
- christianmann 15y ago
  That's pretty.
- mickeyp 15y ago
  If you want to transpose a matrix-like list of lists, but the sizes don't all match up, you can use izip_longest from itertools and give it a fillvalue to "fill in" for the missing elements: >>> m = [(1,2,3), (4,5,6), (7, 8)] >>> zip(*m) [(1, 4, 7), (2, 5, 8)] # Wrong >>> from itertools import izip_longest >>> list(izip_longest(*m, fillvalue=None)) [(1, 4, 7), (2, 5, 8), (3, 6, None)]
samdk 15y ago
I use it rarely in Python, although it's sometimes useful. I use it more often in other languages to replicate Python's enumerate, since zip(range(len(seq)),seq) is basically enumerate(seq) (although with this implementation it's not a generator).
- koenigdavidmj 15y ago
  Ah, but this is: from itertools import izip izip(xrange(len(seq)), seq)
  - kbd 15y ago
    And in Python 3 it's a generator with zip(range(len(seq)), seq)
andylei 15y ago
i use it all the time. mostly to loop over n things at the same time: for a, b in zip(list_a, list_b): print a + b
riobard 15y ago
Quickly make a dict: >>> keys = ["foo", "bar", "spam", "egg"] >>> values = [1, 2, 3, 4] >>> dict(zip(keys, values)) {'egg': 4, 'foo': 1, 'bar': 2, 'spam': 3} Quite useful when you MGET from Redis.
AncientPC 15y ago
For example, trying to solve this problem (CodeJam 2008 minimum scalar product): You are given two vectors v1=(x1,x2,...,xn) and v2=(y1,y2,...,yn). The scalar product of these vectors is a single number, calculated as x1y1+x2y2+...+xnyn. Haskell: sum(zipWith (*) [1,2,3][3,2,1]) Python is not inherently functional, just supports a few convenience idioms. It has zip, but no zipWith. As a result you have to use list comprehension to perform the same: sum([x*y for (x,y) in zip([1,2,3][3,2,1])])
- Lycanthrope 15y ago
  You can use map with several iterables: sum(map(operator.mul, [1,2,3], [3,2,1]))
charliesome 15y ago
I've used zip before to find an LCA: def lca(a,b) a.ancestors.zip(b.ancestors).take_while { |a,b| a == b }.last end

dustingetz 15y ago

while > for > foreach - increasingly higher level abstractions. article stops at iteration (foreach) but misses an even higher level abstraction: filter, map, reduce. each of these can be implemented on top of foreach, and i haven't yet seen a loop that can't be decomposed into a combination of map, filter, reduce. also note that filter/map/reduce are expressions, not statements, so they are more "mathy" and thus easier to reason about. its interesting that these are not necessarily idiomatic python despite being a slightly higher level abstraction. note they are idiomatic in all "real" functional languages, which python is decidedly not. also list comprehensions are faster than map/filter in python. (this is probably python's fault and is really dumb because you could just implement map/filter as a list comprehension. some people have criticized Guido for some of his design choices[1].) [1] http://www.quora.com/What-are-the-main-weaknesses-of-Python-as-a-programming-language http://www.quora.com/What-are-the-main-weaknesses-of-Python-... also on topic: http://docs.python.org/library/itertools.html http://docs.python.org/library/itertools.html

ggchappell 15y ago
He does do a "map", although without using the function of that name: answers = [f(x) for x in iterable] Also, the "zip" operation gives loops that seem to me to be unable to be decomposed into map, filter, reduce. E.g., how would you print a file with line numbers using only map, filter, reduce? Regardless, I agree with your basic point: that thinking of loops in terms of high-level loop primitives, is a Good Thing.
- settrans 15y ago
  Line numbering a file with reduce is straightforward: def line_number_catamorphism(fp): def accumulate(a, line): line_number, lines = a return line_number + 1, lines + [(line_number + 1, line)] return reduce(accumulate, fp, (0, []))[1] It may be worth nothing that zip can be straightforwardly expressed in terms of map as well, i.e.: zip = lambda *a: map(None, *a)
  - ned_batchelder 15y ago

krupan 15y ago

First examples of generators that I've seen that made them look useful. I'd never thought of using a generator as a filter, or for turning nested for-loops into one for-loop. I have some code I can go clean up now.

ff0066mote 15y ago
Here's a pile more to blow your mind with: http://www.dabeaz.com/coroutines/ http://www.dabeaz.com/coroutines/ [2009]

ff0066mote 15y ago

I would have liked to have seen mention of this while loop instead of the index-based one. while my_list: v = my_list.pop(0) print v Although it doesn't fit the theme of the others because it consumes the list, it is certainly a more natural use of "while" to loop over a list.