5 ms·

Python idioms

169 points by preek 15y ago

Ysx 15y ago
Good stuff, though there's a few outdated idioms: Lists, or any iterable, can be reverse-sorted with reversed(sorted(my_list)). That'll give you an iterator, though you can call list() on the result if you need it. "while True" should be used in-place of "while 1". Reads better. In Python 2, xrange() is preferred over range() when looping - it won't create an in-memory list of integers, and behaves mostly the same as range but for a few edge cases. Python 3 renamed xrange() to range(), and removed the original range() function.
- shazow 15y ago
  > reversed(sorted(my_list)) I thought sorted(my_list, reverse=True) would be slightly faster, but it seems not (by a tiny amount). Weird. > "while True" should be used in-place of "while 1". Reads better. If you disassemble these statements, you'll see that "while 1" creates fewer instructions: def foo(): while True: pass def bar(): while 1: pass import dis dis.dis(foo) # 0 SETUP_LOOP 10 (to 13) # >> 3 LOAD_GLOBAL 0 (True) # 6 POP_JUMP_IF_FALSE 12 # # 9 JUMP_ABSOLUTE 3 # >> 12 POP_BLOCK # >> 13 LOAD_CONST 0 (None) # 16 RETURN_VALUE dis.dis(bar) # 0 SETUP_LOOP 3 (to 6) # # >> 3 JUMP_ABSOLUTE 3 # >> 6 LOAD_CONST 0 (None) # 9 RETURN_VALUE Edit: This is in Python 2.7.1. d0mine pointed out that the discrepancy is no longer the case in Python 3. Good to know. :)
  - d0mine 15y ago
    There is no difference on Python 3: >>> dis.dis(foo) 2 0 SETUP_LOOP 3 (to 6) 3 >> 3 JUMP_ABSOLUTE 3 >> 6 LOAD_CONST 0 (None) 9 RETURN_VALUE >>> dis.dis(bar) 2 0 SETUP_LOOP 3 (to 6) 3 >> 3 JUMP_ABSOLUTE 3 >> 6 LOAD_CONST 0 (None) 9 RETURN_VALUE
kqueue 15y ago
~$python -mtimeit "'a' + 'b' + 'c' + 'd'" 10000000 loops, best of 3: 0.026 usec per loop ~$python -mtimeit "''.join(('a','b','c','d'))" 10000000 loops, best of 3: 0.197 usec per loop
- Ysx 15y ago
  Interesting! ''.join() has the advantage on longer strings though: $ python -mtimeit "'aaaaaaaaaaaaaaa' + 'bbbbbbbbbbbbbbb' + 'ccccccccccccccc' + 'ddddddddddddddd'" 1000000 loops, best of 3: 0.224 usec per loop $ python -mtimeit "''.join(('aaaaaaaaaaaaaaa','bbbbbbbbbbbbbbb','ccccccccccccccc','ddddddddddddddd'))" 10000000 loops, best of 3: 0.201 usec per loop
  - kqueue 15y ago
    That's definitely interesting.
- imurray 15y ago
  Always worth checking. Although the relative merits change when doing many joins to build up a single long string. This recommendation is to go with .join() by default: In performance sensitive parts of the library, the ''.join() form should be used instead. This will ensure that concatenation occurs in linear time across various implementations. — http://www.python.org/dev/peps/pep-0008/ http://www.python.org/dev/peps/pep-0008/ despite the fact it might not be better in CPython.
- deleted 15y ago
  [deleted]
- steve-howard 15y ago
  It's not terribly surprising. Things that scale faster tend to have higher setup costs (cf sorting algorithms; insertion sort is the fastest for relatively small n).
- thristian 15y ago
  I believe the Python runtime has special-case handling for concatenating string-literals, and for concatenating string whose reference count is exactly 1. String concatenation isn't wholly defanged, though.
Estragon 15y ago
> [Despite the existence of "enumerate", "xrange(maxint)"] can still > be useful when you want to include an index along with several > other lists, however, e.g. zip(list_1, list_2, indices) I would just use for idx, (elt1, elt2) in enumerate(zip(list_1, list_2)) The following is not necessarily good advice, and he should take his own earlier advice and validate his claims with some profiling. > [map is ] much faster, since the loop takes place entirely in > the C API and never has to bind loop variables to Python > objects. > If you find yourself making the same list comprehension > repeatedly, make utility functions and use map and/or filter... Note that the following "map" snippet is considerably slower than the corresponding list-comprehension snippet. Function calls are expensive in python. Also, his claim that you save time with a "map" by avoiding the loop variable is obviously bogus: you still have to bind the variables in the signature of the function you're mapping, unless the signature is empty. met% python -m timeit "[x**3 for x in xrange(10000)]" 1000 loops, best of 3: 1.27 msec per loop met% python -m timeit "map(lambda x: x**3, xrange(10000))" 1000 loops, best of 3: 1.88 msec per loop
- steve-howard 15y ago
  The rule of thumb I've read is that map is appropriate and (generally) faster when you do not need to create a lambda.
  - Estragon 15y ago
    That's not strictly correct. It really comes down to whether you're calling a native python function or a C extension. If it's a native python function call, it's going to be relatively slow.
- VBprogrammer 15y ago
  The map and filter functions were almost depreciated in Python 3. http://www.artima.com/weblogs/viewpost.jsp?thread=98196 http://www.artima.com/weblogs/viewpost.jsp?thread=98196
  - louisdefunes 15y ago
mixmastamyk 15y ago
I read many years ago that '%s%s' % (a,b) was faster than a + b, the reason given that it was done in C. But after reading this thread and trying myself, it seems to be false also. On Py 2.6: python -m timeit " '%s%s%s' % ('aaaaaaaaaaaaaaaaaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbbbbbbbbbb', 'ccccccccccccccccccccccccccccccc') " 1000000 loops, best of 3: 0.201 usec per loop python -m timeit " 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaa' + 'bbbbbbbbbbbbbbbbbbbbbbbb' + 'ccccccccccccccccccccccccccccccc' " 10000000 loops, best of 3: 0.136 usec per loop Perhaps it changed in new versions of Python, but either way, I guess I'll be using the % form less than I used to.
- wahnfrieden 15y ago
  Use whatever is clearest to read, since the benchmarks vary upon circumstance, VM, and VM version.
1amzave 15y ago
> Calling it from the empty string concatenates the pieces with no separator, which is a Python quirk and rather surprising at first. This is stated twice, but I can't make any sense of it. How is this even remotely surprising? What else would anyone possibly expect joining with the empty string as the separator to do?
- underwater 15y ago
  I think he's referring to backwards nature of string joining in general.
- kragen 15y ago
  The surprising thing is that .join is a method of the separator, not of the list of pieces.
  - dzderic 15y ago
    I'm pretty sure it's a string method so it can take any iterator as an argument.
    - kragen 15y ago
      I think that was the deciding argument, yes.
skimbrel 15y ago
Use function factories to create utility functions. Often, especially if you're using map and filter a lot, you need utility functions that convert other functions or methods to taking a single parameter. In particular, you often want to bind some data to the function once, and then apply it repeatedly to different objects. In the above example, we needed a function that multiplied a particular field of an object by 3, but what we really want is a factory that's able to return for any field name and amount a multiplier function in that family: def multiply_by_field(fieldname, multiplier): """Returns function that multiplies field "fieldname" by multiplier.""" def multiplier(x): return getattr(x, fieldname) * multiplier return multiplier triple = multiply_by_field('Count', 3) quadruple = multiply_by_field('Count', 4) halve_sum = multiply_by_field('Sum', 0.5) Other languages (most prominently Haskell, though you can convince most Lisps to do it through the clever use of macros) have built-in support for doing this with whatever function you want, and it's called partial function application. It's a rather useful technique and it saddens me that it's not supported as such in more languages claiming to support functional programming.
- Lycanthrope 15y ago
  There is functools.partial: from functools import partial def add(x,y): return x+y add3 = partial(add, 3) add3(2) # returns 5
  - skimbrel 15y ago
    Ah! I didn't know about that. It'd be nice to have the syntactic sugar à la Haskell, but oh well. Close enough.
- herdrick 15y ago
  You don't need a macro to do that in a lisp. Typically you just use a built in 'partial' function, which just makes a closure that will call the original function. (It's simple, so you can just make your own if you're using a lisp without it built in.)
j_baker 15y ago
Use if not x instead of if x == 0 or if x == "" or if x == None or if x == False; likewise, if x instead of if x != 0, if x != None, etc. Be careful with this one. "if not x" isn't necessarily the same as "if x == None". It's easy to forget that "if not x" will be true for values other than None. Also, use "if x is None" rather than "if x == None". :-)
astrofinch 15y ago
What's the rationale for preferring map/filter over list comprehensions?
tocomment 15y ago
I wish every language listed it's idioms somewhere.
fijal 15y ago
For what is worth, most "performance" idioms are anti-idioms on PyPy. Especially those: sum = 0 for d in data: sum += d product = 1 for d in data: product *= d Are much faster than reduce/map equivalents. The zip/dict example at the end is even more confusing. I'm convinced PyPy would be the fastest on the simplest-possible code (the first one, marked as "bad")

169 points by preek 15y ago

Ysx 15y ago

Good stuff, though there's a few outdated idioms: Lists, or any iterable, can be reverse-sorted with reversed(sorted(my_list)). That'll give you an iterator, though you can call list() on the result if you need it. "while True" should be used in-place of "while 1". Reads better. In Python 2, xrange() is preferred over range() when looping - it won't create an in-memory list of integers, and behaves mostly the same as range but for a few edge cases. Python 3 renamed xrange() to range(), and removed the original range() function.

shazow 15y ago
> reversed(sorted(my_list)) I thought sorted(my_list, reverse=True) would be slightly faster, but it seems not (by a tiny amount). Weird. > "while True" should be used in-place of "while 1". Reads better. If you disassemble these statements, you'll see that "while 1" creates fewer instructions: def foo(): while True: pass def bar(): while 1: pass import dis dis.dis(foo) # 0 SETUP_LOOP 10 (to 13) # >> 3 LOAD_GLOBAL 0 (True) # 6 POP_JUMP_IF_FALSE 12 # # 9 JUMP_ABSOLUTE 3 # >> 12 POP_BLOCK # >> 13 LOAD_CONST 0 (None) # 16 RETURN_VALUE dis.dis(bar) # 0 SETUP_LOOP 3 (to 6) # # >> 3 JUMP_ABSOLUTE 3 # >> 6 LOAD_CONST 0 (None) # 9 RETURN_VALUE Edit: This is in Python 2.7.1. d0mine pointed out that the discrepancy is no longer the case in Python 3. Good to know. :)
- d0mine 15y ago
  There is no difference on Python 3: >>> dis.dis(foo) 2 0 SETUP_LOOP 3 (to 6) 3 >> 3 JUMP_ABSOLUTE 3 >> 6 LOAD_CONST 0 (None) 9 RETURN_VALUE >>> dis.dis(bar) 2 0 SETUP_LOOP 3 (to 6) 3 >> 3 JUMP_ABSOLUTE 3 >> 6 LOAD_CONST 0 (None) 9 RETURN_VALUE

kqueue 15y ago

~$python -mtimeit "'a' + 'b' + 'c' + 'd'" 10000000 loops, best of 3: 0.026 usec per loop ~$python -mtimeit "''.join(('a','b','c','d'))" 10000000 loops, best of 3: 0.197 usec per loop

Ysx 15y ago
Interesting! ''.join() has the advantage on longer strings though: $ python -mtimeit "'aaaaaaaaaaaaaaa' + 'bbbbbbbbbbbbbbb' + 'ccccccccccccccc' + 'ddddddddddddddd'" 1000000 loops, best of 3: 0.224 usec per loop $ python -mtimeit "''.join(('aaaaaaaaaaaaaaa','bbbbbbbbbbbbbbb','ccccccccccccccc','ddddddddddddddd'))" 10000000 loops, best of 3: 0.201 usec per loop
- kqueue 15y ago
  That's definitely interesting.
imurray 15y ago
Always worth checking. Although the relative merits change when doing many joins to build up a single long string. This recommendation is to go with .join() by default: In performance sensitive parts of the library, the ''.join() form should be used instead. This will ensure that concatenation occurs in linear time across various implementations. — http://www.python.org/dev/peps/pep-0008/ http://www.python.org/dev/peps/pep-0008/ despite the fact it might not be better in CPython.
deleted 15y ago
[deleted]
steve-howard 15y ago
It's not terribly surprising. Things that scale faster tend to have higher setup costs (cf sorting algorithms; insertion sort is the fastest for relatively small n).
thristian 15y ago
I believe the Python runtime has special-case handling for concatenating string-literals, and for concatenating string whose reference count is exactly 1. String concatenation isn't wholly defanged, though.

Estragon 15y ago

> [Despite the existence of "enumerate", "xrange(maxint)"] can still > be useful when you want to include an index along with several > other lists, however, e.g. zip(list_1, list_2, indices) I would just use for idx, (elt1, elt2) in enumerate(zip(list_1, list_2)) The following is not necessarily good advice, and he should take his own earlier advice and validate his claims with some profiling. > [map is ] much faster, since the loop takes place entirely in > the C API and never has to bind loop variables to Python > objects. > If you find yourself making the same list comprehension > repeatedly, make utility functions and use map and/or filter... Note that the following "map" snippet is considerably slower than the corresponding list-comprehension snippet. Function calls are expensive in python. Also, his claim that you save time with a "map" by avoiding the loop variable is obviously bogus: you still have to bind the variables in the signature of the function you're mapping, unless the signature is empty. met% python -m timeit "[x**3 for x in xrange(10000)]" 1000 loops, best of 3: 1.27 msec per loop met% python -m timeit "map(lambda x: x**3, xrange(10000))" 1000 loops, best of 3: 1.88 msec per loop

steve-howard 15y ago
The rule of thumb I've read is that map is appropriate and (generally) faster when you do not need to create a lambda.
- Estragon 15y ago
  That's not strictly correct. It really comes down to whether you're calling a native python function or a C extension. If it's a native python function call, it's going to be relatively slow.
VBprogrammer 15y ago
The map and filter functions were almost depreciated in Python 3. http://www.artima.com/weblogs/viewpost.jsp?thread=98196 http://www.artima.com/weblogs/viewpost.jsp?thread=98196
- louisdefunes 15y ago

mixmastamyk 15y ago

I read many years ago that '%s%s' % (a,b) was faster than a + b, the reason given that it was done in C. But after reading this thread and trying myself, it seems to be false also. On Py 2.6: python -m timeit " '%s%s%s' % ('aaaaaaaaaaaaaaaaaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbbbbbbbbbb', 'ccccccccccccccccccccccccccccccc') " 1000000 loops, best of 3: 0.201 usec per loop python -m timeit " 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaa' + 'bbbbbbbbbbbbbbbbbbbbbbbb' + 'ccccccccccccccccccccccccccccccc' " 10000000 loops, best of 3: 0.136 usec per loop Perhaps it changed in new versions of Python, but either way, I guess I'll be using the % form less than I used to.

wahnfrieden 15y ago
Use whatever is clearest to read, since the benchmarks vary upon circumstance, VM, and VM version.

1amzave 15y ago

> Calling it from the empty string concatenates the pieces with no separator, which is a Python quirk and rather surprising at first. This is stated twice, but I can't make any sense of it. How is this even remotely surprising? What else would anyone possibly expect joining with the empty string as the separator to do?

underwater 15y ago
I think he's referring to backwards nature of string joining in general.
kragen 15y ago
The surprising thing is that .join is a method of the separator, not of the list of pieces.
- dzderic 15y ago
  I'm pretty sure it's a string method so it can take any iterator as an argument.
  - kragen 15y ago
    I think that was the deciding argument, yes.

skimbrel 15y ago

Use function factories to create utility functions. Often, especially if you're using map and filter a lot, you need utility functions that convert other functions or methods to taking a single parameter. In particular, you often want to bind some data to the function once, and then apply it repeatedly to different objects. In the above example, we needed a function that multiplied a particular field of an object by 3, but what we really want is a factory that's able to return for any field name and amount a multiplier function in that family: def multiply_by_field(fieldname, multiplier): """Returns function that multiplies field "fieldname" by multiplier.""" def multiplier(x): return getattr(x, fieldname) * multiplier return multiplier triple = multiply_by_field('Count', 3) quadruple = multiply_by_field('Count', 4) halve_sum = multiply_by_field('Sum', 0.5) Other languages (most prominently Haskell, though you can convince most Lisps to do it through the clever use of macros) have built-in support for doing this with whatever function you want, and it's called partial function application. It's a rather useful technique and it saddens me that it's not supported as such in more languages claiming to support functional programming.

Lycanthrope 15y ago
There is functools.partial: from functools import partial def add(x,y): return x+y add3 = partial(add, 3) add3(2) # returns 5
- skimbrel 15y ago
  Ah! I didn't know about that. It'd be nice to have the syntactic sugar à la Haskell, but oh well. Close enough.
herdrick 15y ago
You don't need a macro to do that in a lisp. Typically you just use a built in 'partial' function, which just makes a closure that will call the original function. (It's simple, so you can just make your own if you're using a lisp without it built in.)

j_baker 15y ago

Use if not x instead of if x == 0 or if x == "" or if x == None or if x == False; likewise, if x instead of if x != 0, if x != None, etc. Be careful with this one. "if not x" isn't necessarily the same as "if x == None". It's easy to forget that "if not x" will be true for values other than None. Also, use "if x is None" rather than "if x == None". :-)

astrofinch 15y ago

What's the rationale for preferring map/filter over list comprehensions?

tocomment 15y ago

I wish every language listed it's idioms somewhere.

fijal 15y ago

For what is worth, most "performance" idioms are anti-idioms on PyPy. Especially those: sum = 0 for d in data: sum += d product = 1 for d in data: product *= d Are much faster than reduce/map equivalents. The zip/dict example at the end is even more confusing. I'm convinced PyPy would be the fastest on the simplest-possible code (the first one, marked as "bad")