3 ms·

There are many programmers out there using Python as if it was C, and that leads to slower than necessary code. It takes some time getting used to and learning

There are many programmers out there using Python as if it was C, and that leads to slower than necessary code. It takes some time getting used to and learning the performant way to write loops. For example, if 'trkpts' was a list of (lat, lon) tuples/lists, he could've avoided the lookups (but it would also mean having a different structure). Another example, if memory isn't a concern he could write a list comprehension with all the distance values, and then get the index of the smallest. This, however, has the problem that, though the list comprhension surely runs faster than the for loop, it takes more memory and then looking for the lowest value can take all the time you saved (and maybe some more). Without prfiling in his use case, it's difficult to say what is "the best solution", but his problem comes mostly from coding Python as if it were C. Edit: yes, I ignored the fact tha he used numpy (a good solution, giving his 300x speedup with changing the structures), because sometimes your data isn't prone to "numpy array conversion" -- for example, if you aren't programming numerical code.

odonnellryan 11y ago
> the list comprhension surely runs faster than the for loop How much quicker? It isn't too significant, is it? I mean, we have two O(1) operations to worry about, I suppose: lookup and appending to the list.
- dr_zoidberg 11y ago
  It depends on what the for loop does, but main difference is that a for loop runs "interpreted", vs the list comprehensions that runs "compiled" (or in "C-space"). The list comprehension has some additional benefits, the pattern: squares = [] for x in xrange(100): squares.append(x*x) in which you have an append call on every iteration is a mess because it may lead to a lot of memory operations which are really unnecesary. The LC version of it is much shorter an runs faster, because the memory allocation is handled differently: squares = [x*x for x in xrange(100)] Append may be O(1) but in some cases it doesn't behave nicely. On large lists, you can hit spots in which it reallocates parts of the list in memory, leading to weird slowdowns. Timings from ipython on a Windows 8 64bit Python 2.7.9 machine: In [12]: def f1(): ....: sq = [] ....: for x in xrange(100): ....: sq.append(x*x) ....: return sq ....: In [13]: def f2(): ....: return [x*x for x in xrange(100)] ....: In [14]: %timeit f1() 10000 loops, best of 3: 20.7 µs per loop In [15]: %timeit f2() 100000 loops, best of 3: 11.9 µs per loop
  - odonnellryan 11y ago
    Very interesting! That's a pretty good speed-up! Makes me reconsider writing some "complex" LC as for-loops.
    - dr_zoidberg 11y ago
      Bear in mind that the LC will take all the memory needed at once, and that can bring some problems. You can also use a generator expresion (similar to a LC, but you use parenthesis instead of brackets) to avoid this problem, if you're expecting an iterable object and have no further need for the values after the computation is done: In [1]: (x*x for x in xrange(100)) Out[1]: <generator object <genexpr> at 0x0000000003558438> And the generator expression could be used inside a for loop in cases where de LC would be awkward or take more time (because you have to allocate memory for a billion numbers all at once, just to check those that are meet a certain requirement).
jfpuget 11y ago
You missed the point: I did not had to change data structure to get the speedup.
- dr_zoidberg 11y ago
  I aknowledged it, see the "Edit" at the end ;) In your case reaching to numpy was probably the best you could do, as it abstracts and solves many things. Using properly "vectorized" numpy is also a big plus for readability. I just wanted to mention a few things regarding the "pure python optimization" views which could help in cases where you're not writing numerical code that won't benefit from numpy.
discardorama 11y ago
> There are many programmers out there using Python as if it was C, and that leads to slower than necessary code. I think it's because a lot of the Python (and I figure, other languages too) learning resources treat the learners the same way, whether you do not know any programming or are a 20-year veteran of C/C++. I'd love to see resources which show the ideological way of doing things in a language. Take some of the most common tasks performed in other languages, and show how they're mapped properly to constructs in Python. In this particular instance: I'm guessing the author could have used pure Python alone to get a decent speedup, without reaching for NumPy.

odonnellryan 11y ago

> the list comprhension surely runs faster than the for loop How much quicker? It isn't too significant, is it? I mean, we have two O(1) operations to worry about, I suppose: lookup and appending to the list.

dr_zoidberg 11y ago
It depends on what the for loop does, but main difference is that a for loop runs "interpreted", vs the list comprehensions that runs "compiled" (or in "C-space"). The list comprehension has some additional benefits, the pattern: squares = [] for x in xrange(100): squares.append(x*x) in which you have an append call on every iteration is a mess because it may lead to a lot of memory operations which are really unnecesary. The LC version of it is much shorter an runs faster, because the memory allocation is handled differently: squares = [x*x for x in xrange(100)] Append may be O(1) but in some cases it doesn't behave nicely. On large lists, you can hit spots in which it reallocates parts of the list in memory, leading to weird slowdowns. Timings from ipython on a Windows 8 64bit Python 2.7.9 machine: In [12]: def f1(): ....: sq = [] ....: for x in xrange(100): ....: sq.append(x*x) ....: return sq ....: In [13]: def f2(): ....: return [x*x for x in xrange(100)] ....: In [14]: %timeit f1() 10000 loops, best of 3: 20.7 µs per loop In [15]: %timeit f2() 100000 loops, best of 3: 11.9 µs per loop
- odonnellryan 11y ago
  Very interesting! That's a pretty good speed-up! Makes me reconsider writing some "complex" LC as for-loops.
  - dr_zoidberg 11y ago
    Bear in mind that the LC will take all the memory needed at once, and that can bring some problems. You can also use a generator expresion (similar to a LC, but you use parenthesis instead of brackets) to avoid this problem, if you're expecting an iterable object and have no further need for the values after the computation is done: In [1]: (x*x for x in xrange(100)) Out[1]: <generator object <genexpr> at 0x0000000003558438> And the generator expression could be used inside a for loop in cases where de LC would be awkward or take more time (because you have to allocate memory for a billion numbers all at once, just to check those that are meet a certain requirement).

jfpuget 11y ago

You missed the point: I did not had to change data structure to get the speedup.

dr_zoidberg 11y ago
I aknowledged it, see the "Edit" at the end ;) In your case reaching to numpy was probably the best you could do, as it abstracts and solves many things. Using properly "vectorized" numpy is also a big plus for readability. I just wanted to mention a few things regarding the "pure python optimization" views which could help in cases where you're not writing numerical code that won't benefit from numpy.

discardorama 11y ago

> There are many programmers out there using Python as if it was C, and that leads to slower than necessary code. I think it's because a lot of the Python (and I figure, other languages too) learning resources treat the learners the same way, whether you do not know any programming or are a 20-year veteran of C/C++. I'd love to see resources which show the ideological way of doing things in a language. Take some of the most common tasks performed in other languages, and show how they're mapped properly to constructs in Python. In this particular instance: I'm guessing the author could have used pure Python alone to get a decent speedup, without reaching for NumPy.