4 ms·

Efficiency of list comprehensions

39 points by llambda 14y ago

jessaustin 14y ago
Interesting. It seems like he also might have wanted to try a generator expression for the "throw away the list" case. You can always wrap a generator expression with max() or any() or something similar to trigger the iteration. You'd at least be avoiding the "LIST_APPEND" bytecode. With the additional function call I'm not sure the generator expression would be better, but I must admit that in my own code I'm pretty anal about converting all list comps to gen exps that I possibly can.
- deleted 14y ago
  [deleted]
- talaketu 14y ago
  "any" short circuits - you probably are not throwing away the boolean result. I do agree with the author that using a comprehension for the side-effect is smelly.
  - adamtj 14y ago
    But "any" short-circuits: that smells the same as a list comprehension for side effects. The beauty of Python is that the one, obvious way to do something is usually the right way. When you try to force something like this, you either run into silly bugs like "any" short-circuiting, or you waste your limited brainpower trying to avoid them. You're usually better off looking for algorithmic or architectural optimizations for speed, or refactoring for readability.
    - talaketu 14y ago
      "any" applied to a generator expression with side-effects is smelly because of the generator expression with side-effects, not because of the short-circuit. "any" applied to a pure generator expression expresses the algorithmic optimization known as short circuiting. Once you are at the point of considering execution time to be a side effect, the purity of any non-trivial expression vanishes.
  - jessaustin 14y ago
    Sure, but it's less expensive to store and maintain a single boolean value than a potentially-huge list. That's sort of the point of any(), sum(), or any other builtin that takes an iterable. This reminds me of reduce functions in MapReduce: if they return variable-sized objects they'll blow up. As others suggest I'm probably all wet about savings vs. a simple loop because of instantiation etc. And I don't think I'd ever use expressions like these merely for their side effects. I do expect that at a certain problem size the cumulative costs of building the list in a list comp will swamp the generator instantiation costs of gen exps consumed by iterator-friendly functions.
- pavpanchekha 14y ago
  Generator expressions are horribly slow. I agree they are prettier, and I put them where they aid readability and don't hurt performance, but in general, generators involve creating (the equivalent of) a class with methods, and calling those methods. A generator is not only a function call, but it also restores state internal to the function, which is very expensive.
koenigdavidmj 14y ago
1. You should be profiling. If this is your bottleneck, then consider tricks like this, or write that loop in C. This is probably not your bottleneck. 2. This is very interpreter-dependent. Any JIT worth its salt (like, say, PyPy) would probably decide to inline the append code if we're in this loop long enough for it to matter.

39 points by llambda 14y ago

Interesting. It seems like he also might have wanted to try a generator expression for the "throw away the list" case. You can always wrap a generator expression with max() or any() or something similar to trigger the iteration. You'd at least be avoiding the "LIST_APPEND" bytecode. With the additional function call I'm not sure the generator expression would be better, but I must admit that in my own code I'm pretty anal about converting all list comps to gen exps that I possibly can.

deleted 14y ago
[deleted]
talaketu 14y ago
"any" short circuits - you probably are not throwing away the boolean result. I do agree with the author that using a comprehension for the side-effect is smelly.
- adamtj 14y ago
  But "any" short-circuits: that smells the same as a list comprehension for side effects. The beauty of Python is that the one, obvious way to do something is usually the right way. When you try to force something like this, you either run into silly bugs like "any" short-circuiting, or you waste your limited brainpower trying to avoid them. You're usually better off looking for algorithmic or architectural optimizations for speed, or refactoring for readability.
  - talaketu 14y ago
    "any" applied to a generator expression with side-effects is smelly because of the generator expression with side-effects, not because of the short-circuit. "any" applied to a pure generator expression expresses the algorithmic optimization known as short circuiting. Once you are at the point of considering execution time to be a side effect, the purity of any non-trivial expression vanishes.
- jessaustin 14y ago
  Sure, but it's less expensive to store and maintain a single boolean value than a potentially-huge list. That's sort of the point of any(), sum(), or any other builtin that takes an iterable. This reminds me of reduce functions in MapReduce: if they return variable-sized objects they'll blow up. As others suggest I'm probably all wet about savings vs. a simple loop because of instantiation etc. And I don't think I'd ever use expressions like these merely for their side effects. I do expect that at a certain problem size the cumulative costs of building the list in a list comp will swamp the generator instantiation costs of gen exps consumed by iterator-friendly functions.
pavpanchekha 14y ago
Generator expressions are horribly slow. I agree they are prettier, and I put them where they aid readability and don't hurt performance, but in general, generators involve creating (the equivalent of) a class with methods, and calling those methods. A generator is not only a function call, but it also restores state internal to the function, which is very expensive.

koenigdavidmj 14y ago

1. You should be profiling. If this is your bottleneck, then consider tricks like this, or write that loop in C. This is probably not your bottleneck. 2. This is very interpreter-dependent. Any JIT worth its salt (like, say, PyPy) would probably decide to inline the append code if we're in this loop long enough for it to matter.