4 ms·

Python performance is a fool's errand

7 points by ntoshev 17y ago

noss 17y ago
I think this rule about clarity-of-code > micro-performance can be generalized to programming as a whole.
- ntoshev 17y ago
  In the case of a full text search engine, you really need that performance. I'm not sure clarity and performance of code are inherently conflicting requirements; in any case the author wouldn't have this gap in performance between the two versions if he used C++.
  - tdoggette 17y ago
    Which is why you don't use Python where performance is one of the biggest priorities. I'm a big fan of Python, but it's not for everything.
Goladus 17y ago
Code optimized for performance is usually optimized based on environmental constraints. The more you optimize, the more the code will look like a description of instructions to manipulate that environment, rather than a succinct description of your solution. There is an inherent conflict. C++ doesn't solve this problem, it's just one of the possible compromises. With Python and similar high-level languages, there are a few layers of "environment" that you have to tailor your instructions for. You have to understand how Python manipulates the machine resources and then you have to understand how to manipulate Python to manipulate those resources. The result is code optimized for python on a particular machine, which might not look anything like what you'd expect being familiar with one or the other but not both.
- stcredzero 17y ago
  The need for optimization is just another example of "leaky abstractions." Ideally, you'd just let the JIT take care of it for you, but it's not able to do everything. There are Smalltalk parsers that use lots of if-then equivalents, because these are faster because of bytecode level optimizations, even though using Dictionary lookup would be clearer and shorter. That's an example of a leaky abstraction in the compiler->bytecode->JIT implementation of the language itself. What's worse, is that the structure of the code -- why it's written in that way -- is highly cryptic. You can't write a bunch of if-then in a way that reveals that they are faster. The Mercurial project's implementation is optimized to ensure high locality of reference for data on disk. You can't make stuff like that intention revealing. You must resort to a comment. I think this reveals a weakness in languages in general. The issue of performance is mostly orthogonal to most of the business problems you're solving. The fact that optimization issues get embroiled in code -- this is something we should try to eliminate. "Correct then fast" is just a compromise. You minimize the amount of optimization done to preserve as much readability as possible. We should have languages that allow us to optimize without mucking up code. (Is it possible with Lisp Macros?)
seunosewa 17y ago
There is a perfectly adequate solution to the specific problem that inspired the above post. It's in the second comment. The author simply didn't know enough about Python at that point.

7 points by ntoshev 17y ago

noss 17y ago

I think this rule about clarity-of-code > micro-performance can be generalized to programming as a whole.

ntoshev 17y ago
In the case of a full text search engine, you really need that performance. I'm not sure clarity and performance of code are inherently conflicting requirements; in any case the author wouldn't have this gap in performance between the two versions if he used C++.
- tdoggette 17y ago
  Which is why you don't use Python where performance is one of the biggest priorities. I'm a big fan of Python, but it's not for everything.

Goladus 17y ago

Code optimized for performance is usually optimized based on environmental constraints. The more you optimize, the more the code will look like a description of instructions to manipulate that environment, rather than a succinct description of your solution. There is an inherent conflict. C++ doesn't solve this problem, it's just one of the possible compromises. With Python and similar high-level languages, there are a few layers of "environment" that you have to tailor your instructions for. You have to understand how Python manipulates the machine resources and then you have to understand how to manipulate Python to manipulate those resources. The result is code optimized for python on a particular machine, which might not look anything like what you'd expect being familiar with one or the other but not both.

stcredzero 17y ago
The need for optimization is just another example of "leaky abstractions." Ideally, you'd just let the JIT take care of it for you, but it's not able to do everything. There are Smalltalk parsers that use lots of if-then equivalents, because these are faster because of bytecode level optimizations, even though using Dictionary lookup would be clearer and shorter. That's an example of a leaky abstraction in the compiler->bytecode->JIT implementation of the language itself. What's worse, is that the structure of the code -- why it's written in that way -- is highly cryptic. You can't write a bunch of if-then in a way that reveals that they are faster. The Mercurial project's implementation is optimized to ensure high locality of reference for data on disk. You can't make stuff like that intention revealing. You must resort to a comment. I think this reveals a weakness in languages in general. The issue of performance is mostly orthogonal to most of the business problems you're solving. The fact that optimization issues get embroiled in code -- this is something we should try to eliminate. "Correct then fast" is just a compromise. You minimize the amount of optimization done to preserve as much readability as possible. We should have languages that allow us to optimize without mucking up code. (Is it possible with Lisp Macros?)

seunosewa 17y ago

There is a perfectly adequate solution to the specific problem that inspired the above post. It's in the second comment. The author simply didn't know enough about Python at that point.