4 ms·

First thing you should do is optimize data structures IMO. This is the advantage Python has over lower level languages - easy way to implement complicated thin

by lqdc13 11y ago

First thing you should do is optimize data structures IMO. This is the advantage Python has over lower level languages - easy way to implement complicated things. Kind of like Linus's quote: "Bad programmers worry about the code. Good programmers worry about data structures and their relationships."

sitkack 11y ago
I would say it is a close second thing. If I have a slow system, I will move to faster runtime before modifying any code. Going from CPython to PyPy, if possible will almost always gain you enough perf increase while you refactor the slow parts.
- fnord123 11y ago
  It depends on what you're doing. I think your strategy can often be high risk since you are making a change that affects every part of your program at once -changing the runtime on a mature Python project is not a trivial choice! Changing the relevant code to tidy up data structures, or moving a few function calls to Cython or C, however, will only touch the obvious places and so it has less impact on the program. Also, profiling pypy is less straightforward than profiling CPython code since the hotspot changes the runtime characteristics of the program. This means you need to run tests many times over to make sure the code warms up. This makes further optimizations slightly more difficult. It's not a problem for people with experience optimizing Python code, but for people who actually hope to learn something from OP's blog post, it might be a sticking point. In my experience in using Python for stats, script type work (as opposed to writing servers and daemons), pypy just isn't that useful. All the Python code is doing is gluing numpy and Cython code together and pypy isn't likely to be able to warm up in time to beat it - and it won't beat it since it's spending most of its time in C. Obviously, if pypy is an ideal choice for you, use it. But I don't think your experience should really be put forward as a general approach.
  - sitkack 11y ago
    I have found PyPy wins if the wall time is over about 1.5 seconds, making it my goto Python environment for 1GB+ ETLs. Everything you say is true, but under the systems I use/write I am able to test for correctness pretty quickly. The nice thing about switching from CPython to PyPy is that everything get faster. I have also found that using PyPy has removed lots of cases where I would want to drop down to native code. Changing platforms can make one's designs simpler and more robust. When it comes to structured storage, I'll start with sqlite, then when it starts to get slow I'll switch to PostgreSQL. It takes almost no work to port from one to the other. You really should give PyPy another shot. It supports more of numpy every day and the startup time is excellent. Maybe give jitpy a try if you are not likely to move off of CPython.

by lqdc13 11y ago

sitkack 11y ago

I would say it is a close second thing. If I have a slow system, I will move to faster runtime before modifying any code. Going from CPython to PyPy, if possible will almost always gain you enough perf increase while you refactor the slow parts.

fnord123 11y ago
It depends on what you're doing. I think your strategy can often be high risk since you are making a change that affects every part of your program at once -changing the runtime on a mature Python project is not a trivial choice! Changing the relevant code to tidy up data structures, or moving a few function calls to Cython or C, however, will only touch the obvious places and so it has less impact on the program. Also, profiling pypy is less straightforward than profiling CPython code since the hotspot changes the runtime characteristics of the program. This means you need to run tests many times over to make sure the code warms up. This makes further optimizations slightly more difficult. It's not a problem for people with experience optimizing Python code, but for people who actually hope to learn something from OP's blog post, it might be a sticking point. In my experience in using Python for stats, script type work (as opposed to writing servers and daemons), pypy just isn't that useful. All the Python code is doing is gluing numpy and Cython code together and pypy isn't likely to be able to warm up in time to beat it - and it won't beat it since it's spending most of its time in C. Obviously, if pypy is an ideal choice for you, use it. But I don't think your experience should really be put forward as a general approach.
- sitkack 11y ago
  I have found PyPy wins if the wall time is over about 1.5 seconds, making it my goto Python environment for 1GB+ ETLs. Everything you say is true, but under the systems I use/write I am able to test for correctness pretty quickly. The nice thing about switching from CPython to PyPy is that everything get faster. I have also found that using PyPy has removed lots of cases where I would want to drop down to native code. Changing platforms can make one's designs simpler and more robust. When it comes to structured storage, I'll start with sqlite, then when it starts to get slow I'll switch to PostgreSQL. It takes almost no work to port from one to the other. You really should give PyPy another shot. It supports more of numpy every day and the startup time is excellent. Maybe give jitpy a try if you are not likely to move off of CPython.