4 ms·
Serious question: if you have some code that really has to be fast, is it viable to keep it in Python, or should you ultimately end up rewriting it in a compile
by clickok 11y ago
Serious question: if you have some code that really has to be fast, is it viable to keep it in Python, or should you ultimately end up rewriting it in a compiled language?
For example, I am writing code that implements networks that evolve over time for AI research. Prototyping it in Python makes it easy to test things out, but I expect that I will have to rewrite it in C++ or maybe something more fun, like Haskell[1].
1. Mostly for the sheer joy of trolling my colleagues with a learning agent monad.
- p1esk 11y agoSame question to you: if you have some code that really has to be fast, is it viable to keep it in C++, or should you ultimately end up rewriting it for GPU?
- wallstop 11y agoIs this really a suitable counter-question? GPUs lend themselvse towards specific kinds of programming problems + added latency of GPU communication. On the contrary, the cost between switching from any on-CPU language is programmer time that may result in significant runtime advantages.
- vetinari 11y agoAlso disadvantages: increased programmer time for implementing new features. Risks: you might optimize the wrong part (i.e. you invest into rewrite, but it will not solve performance problems). That's why you must quantify advantages and disadvantages, including risks minimization and only then you'll see, whether given course of action is viable.
- jermy 11y agoYou should probably first consider seeing if there are critical bits of code that can be rewritten to use MMX/SSE instructions, since your data is in the right cache already, without needing to move it anywhere else.
- thezilch 11y agoYou might "just" migrate some slower parts to numpy or write a C target and interface with it over Python's CFFI.
- eikenberry 11y ago+1 numpy could probably handle this
- ma2rten 11y agoIt depends on your use case. In my experience the reason why you wrote something in python (implement features faster) remain valid reasons later on when you want to add functionality. By using pypy, cpython, rewriting small parts in C/C++ and/or using the libraries which are written in C (such as numpy) you can normally make the hotspots in your code fast enough, while keeping the advantages of python.
- adrianN 11y agoIn my experience it is worthwhile to first try and improve the performance in Python. It's easier to play around with different ideas in Python, and things like numpy quite often enable you to get "fast enough". Only if that is not enough should you consider writing a C extension.
- ryan_sb 11y agoPython makes it easy to do lots of things -- including very inefficient ones. Profiling your existing stuff and trying to optimize in pure Python often gets you pretty far.
- dec0dedab0de 11y agoI've never had to do this, but the common advice I hear is that the right algorithm in a high level language can be faster then the wrong algorithm in a low level language. So even if you do end up going the route of writing it in something lower level, it is likely worth optimizing your code at the higher level first where it is less expensive to try different things.