3 ms·

I doubt it. If you start writting a C++ process first, you will take a lot of time. Python allow you to have a good idea of how the thing works. And if, event

I doubt it. If you start writting a C++ process first, you will take a lot of time. Python allow you to have a good idea of how the thing works. And if, eventually, you find out that it's not fast enough (even with pypy, numpy, a few lines of cython, etc), you may rewrite it in C++. But the rewrite is going to be much, much faster to do, because you know now what's up, and can just translate it to C++ and focus on its gotchas.

thegeomaster 8y ago
But usually, I know how the thing works. I have data in this format, and I need data in this other format. The transformation may include cross-referencing among the data, processing strings in various ways, computing sums and counts, calling external libraries, etc. These are simple to code and well-defined tasks. In my experience, they rarely benefit from prototyping. My recent example is processing around 10m chess games to get statistics on all positions that occurred inside them (above an occurrence threshold). This required parsing the games in chess notation, using a chess library to simulate the moves to get positions, and counting how often each position occurred, in which matches, etc. My first try was with Python. After I realized it's unbearably slow, I tried using PyPy, running multiple processes for each core, etc., and in the end my approximation was that the job would finish in a couple of hours. I tried more optimizations and nothing helped. And there's no number crunching to use numpy for. Then I wrote the same script in Rust, and it ran in a couple of minutes, finishing well before the original Python script would have finished, had I left it to run. I arguably didn't save time by using Python here.
- sametmax 8y ago
  If you knew how it works, and it was fast to write it in rust, why didn't write it in rust in the first place ? Because it was easy in Python. It was easy to write the same script in Rust, then, because you already got the Python version working.
pjmlp 8y ago
Or using a language that supports AOT/JIT from the get go, which most likely will avoid the need of any rewrite.

thegeomaster 8y ago

But usually, I know how the thing works. I have data in this format, and I need data in this other format. The transformation may include cross-referencing among the data, processing strings in various ways, computing sums and counts, calling external libraries, etc. These are simple to code and well-defined tasks. In my experience, they rarely benefit from prototyping. My recent example is processing around 10m chess games to get statistics on all positions that occurred inside them (above an occurrence threshold). This required parsing the games in chess notation, using a chess library to simulate the moves to get positions, and counting how often each position occurred, in which matches, etc. My first try was with Python. After I realized it's unbearably slow, I tried using PyPy, running multiple processes for each core, etc., and in the end my approximation was that the job would finish in a couple of hours. I tried more optimizations and nothing helped. And there's no number crunching to use numpy for. Then I wrote the same script in Rust, and it ran in a couple of minutes, finishing well before the original Python script would have finished, had I left it to run. I arguably didn't save time by using Python here.

sametmax 8y ago
If you knew how it works, and it was fast to write it in rust, why didn't write it in rust in the first place ? Because it was easy in Python. It was easy to write the same script in Rust, then, because you already got the Python version working.

pjmlp 8y ago

Or using a language that supports AOT/JIT from the get go, which most likely will avoid the need of any rewrite.