4 ms·

Python's slowness has been a very real problem for me. It's a great language to write data-wrangling scripts in, but when there's a little bit more data, it's t

Python's slowness has been a very real problem for me. It's a great language to write data-wrangling scripts in, but when there's a little bit more data, it's terribly slow. And then you have to rewrite your script in a faster language, like C++ or Rust. I've observed speedups on the order of 20x this way. Which means that I could've saved myself a lot of time by writing them in another language at the start.

carlmr 8y ago
>And then you have to rewrite your script in a faster language, like C++ or Rust. I've observed speedups on the order of 20x this way. My assumption of possible speedups (from past experience) is roughly. Python -5x-> Perl -2x-> C#/F# -5x-> C++ -2x-> Rust Rust is usually a lot faster than C++, because in my data wrangling I often need a lot of string operations that are much easier to do with zero-allocation in Rust than in C++. While you can write the same in C++ it becomes very hard to manage without the borrow checker. I don't like Perl, so I basically never use that, and I find it's easier to write something that works in Rust than in C++. So usually I'll pick F# or Rust, depending on how fast it needs to be and how easy it needs to be to write it. F# is super nice for writing and Ionide code completion is just miles ahead of Rust code completion at this point. Being forgetful I barely need to look anything up in F#, but in Rust I always need to look at the docs. This will probably improve in the future (I hope). If Rust code completion becomes amazing I think I won't even start in Python anymore. (Pandas is the only thing that pulls me back into Python usually).
- nerpderp82 8y ago
  An F# like language hosted within Rust would be a dream. With a GC and stack allocation, optional lifetime annotations when you want to remove the GC overhead.
  - carlmr 8y ago
    I think with the newer versions of F# you can do some manual memory management if needed.
sametmax 8y ago
I doubt it. If you start writting a C++ process first, you will take a lot of time. Python allow you to have a good idea of how the thing works. And if, eventually, you find out that it's not fast enough (even with pypy, numpy, a few lines of cython, etc), you may rewrite it in C++. But the rewrite is going to be much, much faster to do, because you know now what's up, and can just translate it to C++ and focus on its gotchas.
- thegeomaster 8y ago
  But usually, I know how the thing works. I have data in this format, and I need data in this other format. The transformation may include cross-referencing among the data, processing strings in various ways, computing sums and counts, calling external libraries, etc. These are simple to code and well-defined tasks. In my experience, they rarely benefit from prototyping. My recent example is processing around 10m chess games to get statistics on all positions that occurred inside them (above an occurrence threshold). This required parsing the games in chess notation, using a chess library to simulate the moves to get positions, and counting how often each position occurred, in which matches, etc. My first try was with Python. After I realized it's unbearably slow, I tried using PyPy, running multiple processes for each core, etc., and in the end my approximation was that the job would finish in a couple of hours. I tried more optimizations and nothing helped. And there's no number crunching to use numpy for. Then I wrote the same script in Rust, and it ran in a couple of minutes, finishing well before the original Python script would have finished, had I left it to run. I arguably didn't save time by using Python here.
  - sametmax 8y ago
    If you knew how it works, and it was fast to write it in rust, why didn't write it in rust in the first place ? Because it was easy in Python. It was easy to write the same script in Rust, then, because you already got the Python version working.
- pjmlp 8y ago
  Or using a language that supports AOT/JIT from the get go, which most likely will avoid the need of any rewrite.
mr_overalls 8y ago
And of course this write-it-twice dilemma is one major impetus behind the invention of Julia, which seeks high performance in a language more user-friendly than C/C++.

carlmr 8y ago

>And then you have to rewrite your script in a faster language, like C++ or Rust. I've observed speedups on the order of 20x this way. My assumption of possible speedups (from past experience) is roughly. Python -5x-> Perl -2x-> C#/F# -5x-> C++ -2x-> Rust Rust is usually a lot faster than C++, because in my data wrangling I often need a lot of string operations that are much easier to do with zero-allocation in Rust than in C++. While you can write the same in C++ it becomes very hard to manage without the borrow checker. I don't like Perl, so I basically never use that, and I find it's easier to write something that works in Rust than in C++. So usually I'll pick F# or Rust, depending on how fast it needs to be and how easy it needs to be to write it. F# is super nice for writing and Ionide code completion is just miles ahead of Rust code completion at this point. Being forgetful I barely need to look anything up in F#, but in Rust I always need to look at the docs. This will probably improve in the future (I hope). If Rust code completion becomes amazing I think I won't even start in Python anymore. (Pandas is the only thing that pulls me back into Python usually).

nerpderp82 8y ago
An F# like language hosted within Rust would be a dream. With a GC and stack allocation, optional lifetime annotations when you want to remove the GC overhead.
- carlmr 8y ago
  I think with the newer versions of F# you can do some manual memory management if needed.

sametmax 8y ago

I doubt it. If you start writting a C++ process first, you will take a lot of time. Python allow you to have a good idea of how the thing works. And if, eventually, you find out that it's not fast enough (even with pypy, numpy, a few lines of cython, etc), you may rewrite it in C++. But the rewrite is going to be much, much faster to do, because you know now what's up, and can just translate it to C++ and focus on its gotchas.

thegeomaster 8y ago
But usually, I know how the thing works. I have data in this format, and I need data in this other format. The transformation may include cross-referencing among the data, processing strings in various ways, computing sums and counts, calling external libraries, etc. These are simple to code and well-defined tasks. In my experience, they rarely benefit from prototyping. My recent example is processing around 10m chess games to get statistics on all positions that occurred inside them (above an occurrence threshold). This required parsing the games in chess notation, using a chess library to simulate the moves to get positions, and counting how often each position occurred, in which matches, etc. My first try was with Python. After I realized it's unbearably slow, I tried using PyPy, running multiple processes for each core, etc., and in the end my approximation was that the job would finish in a couple of hours. I tried more optimizations and nothing helped. And there's no number crunching to use numpy for. Then I wrote the same script in Rust, and it ran in a couple of minutes, finishing well before the original Python script would have finished, had I left it to run. I arguably didn't save time by using Python here.
- sametmax 8y ago
  If you knew how it works, and it was fast to write it in rust, why didn't write it in rust in the first place ? Because it was easy in Python. It was easy to write the same script in Rust, then, because you already got the Python version working.
pjmlp 8y ago
Or using a language that supports AOT/JIT from the get go, which most likely will avoid the need of any rewrite.

mr_overalls 8y ago

And of course this write-it-twice dilemma is one major impetus behind the invention of Julia, which seeks high performance in a language more user-friendly than C/C++.