4 ms·

Well, PyPy does not actually help much with raw number crunching -- a scientist will use numpy for native matrix math, or numba to JIT a tight loop that can't b

by rjeli 10y ago

Well, PyPy does not actually help much with raw number crunching -- a scientist will use numpy for native matrix math, or numba to JIT a tight loop that can't be writted in numpy easily. PyPy really helps in the in-between cases: a few weeks ago I had to write a script to pull a few million rows from a database, process them nontrivially, and dump them to a file. What do I use here? CPython and Ruby will take days. Perl will probably take days, unless I am just doing text processing which perl is optimized for. I can write C or C++, and have the headache of debugging memory errors in a script that will only run once. Or, I can write PyPy and have immensely convenient syntax that will run reasonably quickly (Luajit also occupies this space for me).

brianwawok 10y ago
Well if this task is trivial to parallelize.. I would spin up 100 GCE machines for 1 penny each per hour after today's price drop (combined total of $1 per hour for 100 CPUs of action).. and pull those few million files and dump to a cloud drive in 1 hour flat. It's a tricky sweet spot PyPy has. Certain tasks that need CPU power but not too much CPU power and you want to spend time to make it run in parallel but not too much time to make it run across many machines. (By no means am I a PyPy or CPython expert. Just toyed with them and want to understand the bounds...)
- zardeh 10y ago
  But now you have to manage that deployment which is also nontrivial.
  - brianwawok 10y ago
    Depends on your setup I guess? I have a a GCE cluster setup where I can: 1) Point a metadata variable at my git repo 2) Spin up 100 GCEs 3) Each will pip install requirements.txt from the git checkout 4) Each will read a metadata to get the command and run it. (Could hardcode run.py or something). Realize not everyone has this sitting around, but you can do it in an afternoon. Then gives you instant Python supercomputer for anything you want across many projects.
    - zardeh 10y ago
      The setup for that sounds significantly nontrivial, although interesting.

by rjeli 10y ago

brianwawok 10y ago

Well if this task is trivial to parallelize.. I would spin up 100 GCE machines for 1 penny each per hour after today's price drop (combined total of $1 per hour for 100 CPUs of action).. and pull those few million files and dump to a cloud drive in 1 hour flat. It's a tricky sweet spot PyPy has. Certain tasks that need CPU power but not too much CPU power and you want to spend time to make it run in parallel but not too much time to make it run across many machines. (By no means am I a PyPy or CPython expert. Just toyed with them and want to understand the bounds...)

zardeh 10y ago
But now you have to manage that deployment which is also nontrivial.
- brianwawok 10y ago
  Depends on your setup I guess? I have a a GCE cluster setup where I can: 1) Point a metadata variable at my git repo 2) Spin up 100 GCEs 3) Each will pip install requirements.txt from the git checkout 4) Each will read a metadata to get the command and run it. (Could hardcode run.py or something). Realize not everyone has this sitting around, but you can do it in an afternoon. Then gives you instant Python supercomputer for anything you want across many projects.
  - zardeh 10y ago
    The setup for that sounds significantly nontrivial, although interesting.