3 ms·

PyPy is a very underrated project. I really pray that after the C-extensions improvements that the upcoming versions brings, a major wave of adoptions come.

by bratao 9y ago

chubot 9y ago
I've been programming in Python for 15 years and I've known about PyPy for at least 10, and still haven't really found a reason to use it. Part of it is because I don't choose what interpreter we use at work. At home, I just use CPython and it works fine for everything. But I recently started learning more about PyPy, and it's technically crazy and ahead of its time [1]. It started in 2003 and I think something else like it didn't appear for more than 10 years [2], and is still research while PyPy is used in production. So yeah it is hugely impressive, from a technical standpoint. But I see that there's not enough "carrot" to pay the cost of adoption, for most users. It's faster for some workloads, but predicting the speedup is nontrivial. And it also uses more memory. It's unfortunate that the Python/C API is so complex and the extension ecosystem is so fragmented with wrappers as a result (SWIG, ctypes, cffi, etc.). I think there is a network effect with CPython that is very hard to break. Everyone builds for CPython because it already has a large amount of useful C code in its ecosystem -- not to mention stuff like NumPy, Pandas, etc. which are a mix of C and Cython, Cython being a totally separate effort to write "faster Python". [1] https://llvm.org/pubs/2006-10-DLS-PyPy.html https://llvm.org/pubs/2006-10-DLS-PyPy.html [2] Graal and Truffle seems like the closest thing, i.e. a meta-tracing JIT where you get a "JIT for free": https://blog.plan99.net/graal-truffle-134d8f28fb69 https://blog.plan99.net/graal-truffle-134d8f28fb69 http://tratt.net/laurie/research/pubs/html/bolz_tratt__the_impact_of_metatracing_on_vm_design_and_implementation/ http://tratt.net/laurie/research/pubs/html/bolz_tratt__the_i...
- daveFNbuck 9y ago
  At work, I use PyPy to run a Luigi server for our data pipeline. It gave us a pretty dramatic decrease in latency and allowed us to scale our workload with a trivial change. It's unfortunate you can't choose your interpreter, but when you can it's pretty easy to just test both and see which one is faster. I also use it at home to solve Project Euler problems. For anything that takes more than half a second or so, it's usually a big win and can turn minutes into seconds.
  - Myrmornis 9y ago
    Do any of the Project Euler questions actually require lengthy computation, or is that more a sign that one should be looking for some mathematical insight that leads to a more efficient computation?
    - xapata 9y ago
      PyPy won't change your data structures and algorithms, so it's not finding mathematical insight.
      - Myrmornis 9y ago
        Yes that’s the point I was making. Maybe you intended your comment to be a sibling to mine.
        xapata 9y ago
        More likely I missed your point.
        daveFNbuck 9y ago
        It seems that you're saying that I should be finding mathematical insight instead of using PyPy, while xapata is saying that if my code runs fast in PyPy then I've already found enough mathematical insight.
    - daveFNbuck 9y ago
      Most of them don't require lengthy computation, but some of them did for me. I know at least some of the time that's because I didn't know enough math, but sometimes I had the same algorithm as the other people in the solution thread but with much slower runtimes. I think numpy gives a bigger speedup than PyPy for a lot of the problems.
  - gsam 9y ago
    > For anything that takes more than half a second or so, it's usually a big win and can turn minutes into seconds. As a counter to this, I purposefully try to use CPython in order to force me to find algorithmically optimal solutions. It's a useful way to learn.
ojii 9y ago
The main reason I'm not even considering PyPy is that it lags behind CPython by a couple of versions. I'm getting ready to move to 3.7 while PyPy is still on 3.5.

PyPy is a very underrated project. I really pray that after the C-extensions improvements that the upcoming versions brings, a major wave of adoptions come.

by bratao 9y ago

PyPy is a very underrated project. I really pray that after the C-extensions improvements that the upcoming versions brings, a major wave of adoptions come.

chubot 9y ago

I've been programming in Python for 15 years and I've known about PyPy for at least 10, and still haven't really found a reason to use it. Part of it is because I don't choose what interpreter we use at work. At home, I just use CPython and it works fine for everything. But I recently started learning more about PyPy, and it's technically crazy and ahead of its time [1]. It started in 2003 and I think something else like it didn't appear for more than 10 years [2], and is still research while PyPy is used in production. So yeah it is hugely impressive, from a technical standpoint. But I see that there's not enough "carrot" to pay the cost of adoption, for most users. It's faster for some workloads, but predicting the speedup is nontrivial. And it also uses more memory. It's unfortunate that the Python/C API is so complex and the extension ecosystem is so fragmented with wrappers as a result (SWIG, ctypes, cffi, etc.). I think there is a network effect with CPython that is very hard to break. Everyone builds for CPython because it already has a large amount of useful C code in its ecosystem -- not to mention stuff like NumPy, Pandas, etc. which are a mix of C and Cython, Cython being a totally separate effort to write "faster Python". [1] https://llvm.org/pubs/2006-10-DLS-PyPy.html https://llvm.org/pubs/2006-10-DLS-PyPy.html [2] Graal and Truffle seems like the closest thing, i.e. a meta-tracing JIT where you get a "JIT for free": https://blog.plan99.net/graal-truffle-134d8f28fb69 https://blog.plan99.net/graal-truffle-134d8f28fb69 http://tratt.net/laurie/research/pubs/html/bolz_tratt__the_impact_of_metatracing_on_vm_design_and_implementation/ http://tratt.net/laurie/research/pubs/html/bolz_tratt__the_i...

daveFNbuck 9y ago
At work, I use PyPy to run a Luigi server for our data pipeline. It gave us a pretty dramatic decrease in latency and allowed us to scale our workload with a trivial change. It's unfortunate you can't choose your interpreter, but when you can it's pretty easy to just test both and see which one is faster. I also use it at home to solve Project Euler problems. For anything that takes more than half a second or so, it's usually a big win and can turn minutes into seconds.
- Myrmornis 9y ago
  Do any of the Project Euler questions actually require lengthy computation, or is that more a sign that one should be looking for some mathematical insight that leads to a more efficient computation?
  - xapata 9y ago
    PyPy won't change your data structures and algorithms, so it's not finding mathematical insight.
    - Myrmornis 9y ago
      Yes that’s the point I was making. Maybe you intended your comment to be a sibling to mine.
      - xapata 9y ago
        More likely I missed your point.
      - daveFNbuck 9y ago
        It seems that you're saying that I should be finding mathematical insight instead of using PyPy, while xapata is saying that if my code runs fast in PyPy then I've already found enough mathematical insight.
  - daveFNbuck 9y ago
    Most of them don't require lengthy computation, but some of them did for me. I know at least some of the time that's because I didn't know enough math, but sometimes I had the same algorithm as the other people in the solution thread but with much slower runtimes. I think numpy gives a bigger speedup than PyPy for a lot of the problems.
- gsam 9y ago
  > For anything that takes more than half a second or so, it's usually a big win and can turn minutes into seconds. As a counter to this, I purposefully try to use CPython in order to force me to find algorithmically optimal solutions. It's a useful way to learn.

ojii 9y ago

The main reason I'm not even considering PyPy is that it lags behind CPython by a couple of versions. I'm getting ready to move to 3.7 while PyPy is still on 3.5.