5 ms·

This project keeps amazing me. It seems like a R&D lab going for moonshots on the python language, and yet achieving incredible practical results. Does anyone

by bsaul 10y ago

This project keeps amazing me. It seems like a R&D lab going for moonshots on the python language, and yet achieving incredible practical results. Does anyone know why it still hasn't managed to get enough traction to become the official path for the language, and why it still needs mozilla support to get funds ? My intuition is that it's just too complex. But haskell,scala,swift and rust also have a lot of complexity so that can't be the only reason.

rafinha 10y ago
PIL
- LeifCarrotson 10y ago
  Expound, please? Do you mean to refer to the Global Interpreter Lock as the factor which prevents it from becoming the official path for the language? CPython, the reference implementation, has the GIL as well. Or the PIL library (though everyone seems to use Pillow now?)
- rafinha 10y ago
  oops, I meant GIL. I work with scientific computing. In my experience, the only justification I have of using other language (mostly C++) over python is to get over the interpreter lock, which over complicates your life if you want parallelisation. Multiprocessing is just awful, specially if you work with big chunks of data (I do mostly medical imaging). PyPy, even if it's faster, still has the same GIL constraints as CPython, so not really a big advantage IMO.
  - fijal 10y ago
    We're working on it :-) In fact there are two projects - one is to just remove the GIL, one is to add STM-capabilities.
afarrell 10y ago
What is the experience for someone trying it out for the first time and trying to install packages in it? What is the UX for someone building packages and verifying that they work?
- mikeash 10y ago
  I have a performance-critical Python program that I started running in pypy for an extra speedup. It was trivial: I downloaded the package, untarred it, and then ran my program with ~/Downloads/pypy-whateverversion/bin/pypy ./myprogram.py and it worked. Fancier stuff may require more effort, but the basics are really simple.
wyldfire 10y ago
> Does anyone know why it still hasn't managed to get enough traction to become the official path for the language I'd wager BDFL prefers the CPython implementation because it's probably simpler. Also, it supports tons of targets and is super simple to build. It's healthy for a language to have more than one implementation. A big win would be a linux distro selecting pypy for use as the default Python. But IMO many python scripts that folks use just don't run long enough to see a benefit from PyPy.
- data_hope 10y ago
  > I'd wager BDFL prefers the CPython implementation because it's probably simpler. Also, it supports tons of targets and is super simple to build. At a EuroPython Keynote, the BDFL mentioned that he hasn't had a closer look at PyPy (he mentioned downloading it and playing with it for a few minutes). I.e. there is a certain disinterest. Also, remember that the "Zen of Python" (https://www.python.org/dev/peps/pep-0020/#id3 https://www.python.org/dev/peps/pep-0020/#id3) was written about the design principles of the Python interpreter, and PyPy is not exactly the Zen of python. Personally, I'd love to see Python 4 to be based entirely on PyPy.
- ronjouch 10y ago
  > "I'd wager BDFL prefers the CPython implementation because it's probably simpler. Also, it supports tons of targets and is super simple to build." You're doing good wagering, Guido mentioned just this at several keynotes (source: skim through his PyCon/EuroPython keynotes, there's a good chance he addresses the point or answers a question about it). I'd paraphrase him with: although now a big project with optimizations here and there obscuring the purpose of many parts, CPython remains a boring well-understood "no rocket science" C project. As an example of this, I remember him mentioning how the opcode dispatcher remains a giant `switch` statement rather than something more intricate. That doesn't mean he dislikes or wants to hinder PyPy (or IronPython or Jython or Pyston or ...), just that he's fine with CPython reach and trade-offs as the "default python", and if you prefer another Python with different performance/compatibility/featureset/xyz characteristics, you're welcome to grab it :)
  - 10y ago
catdog 10y ago
> Does anyone know why it still hasn't managed to get enough traction to become the official path for the language One reason is that CPython is kept relatively simple on purpose, so it abstains from implementing things like a jit or other very complex optimizations. The biggest obstacle for pypy adoption is that it can't serve as a drop in replacement in the majority of cases, be it because its behind (until now far behind) in supporting new versions of the language or because a lot of popular packages rely on C extensions. The CPython way of binding to C on which these packages rely on does not fit the architecture of pypy very well so they did not really support it at all until recently.
hyperbovine 10y ago
> Does anyone know why it still hasn't managed to get enough traction to become the official path for the language The people who really care about Python performance (sci. comp & HPC) are the ones driving the (non-)adoption of PyPy. One, the existing solution (CPython + Cython) is pretty mature. Two, they require the holy trinity: Numpy, Scipy and Matplotlib. None of these are supported on PyPy, and only Numpy is even close.
- fijal 10y ago
  numpy works, scipy & matplotlib are very close. Stop spreading false information please. That said, the cpython+cython solution is not going away any time soon.
  - hyperbovine 10y ago
    > numpy works, scipy & matplotlib are very close. Care to elaborate? I agree that numpypy is pretty much complete, though I have not found it to be faster in my limited testing. I was under the impression that a great deal of work would be required to get the other two running, even using CPyExt (which would not be faster, which was the whole point to begin with). I'd love to be proved wrong though.
    - mattip 10y ago
      Real world use cases are large and messy. Now that the cython/pandas/numpy stack works out of the box on PyPy, we can begin to see how often the mix of python code (jittable, fast) and c code (fast) needs to cross the c-to-python barrier (slow). There may be some cases, like NLP where speeding up parsing and io can make a difference
    - deleted 10y ago
      [deleted]
    - fijal 10y ago
      numpy, pandas (mostly) and (scipy) mostly work through cpyext. We're considering how to make them fast now. It would never be faster on large_array + large_array, but it should be faster (by a lot) with element access etc., bit like numba

by bsaul 10y ago

rafinha 10y ago

PIL

LeifCarrotson 10y ago
Expound, please? Do you mean to refer to the Global Interpreter Lock as the factor which prevents it from becoming the official path for the language? CPython, the reference implementation, has the GIL as well. Or the PIL library (though everyone seems to use Pillow now?)
rafinha 10y ago
oops, I meant GIL. I work with scientific computing. In my experience, the only justification I have of using other language (mostly C++) over python is to get over the interpreter lock, which over complicates your life if you want parallelisation. Multiprocessing is just awful, specially if you work with big chunks of data (I do mostly medical imaging). PyPy, even if it's faster, still has the same GIL constraints as CPython, so not really a big advantage IMO.
- fijal 10y ago
  We're working on it :-) In fact there are two projects - one is to just remove the GIL, one is to add STM-capabilities.

afarrell 10y ago

What is the experience for someone trying it out for the first time and trying to install packages in it? What is the UX for someone building packages and verifying that they work?

mikeash 10y ago
I have a performance-critical Python program that I started running in pypy for an extra speedup. It was trivial: I downloaded the package, untarred it, and then ran my program with ~/Downloads/pypy-whateverversion/bin/pypy ./myprogram.py and it worked. Fancier stuff may require more effort, but the basics are really simple.

wyldfire 10y ago

> Does anyone know why it still hasn't managed to get enough traction to become the official path for the language I'd wager BDFL prefers the CPython implementation because it's probably simpler. Also, it supports tons of targets and is super simple to build. It's healthy for a language to have more than one implementation. A big win would be a linux distro selecting pypy for use as the default Python. But IMO many python scripts that folks use just don't run long enough to see a benefit from PyPy.

data_hope 10y ago
> I'd wager BDFL prefers the CPython implementation because it's probably simpler. Also, it supports tons of targets and is super simple to build. At a EuroPython Keynote, the BDFL mentioned that he hasn't had a closer look at PyPy (he mentioned downloading it and playing with it for a few minutes). I.e. there is a certain disinterest. Also, remember that the "Zen of Python" (https://www.python.org/dev/peps/pep-0020/#id3 https://www.python.org/dev/peps/pep-0020/#id3) was written about the design principles of the Python interpreter, and PyPy is not exactly the Zen of python. Personally, I'd love to see Python 4 to be based entirely on PyPy.
ronjouch 10y ago
> "I'd wager BDFL prefers the CPython implementation because it's probably simpler. Also, it supports tons of targets and is super simple to build." You're doing good wagering, Guido mentioned just this at several keynotes (source: skim through his PyCon/EuroPython keynotes, there's a good chance he addresses the point or answers a question about it). I'd paraphrase him with: although now a big project with optimizations here and there obscuring the purpose of many parts, CPython remains a boring well-understood "no rocket science" C project. As an example of this, I remember him mentioning how the opcode dispatcher remains a giant `switch` statement rather than something more intricate. That doesn't mean he dislikes or wants to hinder PyPy (or IronPython or Jython or Pyston or ...), just that he's fine with CPython reach and trade-offs as the "default python", and if you prefer another Python with different performance/compatibility/featureset/xyz characteristics, you're welcome to grab it :)
- 10y ago

catdog 10y ago

> Does anyone know why it still hasn't managed to get enough traction to become the official path for the language One reason is that CPython is kept relatively simple on purpose, so it abstains from implementing things like a jit or other very complex optimizations. The biggest obstacle for pypy adoption is that it can't serve as a drop in replacement in the majority of cases, be it because its behind (until now far behind) in supporting new versions of the language or because a lot of popular packages rely on C extensions. The CPython way of binding to C on which these packages rely on does not fit the architecture of pypy very well so they did not really support it at all until recently.

hyperbovine 10y ago

> Does anyone know why it still hasn't managed to get enough traction to become the official path for the language The people who really care about Python performance (sci. comp & HPC) are the ones driving the (non-)adoption of PyPy. One, the existing solution (CPython + Cython) is pretty mature. Two, they require the holy trinity: Numpy, Scipy and Matplotlib. None of these are supported on PyPy, and only Numpy is even close.

fijal 10y ago
numpy works, scipy & matplotlib are very close. Stop spreading false information please. That said, the cpython+cython solution is not going away any time soon.
- hyperbovine 10y ago
  > numpy works, scipy & matplotlib are very close. Care to elaborate? I agree that numpypy is pretty much complete, though I have not found it to be faster in my limited testing. I was under the impression that a great deal of work would be required to get the other two running, even using CPyExt (which would not be faster, which was the whole point to begin with). I'd love to be proved wrong though.
  - mattip 10y ago
    Real world use cases are large and messy. Now that the cython/pandas/numpy stack works out of the box on PyPy, we can begin to see how often the mix of python code (jittable, fast) and c code (fast) needs to cross the c-to-python barrier (slow). There may be some cases, like NLP where speeding up parsing and io can make a difference
  - deleted 10y ago
    [deleted]
  - fijal 10y ago
    numpy, pandas (mostly) and (scipy) mostly work through cpyext. We're considering how to make them fast now. It would never be faster on large_array + large_array, but it should be faster (by a lot) with element access etc., bit like numba