5 ms·

The scientific community should focus on moving to PyPy and helping to excise CPython C extensions from more scientific libraries. That's at least a change tha

The scientific community should focus on moving to PyPy and helping to excise CPython C extensions from more scientific libraries. That's at least a change that will benefit them.

coliveira 11y ago
The scientific community will use whatever makes it easier to achieve its own goals. The idea that you need to use something because it is cooler, more open, fancier, etc., works only in the programming community, where software itself is the end goal.
ngoldbaum 11y ago
At a minimum, that would require largely rewriting scipy, pandas, and any other piece of software that uses the CPython C extension interface rather than the FFI. That's a pretty tall order...
- filmor 11y ago
  SciPy and Pandas use Cython, so if that can ever be made runnable semi-directly in PyPy, those two are dealt with. Only NumPy uses the CPython API directly but this is being worked on as NumPyPy (see http://morepypy.blogspot.de/2015/02/linalg-support-in-pypynumpy.html http://morepypy.blogspot.de/2015/02/linalg-support-in-pypynu...).
  - maxerickson 11y ago
    SciPy uses Cython, but it also uses other C interfaces. The most definitive statement I found in a couple minutes is here: http://www.scipy.org/scipylib/faq.html#how-can-scipy-be-fast-if-it-is-written-in-an-interpreted-language-like-python http://www.scipy.org/scipylib/faq.html#how-can-scipy-be-fast... SciPy uses a variety of methods to generate “wrappers” around these algorithms so that they can be used in Python. Some wrappers were generated by hand coding them in C. The rest were generated using either SWIG or f2py. Some of the newer contributions to SciPy are either written entirely or wrapped with Cython.
    - pyre 11y ago
      > f2py This is a Fortran-to-Python interface, and has some support for Python 3 so far as I can tell.
rhodysurf 11y ago
The C extensions shouldnt be cut out of Python. I get that PyPy is better but cutting out c sxtensions and rewriting them in pure python would hurt performance more.
- dalke 11y ago
  There are several other options. The extension can be written as a pure C extension, with CFFI for the bindings, and code in Python to present the public API. Or write the extension in Cython, which has backends for both CPython and PyPy's cpyext (I have no experience with this option).
- reipahb 11y ago
  Personally, I would really like to see more Python extensions written using the ctypes foreign function library. This has two advantages: * Supported on more than just CPython. I.e. you can use them on PyPy as well. * The extensions are far easier to install, since one doesn't need both a full C compiler as well as all development headers for the library installed on the server where you install the extension.
  - fdej 11y ago
    ctypes overhead is horrendous though, at least in CPython (I don't know about PyPy). Fine if you're doing array operations on huge arrays, but not so if you have lots of small objects. I wanted to use ctypes to wrap a C library for a recent project (for the ease of installation and development that you mention) but had to give up when it turned out to be more than 10x slower than a wrapper written in Cython, and barely faster than doing a pure Python implementation of the C library itself.
- JulianWasTaken 11y ago
  It's unclear that that's true (that performance would be worse), but regardless, there are many other, better ways to run C code from within Python (CFFI being the easiest and nicest currently)
  - deleted 11y ago
syllogism 11y ago
How about...no? PyPy is a nice way to make most Python programs 20-100% faster, often at the cost of using slightly more memory. Writing in C/C++/Cython etc, we typically see 100x run-time and memory improvements. These extensions were written in C for a reason. If I had to write in pure Python, I'd find another language for the bulk of my work.
- JulianWasTaken 11y ago
  The reason is that PyPy didn't exist, and C extensions were the way to go back then. That is no longer true. Anything else is misinformation. I don't use numpy (or really very many scientific libraries), and talking about generalities like you mentioned just isn't meaningful, but here's a really old benchmark which is (small) evidence that it's just not true that these things need to be written in C anymore: http://morepypy.blogspot.com/2012/01/numpypy-progress-report-running.html http://morepypy.blogspot.com/2012/01/numpypy-progress-report...
  - syllogism 11y ago
    Why are you trying to push this point, when you plainly admit you don't know much about this? And so strongly, too! "Anything else is misinformation." I've seldom seen someone so confidently wrong. PyPy can address Python's slow loop performance and slow numeric calculations, in particular examples. This can help a lot. But if the problem requires you to carefully manage your data structures so that you get good cache performance, PyPy cannot help much at all. Consider this: there are very few non-toy examples of PyPy giving order of magnitude efficiency improvements over CPython, while other languages do typically run that much faster than Python implementations. Finally, if you don't want to talk about generalities...Consider my NLP library, spaCy: http://honnibal.github.io/spaCy/#speed-comparison http://honnibal.github.io/spaCy/#speed-comparison . It's the fastest library with this functionality in the world, among any language. This library is an example where you need C data structures just to make the working set manageable. You would need an enormous amount of memory just to load the models. In pure Python, the library would not be of much use to anyone --- so I would have written it in Java or C++.
eeZi 11y ago
Actually, most of the time, the performance-critical computations happens in the highly optimized C modules already, so there's not /that/ much to gain. That being said, the PyPy team is working on a NumPy integration (NumPyPy). In many cases, you can use something like python-bond to call PyPy from a regular CPython program: https://pypi.python.org/pypi/python-bond https://pypi.python.org/pypi/python-bond (or just use multiprocessing).

The scientific community should focus on moving to PyPy and helping to excise CPython C extensions from more scientific libraries. That's at least a change that will benefit them.

coliveira 11y ago

The scientific community will use whatever makes it easier to achieve its own goals. The idea that you need to use something because it is cooler, more open, fancier, etc., works only in the programming community, where software itself is the end goal.

ngoldbaum 11y ago

At a minimum, that would require largely rewriting scipy, pandas, and any other piece of software that uses the CPython C extension interface rather than the FFI. That's a pretty tall order...

filmor 11y ago
SciPy and Pandas use Cython, so if that can ever be made runnable semi-directly in PyPy, those two are dealt with. Only NumPy uses the CPython API directly but this is being worked on as NumPyPy (see http://morepypy.blogspot.de/2015/02/linalg-support-in-pypynumpy.html http://morepypy.blogspot.de/2015/02/linalg-support-in-pypynu...).
- maxerickson 11y ago
  SciPy uses Cython, but it also uses other C interfaces. The most definitive statement I found in a couple minutes is here: http://www.scipy.org/scipylib/faq.html#how-can-scipy-be-fast-if-it-is-written-in-an-interpreted-language-like-python http://www.scipy.org/scipylib/faq.html#how-can-scipy-be-fast... SciPy uses a variety of methods to generate “wrappers” around these algorithms so that they can be used in Python. Some wrappers were generated by hand coding them in C. The rest were generated using either SWIG or f2py. Some of the newer contributions to SciPy are either written entirely or wrapped with Cython.
  - pyre 11y ago
    > f2py This is a Fortran-to-Python interface, and has some support for Python 3 so far as I can tell.

rhodysurf 11y ago

The C extensions shouldnt be cut out of Python. I get that PyPy is better but cutting out c sxtensions and rewriting them in pure python would hurt performance more.

dalke 11y ago
There are several other options. The extension can be written as a pure C extension, with CFFI for the bindings, and code in Python to present the public API. Or write the extension in Cython, which has backends for both CPython and PyPy's cpyext (I have no experience with this option).
reipahb 11y ago
Personally, I would really like to see more Python extensions written using the ctypes foreign function library. This has two advantages: * Supported on more than just CPython. I.e. you can use them on PyPy as well. * The extensions are far easier to install, since one doesn't need both a full C compiler as well as all development headers for the library installed on the server where you install the extension.
- fdej 11y ago
  ctypes overhead is horrendous though, at least in CPython (I don't know about PyPy). Fine if you're doing array operations on huge arrays, but not so if you have lots of small objects. I wanted to use ctypes to wrap a C library for a recent project (for the ease of installation and development that you mention) but had to give up when it turned out to be more than 10x slower than a wrapper written in Cython, and barely faster than doing a pure Python implementation of the C library itself.
JulianWasTaken 11y ago
It's unclear that that's true (that performance would be worse), but regardless, there are many other, better ways to run C code from within Python (CFFI being the easiest and nicest currently)
- deleted 11y ago

syllogism 11y ago

How about...no? PyPy is a nice way to make most Python programs 20-100% faster, often at the cost of using slightly more memory. Writing in C/C++/Cython etc, we typically see 100x run-time and memory improvements. These extensions were written in C for a reason. If I had to write in pure Python, I'd find another language for the bulk of my work.

JulianWasTaken 11y ago
The reason is that PyPy didn't exist, and C extensions were the way to go back then. That is no longer true. Anything else is misinformation. I don't use numpy (or really very many scientific libraries), and talking about generalities like you mentioned just isn't meaningful, but here's a really old benchmark which is (small) evidence that it's just not true that these things need to be written in C anymore: http://morepypy.blogspot.com/2012/01/numpypy-progress-report-running.html http://morepypy.blogspot.com/2012/01/numpypy-progress-report...
- syllogism 11y ago
  Why are you trying to push this point, when you plainly admit you don't know much about this? And so strongly, too! "Anything else is misinformation." I've seldom seen someone so confidently wrong. PyPy can address Python's slow loop performance and slow numeric calculations, in particular examples. This can help a lot. But if the problem requires you to carefully manage your data structures so that you get good cache performance, PyPy cannot help much at all. Consider this: there are very few non-toy examples of PyPy giving order of magnitude efficiency improvements over CPython, while other languages do typically run that much faster than Python implementations. Finally, if you don't want to talk about generalities...Consider my NLP library, spaCy: http://honnibal.github.io/spaCy/#speed-comparison http://honnibal.github.io/spaCy/#speed-comparison . It's the fastest library with this functionality in the world, among any language. This library is an example where you need C data structures just to make the working set manageable. You would need an enormous amount of memory just to load the models. In pure Python, the library would not be of much use to anyone --- so I would have written it in Java or C++.

eeZi 11y ago

Actually, most of the time, the performance-critical computations happens in the highly optimized C modules already, so there's not /that/ much to gain. That being said, the PyPy team is working on a NumPy integration (NumPyPy). In many cases, you can use something like python-bond to call PyPy from a regular CPython program: https://pypi.python.org/pypi/python-bond https://pypi.python.org/pypi/python-bond (or just use multiprocessing).