5 ms·

It seems like the biggest point here is that compatibility with the extension API in Python is the anchor dragging down performance. Similarly, Ruby suffers fro

It seems like the biggest point here is that compatibility with the extension API in Python is the anchor dragging down performance. Similarly, Ruby suffers from continued compatibility with its extension API. In the paper they compare with Lua and JavaScript, but it's worth noting that JavaScript doesn't have an extension API (well, JavaScript proper...not saying anything about node.js), and the team behind Lua are notorious for not caring about breaking API compatibility between versions. I guess what I'm trying to say is, while this is a laudable effort, it seems what Python (and Ruby) really needs is a way to free itself from the chains of extension API compatibility.

justincormack 13y ago
Lua versions that break API compatibility are only every few years, but not keeping compatibility forever is sensible.
ProblemFactory 13y ago
On the other hand, having a stable C extension API is what allows Numpy and Cython (http://docs.cython.org/src/quickstart/cythonize.html http://docs.cython.org/src/quickstart/cythonize.html) to be as good and fast as they are. My experience with numerical algorithms in Python has been a 10000x speedup and 10x memory reduction by using Numpy arrays and C/fortran functions. And when Numpy doesn't have a necessary function, a few extra lines of static C types and on-import compilation with Cython has still resulted in a 1000x speedup. "Optimizing Python" needs to look at the whole existing ecosystem of C libraries - to be useful the end result has to be faster than what's possible now, not just faster than pure Python.
- tomrod 13y ago
  What exactly is Cython? I've not had a reason to use it yet; are the speed improvements worth it?
  - ProblemFactory 13y ago
    Cython is a runtime Python-to-C compiler. The speed improvements are worth it given how easy it is to use, I have seen literally a 1000x speedup on a time-series data analysis script from adding about 10 lines of static type annotations. The functionality I like best about it are: * It can be used transparently without any makefiles or compilation steps. Add "import pyximport; pyximport.install()" at the top of your main script, and now every imported Cython-capable module is compiled on the fly at runtime. * You can start out with no changes to your python modules. All libraries and features still work within compiled modules. Then you can slowly start adding static types to a few variables at a time. The annotated variables become very fast native C integers/doubles/functions, instead of Python objects.
    - tomrod 13y ago
      Sounds intriguing. I know a fair bit of C, python, Fortran. I've used f2py a bit. Do you have any good Cython tutorial you like to recommend?
      - ProblemFactory 13y ago
        For my use cases, the standard documentation and tutorials (http://docs.cython.org/index.html http://docs.cython.org/index.html) have been enough. * Install it with pip, the packages in your OS distribution may be out of date, * Rename your numeric code module's .py file to .pyx, * Use pyximport from the main script ( http://docs.cython.org/src/userguide/source_files_and_compilation.html#pyximport http://docs.cython.org/src/userguide/source_files_and_compil...) to have it compiled at runtime without any extra build steps, and then * Start experimenting with adding a few "cdef"s (http://docs.cython.org/src/quickstart/cythonize.html http://docs.cython.org/src/quickstart/cythonize.html and http://docs.cython.org/src/tutorial/numpy.html http://docs.cython.org/src/tutorial/numpy.html)

justincormack 13y ago

Lua versions that break API compatibility are only every few years, but not keeping compatibility forever is sensible.

ProblemFactory 13y ago

On the other hand, having a stable C extension API is what allows Numpy and Cython (http://docs.cython.org/src/quickstart/cythonize.html http://docs.cython.org/src/quickstart/cythonize.html) to be as good and fast as they are. My experience with numerical algorithms in Python has been a 10000x speedup and 10x memory reduction by using Numpy arrays and C/fortran functions. And when Numpy doesn't have a necessary function, a few extra lines of static C types and on-import compilation with Cython has still resulted in a 1000x speedup. "Optimizing Python" needs to look at the whole existing ecosystem of C libraries - to be useful the end result has to be faster than what's possible now, not just faster than pure Python.

tomrod 13y ago
What exactly is Cython? I've not had a reason to use it yet; are the speed improvements worth it?
- ProblemFactory 13y ago
  Cython is a runtime Python-to-C compiler. The speed improvements are worth it given how easy it is to use, I have seen literally a 1000x speedup on a time-series data analysis script from adding about 10 lines of static type annotations. The functionality I like best about it are: * It can be used transparently without any makefiles or compilation steps. Add "import pyximport; pyximport.install()" at the top of your main script, and now every imported Cython-capable module is compiled on the fly at runtime. * You can start out with no changes to your python modules. All libraries and features still work within compiled modules. Then you can slowly start adding static types to a few variables at a time. The annotated variables become very fast native C integers/doubles/functions, instead of Python objects.
  - tomrod 13y ago
    Sounds intriguing. I know a fair bit of C, python, Fortran. I've used f2py a bit. Do you have any good Cython tutorial you like to recommend?
    - ProblemFactory 13y ago
      For my use cases, the standard documentation and tutorials (http://docs.cython.org/index.html http://docs.cython.org/index.html) have been enough. * Install it with pip, the packages in your OS distribution may be out of date, * Rename your numeric code module's .py file to .pyx, * Use pyximport from the main script ( http://docs.cython.org/src/userguide/source_files_and_compilation.html#pyximport http://docs.cython.org/src/userguide/source_files_and_compil...) to have it compiled at runtime without any extra build steps, and then * Start experimenting with adding a few "cdef"s (http://docs.cython.org/src/quickstart/cythonize.html http://docs.cython.org/src/quickstart/cythonize.html and http://docs.cython.org/src/tutorial/numpy.html http://docs.cython.org/src/tutorial/numpy.html)