4 ms·

On the other hand, having a stable C extension API is what allows Numpy and Cython (http://docs.cython.org/src/quickstart/cythonize.html http://docs.cython.org/

On the other hand, having a stable C extension API is what allows Numpy and Cython (http://docs.cython.org/src/quickstart/cythonize.html http://docs.cython.org/src/quickstart/cythonize.html) to be as good and fast as they are. My experience with numerical algorithms in Python has been a 10000x speedup and 10x memory reduction by using Numpy arrays and C/fortran functions. And when Numpy doesn't have a necessary function, a few extra lines of static C types and on-import compilation with Cython has still resulted in a 1000x speedup. "Optimizing Python" needs to look at the whole existing ecosystem of C libraries - to be useful the end result has to be faster than what's possible now, not just faster than pure Python.

tomrod 13y ago
What exactly is Cython? I've not had a reason to use it yet; are the speed improvements worth it?
- ProblemFactory 13y ago
  Cython is a runtime Python-to-C compiler. The speed improvements are worth it given how easy it is to use, I have seen literally a 1000x speedup on a time-series data analysis script from adding about 10 lines of static type annotations. The functionality I like best about it are: * It can be used transparently without any makefiles or compilation steps. Add "import pyximport; pyximport.install()" at the top of your main script, and now every imported Cython-capable module is compiled on the fly at runtime. * You can start out with no changes to your python modules. All libraries and features still work within compiled modules. Then you can slowly start adding static types to a few variables at a time. The annotated variables become very fast native C integers/doubles/functions, instead of Python objects.
  - tomrod 13y ago
    Sounds intriguing. I know a fair bit of C, python, Fortran. I've used f2py a bit. Do you have any good Cython tutorial you like to recommend?
    - ProblemFactory 13y ago
      For my use cases, the standard documentation and tutorials (http://docs.cython.org/index.html http://docs.cython.org/index.html) have been enough. * Install it with pip, the packages in your OS distribution may be out of date, * Rename your numeric code module's .py file to .pyx, * Use pyximport from the main script ( http://docs.cython.org/src/userguide/source_files_and_compilation.html#pyximport http://docs.cython.org/src/userguide/source_files_and_compil...) to have it compiled at runtime without any extra build steps, and then * Start experimenting with adding a few "cdef"s (http://docs.cython.org/src/quickstart/cythonize.html http://docs.cython.org/src/quickstart/cythonize.html and http://docs.cython.org/src/tutorial/numpy.html http://docs.cython.org/src/tutorial/numpy.html)

tomrod 13y ago

What exactly is Cython? I've not had a reason to use it yet; are the speed improvements worth it?

ProblemFactory 13y ago
Cython is a runtime Python-to-C compiler. The speed improvements are worth it given how easy it is to use, I have seen literally a 1000x speedup on a time-series data analysis script from adding about 10 lines of static type annotations. The functionality I like best about it are: * It can be used transparently without any makefiles or compilation steps. Add "import pyximport; pyximport.install()" at the top of your main script, and now every imported Cython-capable module is compiled on the fly at runtime. * You can start out with no changes to your python modules. All libraries and features still work within compiled modules. Then you can slowly start adding static types to a few variables at a time. The annotated variables become very fast native C integers/doubles/functions, instead of Python objects.
- tomrod 13y ago
  Sounds intriguing. I know a fair bit of C, python, Fortran. I've used f2py a bit. Do you have any good Cython tutorial you like to recommend?
  - ProblemFactory 13y ago
    For my use cases, the standard documentation and tutorials (http://docs.cython.org/index.html http://docs.cython.org/index.html) have been enough. * Install it with pip, the packages in your OS distribution may be out of date, * Rename your numeric code module's .py file to .pyx, * Use pyximport from the main script ( http://docs.cython.org/src/userguide/source_files_and_compilation.html#pyximport http://docs.cython.org/src/userguide/source_files_and_compil...) to have it compiled at runtime without any extra build steps, and then * Start experimenting with adding a few "cdef"s (http://docs.cython.org/src/quickstart/cythonize.html http://docs.cython.org/src/quickstart/cythonize.html and http://docs.cython.org/src/tutorial/numpy.html http://docs.cython.org/src/tutorial/numpy.html)