4 ms·

I'd like to see optimizations targeting ctypes, or a successor to ctypes. I wish I could write elegant, performant C wrappers in pure Python. Right now the best

by fdej 5y ago

I'd like to see optimizations targeting ctypes, or a successor to ctypes. I wish I could write elegant, performant C wrappers in pure Python. Right now the best choices are Cython, which is a hassle (separate, slow compilation, various warts), and ctypes, which is slow (and has some design problems of its own). Julia's ccall alone is a major selling point over Python right now.

BiteCode_dev 5y ago
It's not nuitka main purpose, but it can produce a lib and speed up the code: https://nuitka.net/ https://nuitka.net/
ihnorton 5y ago
CFFI [1] is a step in the right direction, inspired by LuaJIT's CFFI. It originated from the PyPy folks and is supported in PyPy [2]; it's also supported to some degree by Numba [3]. I don't know what level of C call optimization is available when using the JITs, so I can't speak to the performance, but I've used it casually via CPython and was impressed by the API. That said, it has been around for a while and the traction seems somewhat limited -- I would guess because most people who have this kind of problem also need more than "just" FFI. [1] https://cffi.readthedocs.io/en/latest/index.html https://cffi.readthedocs.io/en/latest/index.html [2] https://doc.pypy.org/en/latest/extending.html#cffi https://doc.pypy.org/en/latest/extending.html#cffi [3] https://numba.readthedocs.io/en/stable/reference/pysupported.html#cffi-support https://numba.readthedocs.io/en/stable/reference/pysupported...
adsharma 5y ago
Please take a look at py2many which does this. Looking for feedback.
- adsharma 5y ago
  There is now a pull request for this: https://github.com/adsharma/py2many/issues/62 https://github.com/adsharma/py2many/issues/62
adsharma 5y ago
@fdej: what successor to ctypes did you have in mind? Other than the names (I prefer i8 over c_int8, but easy to work around) and lack of ergonomic interop with python ints, are there other issues you're aware of?
- fdej 5y ago
  My number one wish would be the ability to read definitions from .h files automatically. Other than that, I've had some issues with memory management with ctypes (objects being deallocated prematurely) that I never had with Cython, but that may just be my own fault. I agree about the lack of interop with Python ints.
  - morelisp 5y ago
    > My number one wish would be the ability to read definitions from .h files automatically. This was originally a feature of ctypes but was dropped before stdlib inclusion because it's difficult to solve in the general, "public distribution" case. Nonetheless I don't think there's any need to replace ctypes to reintroduce it, it was done as a library / external tool before and still could be. https://svn.python.org/projects/ctypes/trunk/ctypeslib/ctypeslib/codegen/ https://svn.python.org/projects/ctypes/trunk/ctypeslib/ctype...
simonw 5y ago
I'm excited about WASM here. https://github.com/wasmerio/wasmer-python https://github.com/wasmerio/wasmer-python lets you call WASM binaries from Python, which means if you can compile a C library to WASM you can then call it from Python... without having to worry about introducing crashing bugs and security vulnerabilities thanks to the WASM sandbox.

by fdej 5y ago

BiteCode_dev 5y ago

It's not nuitka main purpose, but it can produce a lib and speed up the code: https://nuitka.net/ https://nuitka.net/

ihnorton 5y ago

CFFI [1] is a step in the right direction, inspired by LuaJIT's CFFI. It originated from the PyPy folks and is supported in PyPy [2]; it's also supported to some degree by Numba [3]. I don't know what level of C call optimization is available when using the JITs, so I can't speak to the performance, but I've used it casually via CPython and was impressed by the API. That said, it has been around for a while and the traction seems somewhat limited -- I would guess because most people who have this kind of problem also need more than "just" FFI. [1] https://cffi.readthedocs.io/en/latest/index.html https://cffi.readthedocs.io/en/latest/index.html [2] https://doc.pypy.org/en/latest/extending.html#cffi https://doc.pypy.org/en/latest/extending.html#cffi [3] https://numba.readthedocs.io/en/stable/reference/pysupported.html#cffi-support https://numba.readthedocs.io/en/stable/reference/pysupported...

adsharma 5y ago

Please take a look at py2many which does this. Looking for feedback.

adsharma 5y ago
There is now a pull request for this: https://github.com/adsharma/py2many/issues/62 https://github.com/adsharma/py2many/issues/62

adsharma 5y ago

@fdej: what successor to ctypes did you have in mind? Other than the names (I prefer i8 over c_int8, but easy to work around) and lack of ergonomic interop with python ints, are there other issues you're aware of?

fdej 5y ago
My number one wish would be the ability to read definitions from .h files automatically. Other than that, I've had some issues with memory management with ctypes (objects being deallocated prematurely) that I never had with Cython, but that may just be my own fault. I agree about the lack of interop with Python ints.
- morelisp 5y ago
  > My number one wish would be the ability to read definitions from .h files automatically. This was originally a feature of ctypes but was dropped before stdlib inclusion because it's difficult to solve in the general, "public distribution" case. Nonetheless I don't think there's any need to replace ctypes to reintroduce it, it was done as a library / external tool before and still could be. https://svn.python.org/projects/ctypes/trunk/ctypeslib/ctypeslib/codegen/ https://svn.python.org/projects/ctypes/trunk/ctypeslib/ctype...

simonw 5y ago

I'm excited about WASM here. https://github.com/wasmerio/wasmer-python https://github.com/wasmerio/wasmer-python lets you call WASM binaries from Python, which means if you can compile a C library to WASM you can then call it from Python... without having to worry about introducing crashing bugs and security vulnerabilities thanks to the WASM sandbox.