3 ms·

This looks like the holy grail of extension development, effectively a Python runtime HAL, it's wildly exciting. Getting anything to work optimally on PyPy and

by dmw_ng 6y ago

This looks like the holy grail of extension development, effectively a Python runtime HAL, it's wildly exciting. Getting anything to work optimally on PyPy and CPython more or less means writing it twice. CFFI overhead on CPython really sucks for anything where the runtime of the wrapped function does not eclipse the overhead of the binding. HPy looks like a fantastic approach. Only complaint from a quick scan -- carrying over that horrid PyArg_Parse() API. It's huge, needs varargs, isn't type safe, repeatedly rebuilds strings to look them up in kwargs, and isn't even all that convenient to use. Also, if you haven't dealt with a SEGV due to PyArg_Parse() before, it means you probably have never tried to write an extension PyPy will become about 1000x more practical if something like this ever makes it into Cython. That would make it easy to bring over huge swathes of existing extensions like lxml.

TazeTSchnitzel 6y ago
I'm not familiar with the Python runtime, but PyArg_Parse sounds very much like a PHP C API function (zend_parse_parameters) which has the same purpose and problems. Several years ago, an alternative API was added where you did one function call (actually macro) per argument. It has worse code size but better performance. Maybe Python can do the same?
antocuni 6y ago
We plan to introduce something similar to CPython's argument clinic eventually. I.e., you will be able to declare what is the C signature of your impl function and HPy will generate the argument parsing code automatically. One nice effect of such an API is that under certain conditions, implementations will be allowed to bypass argument parsing entirely: e.g., if I have a function which takes two C longs, the PyPy JIT will be able to emit a call directly to it, without having to box/unbox the arguments just to do the call as it happens right now. However, on the other hand we will need to provide an API similar to the existing PyArg_ParseTuple, because we want to make it easy to migrate existing extensions

by dmw_ng 6y ago

TazeTSchnitzel 6y ago

I'm not familiar with the Python runtime, but PyArg_Parse sounds very much like a PHP C API function (zend_parse_parameters) which has the same purpose and problems. Several years ago, an alternative API was added where you did one function call (actually macro) per argument. It has worse code size but better performance. Maybe Python can do the same?

antocuni 6y ago

We plan to introduce something similar to CPython's argument clinic eventually. I.e., you will be able to declare what is the C signature of your impl function and HPy will generate the argument parsing code automatically. One nice effect of such an API is that under certain conditions, implementations will be allowed to bypass argument parsing entirely: e.g., if I have a function which takes two C longs, the PyPy JIT will be able to emit a call directly to it, without having to box/unbox the arguments just to do the call as it happens right now. However, on the other hand we will need to provide an API similar to the existing PyArg_ParseTuple, because we want to make it easy to migrate existing extensions