4 ms·

There is a lot of detail here, but the higher-level takeaway is this: the Python C API is a very "wide" API. It exposes lots of details of CPython: the good, t

There is a lot of detail here, but the higher-level takeaway is this: the Python C API is a very "wide" API. It exposes lots of details of CPython: the good, the bad, and the ugly. You get direct access to pointers to the underlying objects. You manually manage reference counts. For the polar opposite of this, consider the Lua API. You don't get pointers to any VM data structures except a single pointer to the "Lua state". You do not perform any manual memory management. Lua's approach has yielded amazing results. LuaJIT is not only source-compatible with pre-existing Lua 5.1 extensions, it is binary compatible with them. You can take a .so that you built before LuaJIT ever existed and it will work with LuaJIT without recompiling. This is astounding to me. Moral of the story: keep your interfaces as narrow and encapsulated as possible!

Pxtl 8y ago
Yup. People always complain about languages with a "reference implementation" instead of a spec, but because the CPython interpreter exposes so much of its guts into the API, it basically means that Python as a platform is defined by a "reference implementation" instead of a spec.
- deleted 8y ago
  [deleted]
typomatic 8y ago
Worth noting that the PyPy article links to a whole site devoted to the troubles and rehabilitation of the CPython API: https://pythoncapi.readthedocs.io/ https://pythoncapi.readthedocs.io/ . It's really exciting to see these backend concerns being addressed with depth and care, especially against the backdrop of syntactic sugar PEPs blowing up uglily.
- antt 8y ago
  PyPy is one of the few projects that are doing good work, communicating their work well and making the right strategic decisions. At his rate I won't be surprised if python4 is just PyPy.
  - sitkack 8y ago
    If the PyPy team made something that was compatible with both Python2 and Python3, that Python would be that Python.
- quotemstr 8y ago
  Thanks. That's a good link. The proposal is a good start, although a bit lacking on the concrete design for a new API. I'm not sure there's a big need for innovation here though: JNI basically got it right. It provides full isolation from VM implementation details (especially of memory management) while not sacrificing performance or completeness. JNI has a bad reputation for some reason I've never fully understood, but it seems like an elegant approach to me.
wahern 8y ago
I think the central issue is that PyPy uses a copying GC, which means pointers to objects aren't stable by default. The article implies that this is a consequence of having a generational GC, but you can have a non-copying generational GC, which Lua 5.2 and 5.4 actually have.[1] [1] Lua 5.3 dropped the generational GC because real-world results didn't justify the complexity, but apparently for 5.4 they came up with a better design.
- quotemstr 8y ago
  Object stability precludes bump-pointer allocation of short-lived temporaries though, and in a very allocation-y language like Python, that's a pretty important.
  - weberc2 8y ago
    I wonder if escape analysis could compensate as it does with Go.
    - deleted 8y ago
      [deleted]
Boxxed 8y ago
I don't even think it should be considered an API. The python "API" is just all the interpreter internals.
- int_19h 8y ago
  The more accurate term would be "object model with an ABI", in the same sense COM or GLib is an object model. So CPython is actually several things layered on top of each other. - PyObject-based object model; this includes PyObject, PyTypeObject, PyUnicodeObject... and I think that's it? This is the equivalent of COM IUnknown. It doesn't actually know anything about Python proper, but it defines the operations in terms of which language itself will later be defined (like the idea that objects have a refcount-centric lifetime, and have attributes, and operations like "call" and "add" etc). - A bunch of standard data structures built on top of that, like PyLongObject and PyListObject. This is just a pure extension of the above - again, defining more terms in which the language is defined, like what happens when you add two ints. - A bunch of specialized data classes which store Python bytecode and provide the framework for its execution, like PyCodeObject, PyFrameObject and PyFunctionObject. Note that these don't know anything about how the bytecode is produced, nor about how to actually execute it. But they do know about things like local variables (so PyCodeObject will store the list of locals, and PyFrameObject will allocate space for them), so Python-the-language starts creeping in here. - The parser which produces AST (which is itself a bunch of Python objects), and the bytecode compiler that produces PyCodeObjects out of that AST. - And finally, the actual interpreter, that ties it all together by providing semantics for the bytecode contained in PyCodeObjects. This layering is even visible in the Python source itself (https://github.com/python/cpython/tree/v3.7.0 https://github.com/python/cpython/tree/v3.7.0) - the first three things live under ./Objects in the source tree, and the parser and the interpreter is under ./Python (with some bits under ./Grammar and ./Parser). So, roughly speaking, ./Objects is the object model, and ./Python is the language proper. The headers are interdependent, unfortunately, but it's not that hard to break them apart if anyone cared to.
dtech 8y ago
It's a common problem because it is so tempting. Exposing the compiler/runtime internals allows very powerful things at minimal compiler developer effort.
- nicolaslem 8y ago
  This reminds me the case of Firefox: extensions could access almost everything internally which made them very powerful but ultimately made refactoring very hard, hence the switch to web extensions.
dasyatidprime 8y ago
… and then LuaJIT converts started saying “don't ever use the traditional C API, it's so slow and indirect, and the FFI is so easy”, and then anyone not running LuaJIT lost access to a bunch of libraries who stopped caring about the old ways.
kbumsik 8y ago
Never heard of Lua's approach but that sounds interesting. What about overhead between Lua and native? I think the Python C API is one of the reason Python get popular, by doing "prototype in Python then make C codes later for performance" approach. This approach sometime is not desirable in other platforms such as NodeJs because calling a native module from JS is quite costly.
- dividuum 8y ago
  The overhead is really minimal. I happened to have an old program around that uses LuaJIT or Lua5.1. It calls a two functions that each increments a C int. Using the FFI, calling this function 500 million times takes 0.7 seconds. Calling it as an old-style exported function takes 5.4 seconds for 500 million calls. Disabling the JIT increases the FFI time to 32.7s and decreases the old-style time to 4.7 seconds. On Lua5.1, calling the function takes 6.7s. This is in no way representative of any reach benchmark, but as you can see: Lua is pretty fast.

Pxtl 8y ago

Yup. People always complain about languages with a "reference implementation" instead of a spec, but because the CPython interpreter exposes so much of its guts into the API, it basically means that Python as a platform is defined by a "reference implementation" instead of a spec.

deleted 8y ago
[deleted]

typomatic 8y ago

Worth noting that the PyPy article links to a whole site devoted to the troubles and rehabilitation of the CPython API: https://pythoncapi.readthedocs.io/ https://pythoncapi.readthedocs.io/ . It's really exciting to see these backend concerns being addressed with depth and care, especially against the backdrop of syntactic sugar PEPs blowing up uglily.

antt 8y ago
PyPy is one of the few projects that are doing good work, communicating their work well and making the right strategic decisions. At his rate I won't be surprised if python4 is just PyPy.
- sitkack 8y ago
  If the PyPy team made something that was compatible with both Python2 and Python3, that Python would be that Python.
quotemstr 8y ago
Thanks. That's a good link. The proposal is a good start, although a bit lacking on the concrete design for a new API. I'm not sure there's a big need for innovation here though: JNI basically got it right. It provides full isolation from VM implementation details (especially of memory management) while not sacrificing performance or completeness. JNI has a bad reputation for some reason I've never fully understood, but it seems like an elegant approach to me.

wahern 8y ago

I think the central issue is that PyPy uses a copying GC, which means pointers to objects aren't stable by default. The article implies that this is a consequence of having a generational GC, but you can have a non-copying generational GC, which Lua 5.2 and 5.4 actually have.[1] [1] Lua 5.3 dropped the generational GC because real-world results didn't justify the complexity, but apparently for 5.4 they came up with a better design.

quotemstr 8y ago
Object stability precludes bump-pointer allocation of short-lived temporaries though, and in a very allocation-y language like Python, that's a pretty important.
- weberc2 8y ago
  I wonder if escape analysis could compensate as it does with Go.
  - deleted 8y ago
    [deleted]

Boxxed 8y ago

I don't even think it should be considered an API. The python "API" is just all the interpreter internals.

int_19h 8y ago
The more accurate term would be "object model with an ABI", in the same sense COM or GLib is an object model. So CPython is actually several things layered on top of each other. - PyObject-based object model; this includes PyObject, PyTypeObject, PyUnicodeObject... and I think that's it? This is the equivalent of COM IUnknown. It doesn't actually know anything about Python proper, but it defines the operations in terms of which language itself will later be defined (like the idea that objects have a refcount-centric lifetime, and have attributes, and operations like "call" and "add" etc). - A bunch of standard data structures built on top of that, like PyLongObject and PyListObject. This is just a pure extension of the above - again, defining more terms in which the language is defined, like what happens when you add two ints. - A bunch of specialized data classes which store Python bytecode and provide the framework for its execution, like PyCodeObject, PyFrameObject and PyFunctionObject. Note that these don't know anything about how the bytecode is produced, nor about how to actually execute it. But they do know about things like local variables (so PyCodeObject will store the list of locals, and PyFrameObject will allocate space for them), so Python-the-language starts creeping in here. - The parser which produces AST (which is itself a bunch of Python objects), and the bytecode compiler that produces PyCodeObjects out of that AST. - And finally, the actual interpreter, that ties it all together by providing semantics for the bytecode contained in PyCodeObjects. This layering is even visible in the Python source itself (https://github.com/python/cpython/tree/v3.7.0 https://github.com/python/cpython/tree/v3.7.0) - the first three things live under ./Objects in the source tree, and the parser and the interpreter is under ./Python (with some bits under ./Grammar and ./Parser). So, roughly speaking, ./Objects is the object model, and ./Python is the language proper. The headers are interdependent, unfortunately, but it's not that hard to break them apart if anyone cared to.

dtech 8y ago

It's a common problem because it is so tempting. Exposing the compiler/runtime internals allows very powerful things at minimal compiler developer effort.

nicolaslem 8y ago
This reminds me the case of Firefox: extensions could access almost everything internally which made them very powerful but ultimately made refactoring very hard, hence the switch to web extensions.

dasyatidprime 8y ago

… and then LuaJIT converts started saying “don't ever use the traditional C API, it's so slow and indirect, and the FFI is so easy”, and then anyone not running LuaJIT lost access to a bunch of libraries who stopped caring about the old ways.

kbumsik 8y ago

Never heard of Lua's approach but that sounds interesting. What about overhead between Lua and native? I think the Python C API is one of the reason Python get popular, by doing "prototype in Python then make C codes later for performance" approach. This approach sometime is not desirable in other platforms such as NodeJs because calling a native module from JS is quite costly.

dividuum 8y ago
The overhead is really minimal. I happened to have an old program around that uses LuaJIT or Lua5.1. It calls a two functions that each increments a C int. Using the FFI, calling this function 500 million times takes 0.7 seconds. Calling it as an old-style exported function takes 5.4 seconds for 500 million calls. Disabling the JIT increases the FFI time to 32.7s and decreases the old-style time to 4.7 seconds. On Lua5.1, calling the function takes 6.7s. This is in no way representative of any reach benchmark, but as you can see: Lua is pretty fast.