3 ms·

> now you lose all the already actually fast third party modules like NumPy, PyTorch, TensorFlow, and what not that all rely on having the internal API to call

> now you lose all the already actually fast third party modules like NumPy, PyTorch, TensorFlow, and what not that all rely on having the internal API to call C, Fortran, and CUDA Do you? Rust can work with C ABIs; why wouldn't it be able to do a passthrough for Python?

nonameiguess 5y ago
It's not an issue of being able to call C code in general, which almost anything can do. The codebases for these modules use CPython API bindings to do it, though, and those won't be available in an alternative interpreter, so you'd need to completely re-write all of these libraries to use different bindings to interface between the underlying BLAS and CUDA libs and your interpreter. Which they can do, but that is a heck of a lot more work than just writing a Python interpreter.
- brundolf 5y ago
  Oh. So the libraries are hardcoded to only work with one specific interpreter, regardless of language? That's unfortunate.
  - rcxdude 5y ago
    Pretty much. In principle rust (or any other language) could recreate the interface but it really limits how you could design the interpreter. You either make the interface really expensive or basically just transliterate the CPython implementation into your language.
liuliu 5y ago
Python's C API basically exposes the interpreter internal objects (these are very well-designed internal objects!). Anyone wants to support C-extension modules for CPython need to either keep their internal representation compatible with CPython's interpreter presentation (which severely constrains what you can do in your new interpreter), or have some kind of translation layer between. See: https://docs.python.org/3.10/c-api/index.html https://docs.python.org/3.10/c-api/index.html and https://www.pypy.org/posts/2018/09/inside-cpyext-why-emulating-cpython-c-8083064623681286567.html https://www.pypy.org/posts/2018/09/inside-cpyext-why-emulati...

nonameiguess 5y ago

It's not an issue of being able to call C code in general, which almost anything can do. The codebases for these modules use CPython API bindings to do it, though, and those won't be available in an alternative interpreter, so you'd need to completely re-write all of these libraries to use different bindings to interface between the underlying BLAS and CUDA libs and your interpreter. Which they can do, but that is a heck of a lot more work than just writing a Python interpreter.

brundolf 5y ago
Oh. So the libraries are hardcoded to only work with one specific interpreter, regardless of language? That's unfortunate.
- rcxdude 5y ago
  Pretty much. In principle rust (or any other language) could recreate the interface but it really limits how you could design the interpreter. You either make the interface really expensive or basically just transliterate the CPython implementation into your language.

liuliu 5y ago

Python's C API basically exposes the interpreter internal objects (these are very well-designed internal objects!). Anyone wants to support C-extension modules for CPython need to either keep their internal representation compatible with CPython's interpreter presentation (which severely constrains what you can do in your new interpreter), or have some kind of translation layer between. See: https://docs.python.org/3.10/c-api/index.html https://docs.python.org/3.10/c-api/index.html and https://www.pypy.org/posts/2018/09/inside-cpyext-why-emulating-cpython-c-8083064623681286567.html https://www.pypy.org/posts/2018/09/inside-cpyext-why-emulati...