5 ms·

Is there any chance that a project like RustPython could supplant CPython?

by mikece 4y ago

macspoofing 4y ago
To solve what problem exactly?
- kilgnad 4y ago
  To solve the problem that is C.
  - wiseowise 4y ago
    Any constructive comments or we’re /r/ProgrammerHumor now?
    - kilgnad 4y ago
      it wasn't suppose to be funny. C has specific problems. Rust is a solution to those specific problems.
      - ameliaquining 4y ago
        I think the question is how those problems are applicable specifically to CPython's use of C. I agree that it would be better, all else being equal, if CPython were implemented in a memory-safe language (though it would almost certainly need to use unsafe escape hatches in some places), but I think this'd be more viable if it were done by incrementally migrating the existing codebase with the involvement of the current maintainers, rather than as a third-party rewrite from scratch.
  - v3ss0n 4y ago
    Then there is python written in python called pypy
orangepurple 4y ago
> For your big question about a CPython replacement I think that this is our current goal. I do not think people will be using RustPython in their production any time soon but I do think we can be competitive in WASM. Having said that I do not think we should be 100% compatible with CPython. For example we have chosen to implement threading without the GIL. Being compatible with CPython is very helpful as we can use the documentation and tests of CPython. https://github.com/RustPython/RustPython/issues/1940#issuecomment-635818817 https://github.com/RustPython/RustPython/issues/1940#issueco...
- deleted 4y ago
  [deleted]
- plonk 4y ago
  Not having a GIL makes this very interesting for all production uses. I hope this works out.
  - ketralnis 4y ago
    It also rules out using most C extensions which rely on it. These libraries are most of the draw of the Python ecosystem
bionade24 4y ago
If they support python packages exposing the C ABI and expose a C ABI for embedding, I can't see why not.
- nilsbunger 4y ago
  They're doing threading without the GIL, which means it's not compatible with existing C extensions. CPython is also building a -no-gil compile-time option. If that gets traction, some of the big performance-oriented C extensions like Numpy etc would likely build a non-GIL extension version too. Then, if RustPython's non-GIL implementation is compatible with CPython's, and their C ABI is compatible, those C extensions might work.
  - guipsp 4y ago
    gil or no gil is mostly irrelevant for numpy, an extension that does most work with gil released.
    - cdavid 4y ago
      Relying on the cpython gil and releasing the gil are two different thing: 1. releasing the gil means multithreading is opt-in for a given code section in NumPy. Only very specific parts of the code need to be threadsafe. 2. not relying on a gil in cpython runtime means multithreading becomes opt out. Now all the code by default needs to be threadsafe, including the libs you depend on. A lot of C/C++/Fortran scientific code is not thread safe, and the whole scientific python ecosystem depends heavily on those codebases.
      - guipsp 4y ago
        I understand what you are saying, but I don't see how it's relevant? Numpy doesn't rely on the gil, which is why it releases it.
        cdavid 4y ago
        It releases the GIL only in very specific sections. Most of the Numpy C code runs under the gil. A quick check on master shows only 10-20 calls using NPY_BEGIN_ALLOW_THREADS (which is an alias to Py_BEGIN_ALLOW_THREADS). A lot of the NumPy code manipulates python runtime objects, and doing so without thread safety would likely break everywhere. A lot of efforts would be needed to gradually make large C extension thread safe.
ehutch79 4y ago
Depends... There are already other implementations. But CPython is the reference implementation. That's part of why we're only now seeing optimizations that reduce code clarity for reading. (i think, dont quote me on this)
- dr_zoidberg 4y ago
  No need to quote you, but GvR himself [0]. That's the Faster CPython initiative, which has been ongoing for about 2 or 3 years I think. 3.11 got some nice speedups from it, and 3.12 is on its way to get more. [1] [0] https://github.com/faster-cpython/ideas/blob/main/FasterCPythonDark.pdf https://github.com/faster-cpython/ideas/blob/main/FasterCPyt... [1] https://github.com/faster-cpython/ideas https://github.com/faster-cpython/ideas
loeg 4y ago
Only if the CPython devs will it so.
zitterbewegung 4y ago
TLDR: No The hardest part of supplanting CPython is the fact that the FFI is already validated on a huge amount of implementations. I believe Cinder and Pyston work out of the box but they are on older versions of python but Pyston wants to be merged back into CPython and so does Cinder (or at least what is relevant). On the other hand JIT in CPython can be achieved by other python packages. Pyston has also extracted the JIT and can be added to 3.7 - 3.10 by installing it. See https://pypi.org/project/pyston/ https://pypi.org/project/pyston/ Pypy is a separate reference implementations that has tried to achieve something different but hasn't supplanted python for years. It has had a JIT and the ability to change garbage collectors but the system stuffers when interacting with C FFIs Pyodide is inthe browser so its something completely different but has no support for CFFIs. IronPython (.net), Jython are inactive. Also GIL removal from CPython has been attempted multiple times but now that CPython is being funded by Microsoft to become faster makes it more unlikely that CPython will be supplanted.
- brucethemoose2 4y ago
  GraalVM also has a nascent Python runtime, which could be extremely performant. But I think OS devs dont really want to put optimization work into GraalVM, as their hard worked could get sucked up and locked behind the Enterprise Edition.
  - zitterbewegung 4y ago
    Mainly resource exhaustion from developers makes the other runtimes fade into the background while resources being put into web deployment (pyodide) and microcontrollers (micropython and circuitpython) are a much better idea.
  - noloblo 4y ago
    What's graalvm and where does it fit in with the larger Python ecosystem
    - brucethemoose2 4y ago
      Its a "universal" VM from Oracle that supports many languages and other bells and whistles like AOT compilation of interpreted languages: https://www.graalvm.org/ https://www.graalvm.org/ But its Java support is most prominant, and the its runtime is significantly faster than OpenJDK. Which is why its so promising for Python. I'd argue that OpenJDK is way ahead of the Pyton runtime, and GraalVM is way ahead of OpenJDK. But its also kinda iffy because: - Python support isn't very good now - Oracle is Oracle. Hence they have the better optimizations walled off behind a "Enteprise Edition" registration and license.
mkoubaa 4y ago
With HPy yes

Is there any chance that a project like RustPython could supplant CPython?

by mikece 4y ago

Is there any chance that a project like RustPython could supplant CPython?

macspoofing 4y ago

To solve what problem exactly?

kilgnad 4y ago
To solve the problem that is C.
- wiseowise 4y ago
  Any constructive comments or we’re /r/ProgrammerHumor now?
  - kilgnad 4y ago
    it wasn't suppose to be funny. C has specific problems. Rust is a solution to those specific problems.
    - ameliaquining 4y ago
      I think the question is how those problems are applicable specifically to CPython's use of C. I agree that it would be better, all else being equal, if CPython were implemented in a memory-safe language (though it would almost certainly need to use unsafe escape hatches in some places), but I think this'd be more viable if it were done by incrementally migrating the existing codebase with the involvement of the current maintainers, rather than as a third-party rewrite from scratch.
- v3ss0n 4y ago
  Then there is python written in python called pypy

orangepurple 4y ago

> For your big question about a CPython replacement I think that this is our current goal. I do not think people will be using RustPython in their production any time soon but I do think we can be competitive in WASM. Having said that I do not think we should be 100% compatible with CPython. For example we have chosen to implement threading without the GIL. Being compatible with CPython is very helpful as we can use the documentation and tests of CPython. https://github.com/RustPython/RustPython/issues/1940#issuecomment-635818817 https://github.com/RustPython/RustPython/issues/1940#issueco...

deleted 4y ago
[deleted]
plonk 4y ago
Not having a GIL makes this very interesting for all production uses. I hope this works out.
- ketralnis 4y ago
  It also rules out using most C extensions which rely on it. These libraries are most of the draw of the Python ecosystem

bionade24 4y ago

If they support python packages exposing the C ABI and expose a C ABI for embedding, I can't see why not.

nilsbunger 4y ago
They're doing threading without the GIL, which means it's not compatible with existing C extensions. CPython is also building a -no-gil compile-time option. If that gets traction, some of the big performance-oriented C extensions like Numpy etc would likely build a non-GIL extension version too. Then, if RustPython's non-GIL implementation is compatible with CPython's, and their C ABI is compatible, those C extensions might work.
- guipsp 4y ago
  gil or no gil is mostly irrelevant for numpy, an extension that does most work with gil released.
  - cdavid 4y ago
    Relying on the cpython gil and releasing the gil are two different thing: 1. releasing the gil means multithreading is opt-in for a given code section in NumPy. Only very specific parts of the code need to be threadsafe. 2. not relying on a gil in cpython runtime means multithreading becomes opt out. Now all the code by default needs to be threadsafe, including the libs you depend on. A lot of C/C++/Fortran scientific code is not thread safe, and the whole scientific python ecosystem depends heavily on those codebases.
    - guipsp 4y ago
      I understand what you are saying, but I don't see how it's relevant? Numpy doesn't rely on the gil, which is why it releases it.
      - cdavid 4y ago
        It releases the GIL only in very specific sections. Most of the Numpy C code runs under the gil. A quick check on master shows only 10-20 calls using NPY_BEGIN_ALLOW_THREADS (which is an alias to Py_BEGIN_ALLOW_THREADS). A lot of the NumPy code manipulates python runtime objects, and doing so without thread safety would likely break everywhere. A lot of efforts would be needed to gradually make large C extension thread safe.

ehutch79 4y ago

Depends... There are already other implementations. But CPython is the reference implementation. That's part of why we're only now seeing optimizations that reduce code clarity for reading. (i think, dont quote me on this)

dr_zoidberg 4y ago
No need to quote you, but GvR himself [0]. That's the Faster CPython initiative, which has been ongoing for about 2 or 3 years I think. 3.11 got some nice speedups from it, and 3.12 is on its way to get more. [1] [0] https://github.com/faster-cpython/ideas/blob/main/FasterCPythonDark.pdf https://github.com/faster-cpython/ideas/blob/main/FasterCPyt... [1] https://github.com/faster-cpython/ideas https://github.com/faster-cpython/ideas

loeg 4y ago

Only if the CPython devs will it so.

zitterbewegung 4y ago

TLDR: No The hardest part of supplanting CPython is the fact that the FFI is already validated on a huge amount of implementations. I believe Cinder and Pyston work out of the box but they are on older versions of python but Pyston wants to be merged back into CPython and so does Cinder (or at least what is relevant). On the other hand JIT in CPython can be achieved by other python packages. Pyston has also extracted the JIT and can be added to 3.7 - 3.10 by installing it. See https://pypi.org/project/pyston/ https://pypi.org/project/pyston/ Pypy is a separate reference implementations that has tried to achieve something different but hasn't supplanted python for years. It has had a JIT and the ability to change garbage collectors but the system stuffers when interacting with C FFIs Pyodide is inthe browser so its something completely different but has no support for CFFIs. IronPython (.net), Jython are inactive. Also GIL removal from CPython has been attempted multiple times but now that CPython is being funded by Microsoft to become faster makes it more unlikely that CPython will be supplanted.

brucethemoose2 4y ago
GraalVM also has a nascent Python runtime, which could be extremely performant. But I think OS devs dont really want to put optimization work into GraalVM, as their hard worked could get sucked up and locked behind the Enterprise Edition.
- zitterbewegung 4y ago
  Mainly resource exhaustion from developers makes the other runtimes fade into the background while resources being put into web deployment (pyodide) and microcontrollers (micropython and circuitpython) are a much better idea.
- noloblo 4y ago
  What's graalvm and where does it fit in with the larger Python ecosystem
  - brucethemoose2 4y ago
    Its a "universal" VM from Oracle that supports many languages and other bells and whistles like AOT compilation of interpreted languages: https://www.graalvm.org/ https://www.graalvm.org/ But its Java support is most prominant, and the its runtime is significantly faster than OpenJDK. Which is why its so promising for Python. I'd argue that OpenJDK is way ahead of the Pyton runtime, and GraalVM is way ahead of OpenJDK. But its also kinda iffy because: - Python support isn't very good now - Oracle is Oracle. Hence they have the better optimizations walled off behind a "Enteprise Edition" registration and license.

mkoubaa 4y ago

With HPy yes