6 ms·

Just came in here briefly to opine that there is a very real risk of fork if the Python core community does not at least offer a viable alternative expediently.

Just came in here briefly to opine that there is a very real risk of fork if the Python core community does not at least offer a viable alternative expediently. The economic pressures surrounding the benefits of gross’s changes will likely influence this more than any tears shed over subtle backwards incompatibility. I believe it was Dropbox that famously released their own private internal Python build a while back and included some concurrency patches. Many teams might go the route of working from Sam Gross’ work and if we see subtle changes in underlying runtime concurrency semantics or something else backwards incompatible that’s it- either that adoption will roll downhill to a new standard or Python core will have to answer with a suitable GIL-less alternative. I for one do not want to think about “ANSI Python” runtimes or give the MSFTs etc of the world an opening to divide the user base.

rbanffy 5y ago
> I believe it was Dropbox that famously released their own private internal Python build a while back and included some concurrency patches. Google also had their Unladen Swallow version, but it seems they lost interest at some point.
- liuliu 5y ago
  Unladen Swallow is a 20% project. Besides, LLVM's JIT turns out to be an disappointment and I believe that no modern interpreted language uses LLVM's JIT at this moment.
  - williamstein 5y ago
    Doesn’t Julia?
    - kristofferc 5y ago
      Julia is not interpreted.
      - Siira 5y ago
        What’s the difference? Wouldn’t any language with a JIT not be “interpreted”?
        adgjlsfhk1 5y ago
        Julia isn't a normal jit either. most jits make speculative guesses based on observed patterns and then has to deoptimize occasionally. Julia only compiles based on type information, so in many ways it is closer to running an ahead of time compiler at runtime. because of this, Julia is often called just ahead of time compiled.
        rbanffy 5y ago
        It's not quite interpreted. Python is compiled to bytecode when first loaded (that's what all those .pyc files are) and there is no separate compile step as with Java.
- westurner 5y ago
  "PEP 3146 -- Merging Unladen Swallow into CPython" > Future Work (2010) https://www.python.org/dev/peps/pep-3146/#future-work https://www.python.org/dev/peps/pep-3146/#future-work Perhaps Google/Grumpy could be updated to compile Python 3.x+ to Go with e.g. the RustPython version of the CPython Python Standard Library modules? "Inside cpyext: Why emulating CPython C API is so Hard" (2018) https://news.ycombinator.com/item?id=18040664 https://news.ycombinator.com/item?id=18040664 Today, conda-forge compiles CPython to relocatable platform+architecture-specific binaries with LLVM. https://github.com/conda-forge/python-feedstock/blob/master/recipe/meta.yaml https://github.com/conda-forge/python-feedstock/blob/master/... conda-forge also compiles PyPy Python to relocatable platform+architecture-specific binaries with LLVM. conda-forge/pypy3.6-feedstock (3.7) https://github.com/conda-forge/pypy3.6-feedstock/blob/master/recipe/meta.yaml https://github.com/conda-forge/pypy3.6-feedstock/blob/master... https://github.com/conda-forge/pypy-meta-feedstock/blob/master/recipe/meta.yaml https://github.com/conda-forge/pypy-meta-feedstock/blob/mast... : > summary: Metapackage to select pypy as python implementation Pyodide (JupyterLite) compiles CPython to WASM (or LLVM IR?) with LLVM/emscripten IIRC. Hopefully there's a clear way to implement the new GIL-less multithreading support with Web Workers in WASM, too? The https://rapids.ai/ https://rapids.ai/ org has a bunch a fast Python for HPC and Cloud; with Dask and pick a scheduler. Less process overhead and less need for interprocess locking of memory handles that transgress contexts due to a new GIL removal approach would be even faster than debuggable one process per core Python.
  - adsharma 5y ago
    Grumpy is unmaintained.There is similar work (py2many) that transpiles python3 to 8 different languages. The approach focuses on functional programming, does away with extensions completely. For that approach to be successful, a pure python implementation of stdlib in the transpileable subset of python 3 would be super helpful.
qwerty456127 5y ago
Isn't there enough Python forks/implementations already? Is it reasonable to expect a new one to become more popular than those we already have?
yjftsjthsd-h 5y ago
I mean, PyPy is over a decade old now, and micropython is a mere 7 years old. What's another fork? If anything, I strongly prefer languages that have more than one implementation.
- mvanveen 5y ago
  I do not disagree with you. However, I would speculate part of PSF's hesitancy is likely specifically around the perceived violence that gross' "GIL-less" changes may incur to the runtime semantics' backwards compatibility. PSF in particular has a responsibility here as well I feel in that CPython is arguably the working spec or standard from which these other implementations work and are defined. Do you also strongly prefer languages with different underlying concurrency semantics? While stackless and pypy etc. are around and available and this could suggest the answer could be "yes" we've been lucky that they haven't fundamentally changed the experience of writing Python. The possibility that a ton of libraries might now be able to use efficient multi-threaded execution where they were previously constrained to multiprocessing will be a landslide of changes on its own, and likely reminiscent of python 2 -> 3 compatibility if we have to preserve "two ways of doing things."
  - yjftsjthsd-h 5y ago
    I can certainly agree that multiple implementations has overhead and creates pain points, but it's not world-ending and it does have advantages.
    - mvanveen 5y ago
      I think it’s important to ask “what guarantees are in place to keep multiple implementations consistent and what is the fallout if they are not” as a lens to determining the degree to which it is or isn’t “world-ending” The Python ecosystem once fell apart not too long ago because it was supporting two versions where one moved from “print “ to “print(“ and these were incompatible and broke things such as doc tests. There’s a reason that ppl strongly started advocating to hard pivot to 3: there was a very real chance that Python 2 could fork the ecosystem. Incompatible concurrency semantics would be a much worse can of worms.
wyldfire 5y ago
> there is a very real risk of fork It might not matter much if Canonical or IBM decided to port a critical mass of open source extensions/packages. Then they could ship the new CPython in place of the old one and mention the differences in the release notes. With one or both throwing their weight behind it, it would gain significant momentum above and beyond the original project.
- dralley 5y ago
  > It might not matter much if Canonical or IBM decided to port a critical mass of open source extensions/packages. Then they could ship the new CPython in place of the old one and mention the differences in the release notes. The lifeblood of Canonical and Red Hat / IBM is long term platform support for companies that want their most critical code to not break underneath them. Even if the open source libraries get ported, there are still plenty of proprietary C extensions out there for which this would be a breaking change - the "dark matter" referred to in this post. It would make zero sense for them to "throw their weight around" and _unilaterally_ break their customers' code if even the upstream devs didn't want to go through with the changes.
fulafel 5y ago
There experimental forks don't aim to change language semantics so they are quite safe from fragmentation pov even if they accidentally get some adoption. But they have so far been explicit about being research and uninterested in anything else.

rbanffy 5y ago

> I believe it was Dropbox that famously released their own private internal Python build a while back and included some concurrency patches. Google also had their Unladen Swallow version, but it seems they lost interest at some point.

liuliu 5y ago
Unladen Swallow is a 20% project. Besides, LLVM's JIT turns out to be an disappointment and I believe that no modern interpreted language uses LLVM's JIT at this moment.
- williamstein 5y ago
  Doesn’t Julia?
  - kristofferc 5y ago
    Julia is not interpreted.
    - Siira 5y ago
      What’s the difference? Wouldn’t any language with a JIT not be “interpreted”?
      - adgjlsfhk1 5y ago
        Julia isn't a normal jit either. most jits make speculative guesses based on observed patterns and then has to deoptimize occasionally. Julia only compiles based on type information, so in many ways it is closer to running an ahead of time compiler at runtime. because of this, Julia is often called just ahead of time compiled.
      - rbanffy 5y ago
        It's not quite interpreted. Python is compiled to bytecode when first loaded (that's what all those .pyc files are) and there is no separate compile step as with Java.
westurner 5y ago
"PEP 3146 -- Merging Unladen Swallow into CPython" > Future Work (2010) https://www.python.org/dev/peps/pep-3146/#future-work https://www.python.org/dev/peps/pep-3146/#future-work Perhaps Google/Grumpy could be updated to compile Python 3.x+ to Go with e.g. the RustPython version of the CPython Python Standard Library modules? "Inside cpyext: Why emulating CPython C API is so Hard" (2018) https://news.ycombinator.com/item?id=18040664 https://news.ycombinator.com/item?id=18040664 Today, conda-forge compiles CPython to relocatable platform+architecture-specific binaries with LLVM. https://github.com/conda-forge/python-feedstock/blob/master/recipe/meta.yaml https://github.com/conda-forge/python-feedstock/blob/master/... conda-forge also compiles PyPy Python to relocatable platform+architecture-specific binaries with LLVM. conda-forge/pypy3.6-feedstock (3.7) https://github.com/conda-forge/pypy3.6-feedstock/blob/master/recipe/meta.yaml https://github.com/conda-forge/pypy3.6-feedstock/blob/master... https://github.com/conda-forge/pypy-meta-feedstock/blob/master/recipe/meta.yaml https://github.com/conda-forge/pypy-meta-feedstock/blob/mast... : > summary: Metapackage to select pypy as python implementation Pyodide (JupyterLite) compiles CPython to WASM (or LLVM IR?) with LLVM/emscripten IIRC. Hopefully there's a clear way to implement the new GIL-less multithreading support with Web Workers in WASM, too? The https://rapids.ai/ https://rapids.ai/ org has a bunch a fast Python for HPC and Cloud; with Dask and pick a scheduler. Less process overhead and less need for interprocess locking of memory handles that transgress contexts due to a new GIL removal approach would be even faster than debuggable one process per core Python.
- adsharma 5y ago
  Grumpy is unmaintained.There is similar work (py2many) that transpiles python3 to 8 different languages. The approach focuses on functional programming, does away with extensions completely. For that approach to be successful, a pure python implementation of stdlib in the transpileable subset of python 3 would be super helpful.

qwerty456127 5y ago

Isn't there enough Python forks/implementations already? Is it reasonable to expect a new one to become more popular than those we already have?

yjftsjthsd-h 5y ago

I mean, PyPy is over a decade old now, and micropython is a mere 7 years old. What's another fork? If anything, I strongly prefer languages that have more than one implementation.

mvanveen 5y ago
I do not disagree with you. However, I would speculate part of PSF's hesitancy is likely specifically around the perceived violence that gross' "GIL-less" changes may incur to the runtime semantics' backwards compatibility. PSF in particular has a responsibility here as well I feel in that CPython is arguably the working spec or standard from which these other implementations work and are defined. Do you also strongly prefer languages with different underlying concurrency semantics? While stackless and pypy etc. are around and available and this could suggest the answer could be "yes" we've been lucky that they haven't fundamentally changed the experience of writing Python. The possibility that a ton of libraries might now be able to use efficient multi-threaded execution where they were previously constrained to multiprocessing will be a landslide of changes on its own, and likely reminiscent of python 2 -> 3 compatibility if we have to preserve "two ways of doing things."
- yjftsjthsd-h 5y ago
  I can certainly agree that multiple implementations has overhead and creates pain points, but it's not world-ending and it does have advantages.
  - mvanveen 5y ago
    I think it’s important to ask “what guarantees are in place to keep multiple implementations consistent and what is the fallout if they are not” as a lens to determining the degree to which it is or isn’t “world-ending” The Python ecosystem once fell apart not too long ago because it was supporting two versions where one moved from “print “ to “print(“ and these were incompatible and broke things such as doc tests. There’s a reason that ppl strongly started advocating to hard pivot to 3: there was a very real chance that Python 2 could fork the ecosystem. Incompatible concurrency semantics would be a much worse can of worms.

wyldfire 5y ago

> there is a very real risk of fork It might not matter much if Canonical or IBM decided to port a critical mass of open source extensions/packages. Then they could ship the new CPython in place of the old one and mention the differences in the release notes. With one or both throwing their weight behind it, it would gain significant momentum above and beyond the original project.

dralley 5y ago
> It might not matter much if Canonical or IBM decided to port a critical mass of open source extensions/packages. Then they could ship the new CPython in place of the old one and mention the differences in the release notes. The lifeblood of Canonical and Red Hat / IBM is long term platform support for companies that want their most critical code to not break underneath them. Even if the open source libraries get ported, there are still plenty of proprietary C extensions out there for which this would be a breaking change - the "dark matter" referred to in this post. It would make zero sense for them to "throw their weight around" and _unilaterally_ break their customers' code if even the upstream devs didn't want to go through with the changes.

fulafel 5y ago

There experimental forks don't aim to change language semantics so they are quite safe from fragmentation pov even if they accidentally get some adoption. But they have so far been explicit about being research and uninterested in anything else.