29 ms·

Our Plan for Python 3.13

531 points by bratao 3y ago

klooney 3y ago
This is exciting. Feels a little like JavaScript 10 years ago.
- andai 3y ago
  You mean 15? ;) https://en.wikipedia.org/wiki/V8_(JavaScript_engine) https://en.wikipedia.org/wiki/V8_(JavaScript_engine)
deleted 3y ago
[deleted]
samwillis 3y ago
The "Faster Python" team are doing a fantastic job, it's incredible to see the progress they are making. However, there is also a bit of a struggle going on between them and the project to remove the GIL (global interpreter lock) from CPython. There is going to be a performance impact on single threaded code if the "no GIL" project is merged, something in the region of 10%. It seems that the faster Python devs are pushing back against that as it impacts their own goals. Their argument is that the "sub interpreters" they are adding (each with its own GIL) will fulfil the same use cases of multithreaded code without a GIL, but they still have the overhead of encoding and passing data in the same way you have to with sub processors. There is also the argument that it could "divide the community" as some C extensions may not be ported to the new ABI that the no GIL project will result in. However again I'm unconvinced by that, the Python community has been through worse (Python 3) and even asyncIO completely divides the community now. It's somewhat unfortunate that this internal battle is happening, both projects are incredible and will push the language forward. Once the GIL has been removed it opens us all sorts of interesting opportunities for new concurrency APIs that could enable making concurrent code much easer to write. My observation is that the Faster Python team are better placed politicly, they have GVR on the team, whereas No GIL is being proposed by an "outsider". It just smells a little of NIH syndrome.
- bratao 3y ago
  Yeah, they clearly stated that here https://discuss.python.org/t/pep-703-making-the-global-interpreter-lock-optional-3-12-updates/26503/118 https://discuss.python.org/t/pep-703-making-the-global-inter... I really wish that GIL go away. It is better to pay this price now, multi-threading is the future.
  - hospitalJail 3y ago
    I'm so conflicted. ~40 of the programs I am responsible for are single threaded. They were relatively quick to develop and were made by Electrical Engineers rather than career/degreed programmers. 2 programs use multithreading, I had to do that. The learning curve was not a huge deal, but the development time adds at least hours. In my case days(due to testing). I imagine its too hard to have an optional flag at the start of each program that can let the user decide?
    - birdyrooster 3y ago
      The problem is that Python types are not thread safe so you have to jump through more hoops to have safe parallelization in Python. These changes would make it so that writing multithreaded code will be much easier it seems like.
    - maleldil 3y ago
      > I imagine its too hard to have an optional flag at the start of each program that can let the user decide? Adding nogil would mean deep changes to the interpreter. I imagine maintaining both versions would be almost like forking the project.
    - coldtea 3y ago
      >but the development time adds at least hours So? some "hours of development time" is nothing.
      - sfink 3y ago
        True, and as you get better at it, those hours will be fewer. The weeks of debugging never go away, though. (Or at least, not as long as you're using shared state, but that's really the only thing under consideration here.)
henrydark 3y ago
Looks cool. This caught my eye > C++ and Java developers expect to be able to run a program at full speed (or very close to it) under a debugger. I haven't worked in Java too much, but in C++ I don't remember having this expectation
- Chabsff 3y ago
  Think of it more as: It's still possible, painful as it may sometimes be, to debug optimised C++ code running at full speed.
  - qsdf38100 3y ago
    Well, yes, but it quickly gets as painful as debugging raw assembly with almost no symbol informations. You might get lucky from time to time and get some debug info available for some variables, but in hot loops and highly inlined code, you're back to raw assembly. Doable for sure, but 100x slower to figure out anything compared to non-optimized code.
- mm007emko 3y ago
  Java programs might not run in debugger very well either, depends on where and how you place breakpoints. However I'd be glad if profilers (and notably memory profiler) would slow down a Python program only as much as valgrind does C.
  - The_Colonel 3y ago
    I noticed that java with a connected debugger can be very fast even with breakpoints, but stepping over can be very slow. Which is a big weird, since "step over" is basically just putting the breakpoint on the next line.
gyrovagueGeist 3y ago
Can someone give a good argument of why subinterpreters are an interesting or useful solution to concurrency in Python? It seems like all the drawbacks of multiprocessing (high memory use, slow serialized object copying) with little benefit and higher complexity for the user. The nogil effort seems like such a better solution, that even if it breaks the C interface, subinterpreters aren't worth considering.
- bratao 3y ago
  I understand that sub-interpreter can perform better on object copying as it is sharing the same memory. But yeah, nogil looks like the correct way.
- samus 3y ago
  Easy interop with C is one of the core selling points of Python. It's not just about performance - it's about Python being able to be a glue language that can interface with any legacy or system library written in C or being accessible via FFI. A LOT of systems exist that take advantage of this, and changes to the C interface impact all of these. Unless the porting story is thought out extraordinary well, it would be another disaster like Python 2->3. Since this affects C code, it won't be anything but tricky.
  - gyrovagueGeist 3y ago
    From reading the nogil repo and related PEP, C ABI breaking does not seem to be the worst problem. An updated Cython, Swig, etc seems like it would be enough in most cases to get something running. More extreme cases might only have ~dozen LoC changes to replace the GIL Ensure/Release patterns. The hidden really hard problem is that extension modules may have been written relying on GIL behavior for thread safety... these may be undocumented and unmaintained. Even so I hope the community decides it is worth it. A glue language with actual MT support would be much more useful.
    - samus 3y ago
      Indeed, thanks for clarifying that. I actually had the extensions in mind when I wrote my comment, but got somehow distracted by the C interop aspect. Extensions indeed have a more intimate relation with the interpreter and (directly or indirectly) rely on the GIL and its associated semantics.
phkahler 3y ago
From the pip on subinterpreters: >> In this case, multiple interpreter support provide a novel concurrency model focused on isolated threads of execution. Furthermore, they provide an opportunity for changes in CPython that will allow simultaneous use of multiple CPU cores (currently prevented by the GIL–see PEP 684). This whole thing seems more like something developer wants to do (hey, it's novel!) than something users want. To the extent they do want it, removing the GIL is probably preferable IMHO. That a global lock is a sacred cow in 2023 seems strange to me. Maybe I'm misunderstanding, but I don't want an API to manage interpreters any more than I want one for managing my CPU cores. Just get me unhindered threads ;-)
- mkoubaa 3y ago
  Most typical application developers don't want, but that doesn't mean it doesn't have very specific important use cases. For example, an application might embed python in memory and need isolated contexts. Or an http server needs to have multiple threads of concurrent work with minimal overhead
jonnycomputer 3y ago
I'm going to admit that what I really want to see is a strong push to standardize and fully incorporate package management and distribution into python's core. Despite the work done on it, it's still a mess as far as I can see, and there is no single source of truth (that I know of) on how to do it. For that matter, pip can't even search for packages any more, and instead directs you to browse the pypi website in a browser. Whatever the technical reasons for that, its a user interface fail. Conda can do it!!!!! (as well as just about any package management system I've ever used)
- asylteltine 3y ago
  Python’s dependency management, or lack there of, its import system, and the lack of strong typing really make me hate it. It’s the first language I really felt adept with, but once I learned Go, I never looked back. Every time I have to use python it’s like coding with crayons.
  - Aperocky 3y ago
    Premature optimization is root of all evil. Python isn't perfect, and mostly unsuitable for any system where performance is in consideration. That leaves everything else including personal utility scripts and packages I use each day to automate random stuff. And I hugely appreciate how fast and simple it is to develop in python, unlike certain languages that literally depends on IDEs due to the verbosity and unnecessary cognitive load.
    - switch007 3y ago
      > And I hugely appreciate how fast and simple it is to develop in python Indeed it’s crazy. A few pip installs and I had a multiprocessing pandas (dask) with a web gui, and a workflow system (also with a web gui), and a pipeline to convert csv to parquet in like 20 lines of code
  - pnt12 3y ago
    I think type hints help a lot. A codebase with classes and type hints reads much better than one using ad-hoc dictionaries for every data structure. But on the other hand I don't really like Go, so maybe it's different languages for different tastes.
orbisvicis 3y ago
How multithreaded programs depend on the GIL rather than use proper locking? The No GIL project might break a lot of multithreaded apps.
- gyrovagueGeist 3y ago
  It would require no changes from pure Python apps (as the nogil implementation preserves the current behavior & thread safety of python objects). But any C extensions that rely on the GIL for thread safety would be an issue..
  - avidphantasm 3y ago
    They must've explored this, but couldn't one make a GIL emulator/wrapper for legacy extensions so that they can still work in GIL-less Python while they are (hopefully) updated to work with the new synchronization primitives?
cellularmitosis 3y ago
> Experiments on the register machine machine are complete This is a worthwhile watch on this topic: https://youtu.be/cMMAGIefZuM https://youtu.be/cMMAGIefZuM
- kzrdude 3y ago
  faster-cpython team has done a lot of work to experiment on it: https://github.com/faster-cpython/ideas/issues/485#issuecomment-1402781915 https://github.com/faster-cpython/ideas/issues/485#issuecomm... I wonder if the plan written there is current.
tasubotadas 3y ago
> PEP 669 – Low Impact Monitoring for CPython Finally, 15 years later Python might also get something similar to VisualVM.
deleted 3y ago
[deleted]
sproketboy 3y ago
[dead]
milliams 3y ago
Notably yesterday Guido posted an update on the Faster Python plans w.r.t. the no-GIL plans (https://discuss.python.org/t/pep-703-making-the-global-interpreter-lock-optional-3-12-updates/26503/118 https://discuss.python.org/t/pep-703-making-the-global-inter...) specifically: Our ultimate goal is to integrate a JIT into CPython
- avidphantasm 3y ago
  Yeah, reading CPython 3.13 plans made it seem like they were backing their way into creating a JIT compiler.
samsquire 3y ago
This is extremely exciting stuff. Thank you for Python and working on improving its performance. I use Python threads for some IO heavy work with Popen and C and Java for my true multithreading work.
retskrad 3y ago
Speaking of Python, as a beginner I have tried to grasp Classes and Objects and watched countless YouTube videos and Reddit/StackOverflow comments to understand them better but I'm just banging my head against the wall and it's not sticking. The whole __init__ method, the concept of self and why it's used, instance and class variables, arguments, all of it. When learning something, when you simply cannot grasp a concept however hard you try, what's the course of action? I have tried taking breaks but that's not helping.
- switch007 3y ago
  Classes and objects exist in all object oriented program languages. Perhaps stepping back from Python and trying some more generic material might help?
  - retskrad 3y ago
    That's a good idea, I'll try that
- harrelchris 3y ago
  Either keep looking for explanations until it clicks, or try to break it down into more fundamental elements. Classes and blueprints have a simple analogy that I'm sure you are familiar with. Blueprints are instructions for how to build something, like a house. Once built, the house is a physical thing you can interact with. It has attributes, such as a height or color. It has things it can do or that can be done to it, like open a door or turn on some lights. Classes are blueprints - they tell a computer how to build something. An object is what is built by the class - it has attributes and methods.
  - retskrad 3y ago
    Thanks, I appreciate it
- ozzy6009 3y ago
  I was the same way when I was a beginner, I didn't really "get" python classes until years later and using other programming languages. I recommend SICP if you want a more first principles understanding: https://web.mit.edu/6.001/6.037/sicp.pdf https://web.mit.edu/6.001/6.037/sicp.pdf
Alifatisk 3y ago
Ruby team approached parallelism & multicore in a cool way, they introduced the concept of the Actor model (like Erlang)!
bkovacev 3y ago
How would this (or any of the recent changes) affect popular web frameworks like Django / Flask / FastAPI? Would this increase performance in serialization or speed in general?
- mkoubaa 3y ago
  Subinterpreters could make parallelism better for independent http server queries
pmoriarty 3y ago
Wish python would sort out their packages mess and let there be one and only one (obvious) way to do it.
kortex 3y ago
Every single HN thread on python performance, ever: Person with limited to zero experience with CPython internals> I hate the GIL, why don't they just remove it? That just is doing an incredible amount of heavy lifting. It'd be like saying, "Why doesn't the USA just switch entirely to the metric system?" It's a huge ask, and after being burned by the 2/3 transition, the python team is loathe to rock to boat too much again. The GIL is deep, arguably one of the deepest abstractions in the codebase, up there with PyObject itself. Think about having to redefine the String implementation of a codebase in your language of choice. Whatever your monday-morning quarterback idea on how to pull out the GIL is, I can almost guarantee, someone has thought about it, probably implemented it, and determined it will have one or more side effects: - Reduced single-thread performance - Breaking ABI/FFI compatibility - Not breaking ABI immediately, but massively introducing the risk of hard to track concurrency bugs that didn't exist before, because GIL - Creating a maintenance nightmare by adding tons of complexity, switches, conditionals, etc to the codebase The team has already decided those tradeoffs are not really justifiable. The GILectomy project has probably come the closest, but it impacts single-thread performance by introducing a mutex (there are some reference type tricks [1] to mitigate the hit, but it still hurts run time 10-50%), and necessitates any extension libraries update their code in a strongly API-breaking way. It's possible that over time, there are improvements which simultaneously improve performance or maintainability, while also lessening the pain of a future GILectomy, e.g. anything which reduces the reliance on checking reference counts. [1] PEP 683 is probably a prerequisite for any future GILectomy attempts, which looks like it has been accepted, which is great https://peps.python.org/pep-0683/ https://peps.python.org/pep-0683/
- arp242 3y ago
  There were already patches to "just" remove the GIL for Python 2.0 or thereabouts, over 20 years ago. Strictly technically speaking it's not even that hard, but as you mentioned it comes with all sorts of trade-offs. Today Python is one of the world's most popular programming language, if not the most popular language. The GIL is limiting and a trade-off in itself of course – designing any programming language is an exercise in trade-offs. But clearly Python has gotten quite far with the set of trade-offs it has chosen: you need to be careful to "just" radically change the set of trade-offs that have proven themselves to be quite successful.
titzer 3y ago
Building a whole new interpreter and accompanying compiler tiers is a lot of work, still in 2023. Many different projects have tried to make this easier, provide reusable components, to offer toolkits, or another VM to build on top of. But it seems that none of these really apply, we're still at the "every language implementation is a special snowflake" territory. That's partly the case in Python just because the codebase has accumulated, rather nucleated, around a core C interpreter from 30 years ago. IMHO the Python community has intentionally torpedoed all competing implementations of Python besides CPython to its own detriment. They seem to constantly make the task of making a compatible VM harder and harder, on purpose. The task they face now is basically building a new, sophisticated VM inside a very much non-sophisticated VM with a ton of legacy baggage. A massive task.
dboreham 3y ago
Just add some PR spin like JS did and declare the GIL a feature.
- voxelghost 3y ago
  Well, if we want to be charitable; - The GIL is a feature that adds value, and was added for a reason. (As explained elsewhere in discussion) And, if we want to be a bit less charitable about it; - The GIL is a feature, in much the same way training-wheels on your bike is a feature. :-)
  - kzrdude 3y ago
    Some people need a three-wheeled motorcycle and for them a two-wheeled one is not an "upgrade". So yes, clearly different tools for different tasks.
PeterStuer 3y ago
Just one question: Do you realy want Python to be held back by the GIL 10 years from now? If not, when do you want to start the change? If so, what do you think a CPU will look like in 10 years, and how would a GIL Python facilitate getting value from it?
- kzrdude 3y ago
  You don't want python to still be reeling from grueling the gil ecosystem split, 10 years from now.
RandyRanderson 3y ago
Many projects start out in Python b/c often new libs are python-first. Many of those run into performance issues and eventually determine that Python will never be fast b/c of the GIL. I think it's very magnanimous of the python team, by not removing the GIL, to give Go, Java and C++ a chance.
fb03 3y ago
I'm actually rooting for RustPython to reach a level of maturity that we'd just be able to ship apis and stuff with it.... https://github.com/RustPython/RustPython https://github.com/RustPython/RustPython
- redsaber 3y ago
  question: If RustPython did indeed reach maturity, are we going to need to choose which python interpreter to install on PC? or like on certain python version we will switch to the new interpreter?
  - kzrdude 3y ago
    CPython developers are still the leaders and those controlling the important Python infrastructure. You have to ask them. It's quite likely RustPython has no influence whatsoever on the question, and also quite likely that CPython is not going to ever throw out its existing work in favour of a different implementation. FWIW, parallel implementations already exist, and maybe RustPython can join them.
fwungy 3y ago
The beauty of the GIL was that it made writing good python libraries easy, from the utility perspective. Python is such an amazing multi-tool because of its rich library. Personally I'd like to see Python continue as the amazing multitool that it is has always been. Go is the natural concurrency sibling for Python. High performance parts of Python applications should be linked to Go binaries, or simply be written entirely in Go. Ideally Go linkage would be highly integrated into Python. There is a huge place in the world for GIL constrained Python. Enabling Python to push further into high performance environments is just going to make a problem worse.
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]

531 points by bratao 3y ago

klooney 3y ago

This is exciting. Feels a little like JavaScript 10 years ago.

andai 3y ago
You mean 15? ;) https://en.wikipedia.org/wiki/V8_(JavaScript_engine) https://en.wikipedia.org/wiki/V8_(JavaScript_engine)

deleted 3y ago

[deleted]

samwillis 3y ago

The "Faster Python" team are doing a fantastic job, it's incredible to see the progress they are making. However, there is also a bit of a struggle going on between them and the project to remove the GIL (global interpreter lock) from CPython. There is going to be a performance impact on single threaded code if the "no GIL" project is merged, something in the region of 10%. It seems that the faster Python devs are pushing back against that as it impacts their own goals. Their argument is that the "sub interpreters" they are adding (each with its own GIL) will fulfil the same use cases of multithreaded code without a GIL, but they still have the overhead of encoding and passing data in the same way you have to with sub processors. There is also the argument that it could "divide the community" as some C extensions may not be ported to the new ABI that the no GIL project will result in. However again I'm unconvinced by that, the Python community has been through worse (Python 3) and even asyncIO completely divides the community now. It's somewhat unfortunate that this internal battle is happening, both projects are incredible and will push the language forward. Once the GIL has been removed it opens us all sorts of interesting opportunities for new concurrency APIs that could enable making concurrent code much easer to write. My observation is that the Faster Python team are better placed politicly, they have GVR on the team, whereas No GIL is being proposed by an "outsider". It just smells a little of NIH syndrome.

bratao 3y ago
Yeah, they clearly stated that here https://discuss.python.org/t/pep-703-making-the-global-interpreter-lock-optional-3-12-updates/26503/118 https://discuss.python.org/t/pep-703-making-the-global-inter... I really wish that GIL go away. It is better to pay this price now, multi-threading is the future.
- hospitalJail 3y ago
  I'm so conflicted. ~40 of the programs I am responsible for are single threaded. They were relatively quick to develop and were made by Electrical Engineers rather than career/degreed programmers. 2 programs use multithreading, I had to do that. The learning curve was not a huge deal, but the development time adds at least hours. In my case days(due to testing). I imagine its too hard to have an optional flag at the start of each program that can let the user decide?
  - birdyrooster 3y ago
    The problem is that Python types are not thread safe so you have to jump through more hoops to have safe parallelization in Python. These changes would make it so that writing multithreaded code will be much easier it seems like.
  - maleldil 3y ago
    > I imagine its too hard to have an optional flag at the start of each program that can let the user decide? Adding nogil would mean deep changes to the interpreter. I imagine maintaining both versions would be almost like forking the project.
  - coldtea 3y ago
    >but the development time adds at least hours So? some "hours of development time" is nothing.
    - sfink 3y ago
      True, and as you get better at it, those hours will be fewer. The weeks of debugging never go away, though. (Or at least, not as long as you're using shared state, but that's really the only thing under consideration here.)

henrydark 3y ago

Looks cool. This caught my eye > C++ and Java developers expect to be able to run a program at full speed (or very close to it) under a debugger. I haven't worked in Java too much, but in C++ I don't remember having this expectation

Chabsff 3y ago
Think of it more as: It's still possible, painful as it may sometimes be, to debug optimised C++ code running at full speed.
- qsdf38100 3y ago
  Well, yes, but it quickly gets as painful as debugging raw assembly with almost no symbol informations. You might get lucky from time to time and get some debug info available for some variables, but in hot loops and highly inlined code, you're back to raw assembly. Doable for sure, but 100x slower to figure out anything compared to non-optimized code.
mm007emko 3y ago
Java programs might not run in debugger very well either, depends on where and how you place breakpoints. However I'd be glad if profilers (and notably memory profiler) would slow down a Python program only as much as valgrind does C.
- The_Colonel 3y ago
  I noticed that java with a connected debugger can be very fast even with breakpoints, but stepping over can be very slow. Which is a big weird, since "step over" is basically just putting the breakpoint on the next line.

gyrovagueGeist 3y ago

Can someone give a good argument of why subinterpreters are an interesting or useful solution to concurrency in Python? It seems like all the drawbacks of multiprocessing (high memory use, slow serialized object copying) with little benefit and higher complexity for the user. The nogil effort seems like such a better solution, that even if it breaks the C interface, subinterpreters aren't worth considering.

bratao 3y ago
I understand that sub-interpreter can perform better on object copying as it is sharing the same memory. But yeah, nogil looks like the correct way.
samus 3y ago
Easy interop with C is one of the core selling points of Python. It's not just about performance - it's about Python being able to be a glue language that can interface with any legacy or system library written in C or being accessible via FFI. A LOT of systems exist that take advantage of this, and changes to the C interface impact all of these. Unless the porting story is thought out extraordinary well, it would be another disaster like Python 2->3. Since this affects C code, it won't be anything but tricky.
- gyrovagueGeist 3y ago
  From reading the nogil repo and related PEP, C ABI breaking does not seem to be the worst problem. An updated Cython, Swig, etc seems like it would be enough in most cases to get something running. More extreme cases might only have ~dozen LoC changes to replace the GIL Ensure/Release patterns. The hidden really hard problem is that extension modules may have been written relying on GIL behavior for thread safety... these may be undocumented and unmaintained. Even so I hope the community decides it is worth it. A glue language with actual MT support would be much more useful.
  - samus 3y ago
    Indeed, thanks for clarifying that. I actually had the extensions in mind when I wrote my comment, but got somehow distracted by the C interop aspect. Extensions indeed have a more intimate relation with the interpreter and (directly or indirectly) rely on the GIL and its associated semantics.

phkahler 3y ago

From the pip on subinterpreters: >> In this case, multiple interpreter support provide a novel concurrency model focused on isolated threads of execution. Furthermore, they provide an opportunity for changes in CPython that will allow simultaneous use of multiple CPU cores (currently prevented by the GIL–see PEP 684). This whole thing seems more like something developer wants to do (hey, it's novel!) than something users want. To the extent they do want it, removing the GIL is probably preferable IMHO. That a global lock is a sacred cow in 2023 seems strange to me. Maybe I'm misunderstanding, but I don't want an API to manage interpreters any more than I want one for managing my CPU cores. Just get me unhindered threads ;-)

mkoubaa 3y ago
Most typical application developers don't want, but that doesn't mean it doesn't have very specific important use cases. For example, an application might embed python in memory and need isolated contexts. Or an http server needs to have multiple threads of concurrent work with minimal overhead

jonnycomputer 3y ago

I'm going to admit that what I really want to see is a strong push to standardize and fully incorporate package management and distribution into python's core. Despite the work done on it, it's still a mess as far as I can see, and there is no single source of truth (that I know of) on how to do it. For that matter, pip can't even search for packages any more, and instead directs you to browse the pypi website in a browser. Whatever the technical reasons for that, its a user interface fail. Conda can do it!!!!! (as well as just about any package management system I've ever used)

asylteltine 3y ago
Python’s dependency management, or lack there of, its import system, and the lack of strong typing really make me hate it. It’s the first language I really felt adept with, but once I learned Go, I never looked back. Every time I have to use python it’s like coding with crayons.
- Aperocky 3y ago
  Premature optimization is root of all evil. Python isn't perfect, and mostly unsuitable for any system where performance is in consideration. That leaves everything else including personal utility scripts and packages I use each day to automate random stuff. And I hugely appreciate how fast and simple it is to develop in python, unlike certain languages that literally depends on IDEs due to the verbosity and unnecessary cognitive load.
  - switch007 3y ago
    > And I hugely appreciate how fast and simple it is to develop in python Indeed it’s crazy. A few pip installs and I had a multiprocessing pandas (dask) with a web gui, and a workflow system (also with a web gui), and a pipeline to convert csv to parquet in like 20 lines of code
- pnt12 3y ago
  I think type hints help a lot. A codebase with classes and type hints reads much better than one using ad-hoc dictionaries for every data structure. But on the other hand I don't really like Go, so maybe it's different languages for different tastes.

orbisvicis 3y ago

How multithreaded programs depend on the GIL rather than use proper locking? The No GIL project might break a lot of multithreaded apps.

gyrovagueGeist 3y ago
It would require no changes from pure Python apps (as the nogil implementation preserves the current behavior & thread safety of python objects). But any C extensions that rely on the GIL for thread safety would be an issue..
- avidphantasm 3y ago
  They must've explored this, but couldn't one make a GIL emulator/wrapper for legacy extensions so that they can still work in GIL-less Python while they are (hopefully) updated to work with the new synchronization primitives?

cellularmitosis 3y ago

> Experiments on the register machine machine are complete This is a worthwhile watch on this topic: https://youtu.be/cMMAGIefZuM https://youtu.be/cMMAGIefZuM

kzrdude 3y ago
faster-cpython team has done a lot of work to experiment on it: https://github.com/faster-cpython/ideas/issues/485#issuecomment-1402781915 https://github.com/faster-cpython/ideas/issues/485#issuecomm... I wonder if the plan written there is current.

tasubotadas 3y ago

> PEP 669 – Low Impact Monitoring for CPython Finally, 15 years later Python might also get something similar to VisualVM.

deleted 3y ago

[deleted]

sproketboy 3y ago

[dead]

milliams 3y ago

Notably yesterday Guido posted an update on the Faster Python plans w.r.t. the no-GIL plans (https://discuss.python.org/t/pep-703-making-the-global-interpreter-lock-optional-3-12-updates/26503/118 https://discuss.python.org/t/pep-703-making-the-global-inter...) specifically: Our ultimate goal is to integrate a JIT into CPython

avidphantasm 3y ago
Yeah, reading CPython 3.13 plans made it seem like they were backing their way into creating a JIT compiler.

samsquire 3y ago

This is extremely exciting stuff. Thank you for Python and working on improving its performance. I use Python threads for some IO heavy work with Popen and C and Java for my true multithreading work.

retskrad 3y ago

Speaking of Python, as a beginner I have tried to grasp Classes and Objects and watched countless YouTube videos and Reddit/StackOverflow comments to understand them better but I'm just banging my head against the wall and it's not sticking. The whole __init__ method, the concept of self and why it's used, instance and class variables, arguments, all of it. When learning something, when you simply cannot grasp a concept however hard you try, what's the course of action? I have tried taking breaks but that's not helping.

switch007 3y ago
Classes and objects exist in all object oriented program languages. Perhaps stepping back from Python and trying some more generic material might help?
- retskrad 3y ago
  That's a good idea, I'll try that
harrelchris 3y ago
Either keep looking for explanations until it clicks, or try to break it down into more fundamental elements. Classes and blueprints have a simple analogy that I'm sure you are familiar with. Blueprints are instructions for how to build something, like a house. Once built, the house is a physical thing you can interact with. It has attributes, such as a height or color. It has things it can do or that can be done to it, like open a door or turn on some lights. Classes are blueprints - they tell a computer how to build something. An object is what is built by the class - it has attributes and methods.
- retskrad 3y ago
  Thanks, I appreciate it
ozzy6009 3y ago
I was the same way when I was a beginner, I didn't really "get" python classes until years later and using other programming languages. I recommend SICP if you want a more first principles understanding: https://web.mit.edu/6.001/6.037/sicp.pdf https://web.mit.edu/6.001/6.037/sicp.pdf

Alifatisk 3y ago

Ruby team approached parallelism & multicore in a cool way, they introduced the concept of the Actor model (like Erlang)!

bkovacev 3y ago

How would this (or any of the recent changes) affect popular web frameworks like Django / Flask / FastAPI? Would this increase performance in serialization or speed in general?

mkoubaa 3y ago
Subinterpreters could make parallelism better for independent http server queries

pmoriarty 3y ago

Wish python would sort out their packages mess and let there be one and only one (obvious) way to do it.

kortex 3y ago

Every single HN thread on python performance, ever: Person with limited to zero experience with CPython internals> I hate the GIL, why don't they just remove it? That just is doing an incredible amount of heavy lifting. It'd be like saying, "Why doesn't the USA just switch entirely to the metric system?" It's a huge ask, and after being burned by the 2/3 transition, the python team is loathe to rock to boat too much again. The GIL is deep, arguably one of the deepest abstractions in the codebase, up there with PyObject itself. Think about having to redefine the String implementation of a codebase in your language of choice. Whatever your monday-morning quarterback idea on how to pull out the GIL is, I can almost guarantee, someone has thought about it, probably implemented it, and determined it will have one or more side effects: - Reduced single-thread performance - Breaking ABI/FFI compatibility - Not breaking ABI immediately, but massively introducing the risk of hard to track concurrency bugs that didn't exist before, because GIL - Creating a maintenance nightmare by adding tons of complexity, switches, conditionals, etc to the codebase The team has already decided those tradeoffs are not really justifiable. The GILectomy project has probably come the closest, but it impacts single-thread performance by introducing a mutex (there are some reference type tricks [1] to mitigate the hit, but it still hurts run time 10-50%), and necessitates any extension libraries update their code in a strongly API-breaking way. It's possible that over time, there are improvements which simultaneously improve performance or maintainability, while also lessening the pain of a future GILectomy, e.g. anything which reduces the reliance on checking reference counts. [1] PEP 683 is probably a prerequisite for any future GILectomy attempts, which looks like it has been accepted, which is great https://peps.python.org/pep-0683/ https://peps.python.org/pep-0683/

arp242 3y ago
There were already patches to "just" remove the GIL for Python 2.0 or thereabouts, over 20 years ago. Strictly technically speaking it's not even that hard, but as you mentioned it comes with all sorts of trade-offs. Today Python is one of the world's most popular programming language, if not the most popular language. The GIL is limiting and a trade-off in itself of course – designing any programming language is an exercise in trade-offs. But clearly Python has gotten quite far with the set of trade-offs it has chosen: you need to be careful to "just" radically change the set of trade-offs that have proven themselves to be quite successful.

titzer 3y ago

Building a whole new interpreter and accompanying compiler tiers is a lot of work, still in 2023. Many different projects have tried to make this easier, provide reusable components, to offer toolkits, or another VM to build on top of. But it seems that none of these really apply, we're still at the "every language implementation is a special snowflake" territory. That's partly the case in Python just because the codebase has accumulated, rather nucleated, around a core C interpreter from 30 years ago. IMHO the Python community has intentionally torpedoed all competing implementations of Python besides CPython to its own detriment. They seem to constantly make the task of making a compatible VM harder and harder, on purpose. The task they face now is basically building a new, sophisticated VM inside a very much non-sophisticated VM with a ton of legacy baggage. A massive task.

dboreham 3y ago

Just add some PR spin like JS did and declare the GIL a feature.

voxelghost 3y ago
Well, if we want to be charitable; - The GIL is a feature that adds value, and was added for a reason. (As explained elsewhere in discussion) And, if we want to be a bit less charitable about it; - The GIL is a feature, in much the same way training-wheels on your bike is a feature. :-)
- kzrdude 3y ago
  Some people need a three-wheeled motorcycle and for them a two-wheeled one is not an "upgrade". So yes, clearly different tools for different tasks.

PeterStuer 3y ago

Just one question: Do you realy want Python to be held back by the GIL 10 years from now? If not, when do you want to start the change? If so, what do you think a CPU will look like in 10 years, and how would a GIL Python facilitate getting value from it?

kzrdude 3y ago
You don't want python to still be reeling from grueling the gil ecosystem split, 10 years from now.

RandyRanderson 3y ago

Many projects start out in Python b/c often new libs are python-first. Many of those run into performance issues and eventually determine that Python will never be fast b/c of the GIL. I think it's very magnanimous of the python team, by not removing the GIL, to give Go, Java and C++ a chance.

fb03 3y ago

I'm actually rooting for RustPython to reach a level of maturity that we'd just be able to ship apis and stuff with it.... https://github.com/RustPython/RustPython https://github.com/RustPython/RustPython

redsaber 3y ago
question: If RustPython did indeed reach maturity, are we going to need to choose which python interpreter to install on PC? or like on certain python version we will switch to the new interpreter?
- kzrdude 3y ago
  CPython developers are still the leaders and those controlling the important Python infrastructure. You have to ask them. It's quite likely RustPython has no influence whatsoever on the question, and also quite likely that CPython is not going to ever throw out its existing work in favour of a different implementation. FWIW, parallel implementations already exist, and maybe RustPython can join them.

fwungy 3y ago

The beauty of the GIL was that it made writing good python libraries easy, from the utility perspective. Python is such an amazing multi-tool because of its rich library. Personally I'd like to see Python continue as the amazing multitool that it is has always been. Go is the natural concurrency sibling for Python. High performance parts of Python applications should be linked to Go binaries, or simply be written entirely in Go. Ideally Go linkage would be highly integrated into Python. There is a huge place in the world for GIL constrained Python. Enabling Python to push further into high performance environments is just going to make a problem worse.

deleted 3y ago

[deleted]

deleted 3y ago

[deleted]