10 ms·
> “We need to lay out a plan of how to proceed,” remarked Pablo Galindo Salgado. “Just creating a PR with 20,000 lines of code changed is infeasible.” One of t
by metadat 5y ago
> “We need to lay out a plan of how to proceed,” remarked Pablo Galindo Salgado. “Just creating a PR with 20,000 lines of code changed is infeasible.”
One of the saddest things about popular open-source projects like Python is how inevitably the maintainers become worn down and jaded over time, due to constantly filtering and protecting Python from dumb or nonsensical ideas.
Then when something like this comes along, a true game changer, maintainers have no energy or goodwill left to collaborate on these infinitely important initiatives, like removing the dogdamn GIL. This is way more significant than everything I'm aware of that's happened since the birth of the language. All other things have been, trivial miuntiae in comparison.
It bears repeating: "everything else has been trivial minutiae in comparison to a GIL-ectamy for Python."
It cannot be understated, removing the GIL will be a HUGE deal for Python! It's an ugly wart which has existed for about 30 years, and nobody else has produced and delivered a working solution to the community.
I wish Team Python would welcome GIL Eradication with open arms and a supportive attitude. This would look like focusing on helping identify and implement solutions to the impediments rather than simply pointing out problems and then helicoptering away.
- pmoriarty 5y ago"It cannot be understated, removing the GIL will be a HUGE deal for Python! It's an ugly wart which has existed for about 30 years, and nobody else has produced and delivered a working solution to the community." If it's such a huge game changer why don't some of the large enterprises which rely on Python fund this work?
- deleted 5y ago[deleted]
- meta_throwaway 5y agothat's exactly what has happened. edit: not sure why I got flagged, HN. Software engineer hours == funding.
- ianai 5y agoWell, I might prefer a gofundme for some devs to do it. That way us average Joe’s can pitch into the support. Otherwise, there’s making a fork and attempting periodic updates from python. That’d be a huge undertaking though. FTA: “”” A lot of the value of Python is the ecosystem, not just the language… CPython really leads the way in terms of the community moving as a block. "Removing the GIL is a really transformative step. Most Python programs just don’t use threads at the moment if they want to run on multiple cores. If nogil is to be a success, the community as a whole has to buy into it." – Sam Gross”””
- Scarbutt 5y agoBecause Java,C#,Golang,Javascript exist.
- mateo411 5y agoWait, does Javascript have good multi-threading support? I dont think it does.
- mftb 5y agoWorker threads[0]. If you don't follow Javascript it's not surprising you hadn't heard about it, it's pretty new. Node also, thanks to v8, pretty much outperforms Python. [0]https://nodejs.org/api/worker_threads.html https://nodejs.org/api/worker_threads.html
- frazbin 5y agoworker threads don't share the parent's heap though, right? They can only share state through a binary channel. It's something, but pretty different from traditional pthreads.
- mftb 5y agoThey can use SharedArrayBuffers[0] which wrap shared memory or they can pass ArrayBuffers. Node may not have the shared arrays yet, but the browsers do. [0]https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/SharedArrayBuffer https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
- ptx 5y agoPython has multiprocessing.shared_memory[1] since 3.8 to share arrays between processes, which accomplishes the same thing and circumvents the GIL issue. [1] https://docs.python.org/3/library/multiprocessing.shared_memory.html https://docs.python.org/3/library/multiprocessing.shared_mem...
- gorgoiler 5y agoRemember when it was discovered that arc welders could produce Carbon 60, and there was a huge run on arc welders? All of a sudden there was a whole new class of customer beyond the usual customer base of people who do welding: people who do materials science research. I would hazard a guess to say that 99% of production Python is people doing the equivalent of welding, but there is also this 1% who want the GIL to be gone so they can repurpose Python as a tool for a whole new class of problem solving. It will be an exciting future for them. For us welders, we’ll carry on gluing stuff together blissfully unaware of the GIL. Async and non-blocking IO are great. I don’t think I’ve ever needed compute-concurrency, not in Python anyway.
- colinmhayes 5y agoEh, even with true multithreading python will still be slow as molasses. If you want speed python will still not be your choice.
- Proven 5y ago
- saghm 5y agoI'm not sure I fully buy this argument. I think programming languages are chosen for projects based on a number of different factors, speed being one of them, but developer productivity being another (and there are certainly many more). Given that there isn't usually going to be one language that is the best choice for every single factor, it seems pretty natural that some teams might pick Python due to productivity being more important but still have some need for better performance, and other teams might not be able to compromise on performance and have to resort to something like Java or Go or C++ but still would be able to iterate faster with something higher level. It's definitely not a given that there are enough potential projects that would get enough benefit from removing the GIL to make it worth it, but it seems silly to claim that anyone would would get any possible benefit from performance would never choose Python for other reasons.
- 5y ago
- latenightcoding 5y agoBecause it's not a huge game changer. What would you do differently if the GIL was gone? when people need speed in Python they write C/C++ extensions or use multiple processes/computers
- thelastbender12 5y ago>Because it's not a huge game changer. That is just not true. 1 - This helps parallelize regular python code too, like the kind that goes into writing web services. 2 - While you'd still write native extensions to get optimum performance for single threaded code, having the thread dispatch layer in python makes parallelizing execution convenient and robust. For example, see the comment from people who maintain scikit-learn below. I'd love to see python get composable task parallelism like Julia where you can freely spawn threads within threads without worrying about their lifetimes. link: https://github.com/colesbury/nogil/issues/36#issuecomment-1034624897 https://github.com/colesbury/nogil/issues/36#issuecomment-10...
- diffeomorphism 5y agoThat does not sound all that enthusiastic for a "game changer": > helps parallelize > convenient and robust So nice but not too noticeable?
- diordiderot 5y agoWell context helps determine impact. What percentage of compute time is python? What percent of Py compute time would be cut? Who's using python?(banks, stem researchers, everyone else)
- diffeomorphism 5y agoThat has nothing to do with what I wrote and does not address the question? > ...bunch of questions... Yeah, if you claim that something is a "game changer" you should provide some arguments, not questions/"help determine". It might very well be a game changer but thud far you have only suggested there might be an argument for that, if ...
- dralley 5y agoI don't know if this is still true, but a few years ago there was only a single Python core developer that worked on it full time (Victor Stinner from Red Hat). The rest were only able to devote somewhere between few hours to one day a week. https://discuss.python.org/t/official-list-of-core-developers/924 https://discuss.python.org/t/official-list-of-core-developer...
- di 5y agoŁukasz Langa now works full time on CPython via the PSF's Developer in Residence program. https://pyfound.blogspot.com/2021/07/ukasz-langa-is-inaugural-cpython.html https://pyfound.blogspot.com/2021/07/ukasz-langa-is-inaugura...
- dralley 5y agoGood to hear!
- waygood 5y agoMicrosoft has also been funding a team to work full-time on making Python faster: https://developers.slashdot.org/story/21/05/17/0225252/microsoft-funds-a-team-with-guido-van-rossum-to-double-the-speed-of-python https://developers.slashdot.org/story/21/05/17/0225252/micro...
- jreese 5y agoSam Gross works for Meta, and to my knowledge, this is his #1 project. So in effect, one of the largest companies depending on Python is funding this work.
- dataangel 5y agoThey've tried several times, see Unladen Swallow, but usually the projects are way more ambitious and also trying to introduce JIT.
- saghm 5y agoIt doesn't sound like the reservations of the people in charge of Python are about funding; I don't think it's as simple as "pay them $1m to merge the PR and then pay them salaries as they maintain it". Unless the assumption is that the people in charge of Python would take a bribe, I don't see how large companies could somehow magically make this happen with money short of just forking Python and trying to maintain it themselves, which would burn a ton of goodwill and likely not even win over the community.
- imron 5y ago> If it's such a huge game changer why don't some of the large enterprises which rely on Python fund this work? Because lots of code in python relies on the GIL to act as a synchronization primitive for threaded code (either explicitly or implicitly) and removing the GIL will break that code in subtle ways and there’s no appetite in those companies to fix that.
- 22SAS 5y agoMy thoughts exactly. Google has already done quite a lot of work on LLVM, as an example. Python is among their most used programming languages, this something they could totally assemble engineers to do.
- riyadparvez 5y agoI am curious what exactly is the reason GIL hasn't been removed in one of the most popular languages ever?
- haney 5y agoUntil now attempts to remove the GIL have come at the cost of reduced single threaded performance. This is the first suggestion that doesn’t degrade single threaded performance.
- kzrdude 5y agoWell, all credit to Sam the author, but he "bought" that no degraded performance by adding performance optimizations! Unfortunately, Python 3.11+faster-cpython work has already superseded some of it, i.e negating the trade-in he's trying to accomplish! However, I still hope everyone helps out and that the nogil work can land as default in CPython.
- coldtea 5y ago>Unfortunately, Python 3.11+faster-cpython work has already superseded some of it, i.e negating the trade-in he's trying to accomplish! If "Python 3.11+faster-cpython" didn't also remove the GIL, then it didn't negate anything he's trying to acomplish. He wasn't going for a faster Cpython alone, but for a Cpython without GIL.
- josefx 5y ago> Unfortunately, Python 3.11+faster-cpython work has already superseded some of it, Why is that unfortunate? If his patch being slower than python 3.11 isn't acceptably fast despite being faster than every other python version before that then python was never acceptably fast to begin with. Linux got rid of the BKL ages ago, that the Python community still holds onto the GIL as if it was some multi threading holy grail isn't even remotely funny anymore.
- kzrdude 5y ago
- ctoth 5y ago> There was also a large amount of concern from the attendees about the impact the introduction of nogil could have on CPython development. Some worried that introducing nogil mode could mean that the number of tests run in CI would have to double. Others worried that the maintenance burden would significantly increase if two separate versions of CPython were supported simultaneously The 2021 workstation I am typing this on has 16 physical and 32 virtual cores, and I expect core counts to continue upward this decade. While the CPython core devs may be excellent programmers and do a good job at maintaining Python, they clearly have a bit of trouble with cost/benefit analysis if the complaint is that the test count will double in CI for a ~16X increase in the amount of code that can be run in parallel on even consumer machines. Yes, yes, I know that this does not mean my programs will run ~16X as fast. Yes, I know there are other objections. But this is an order of magnitude away from being an actual blocker and the fact it was brought up at all as an objection shows a fundamental disconnect between the small potential costs imposed on core devs, and the huge potential upsides for Python users everywhere.
- ben-schaaf 5y ago> they clearly have a bit of trouble with cost/benefit analysis if the complaint is that the test count will double in CI for a ~16X increase in the amount of code that can be run in parallel on even consumer machines You can already use multiple cores by writing C extensions that release the GIL and with multiprocessing. That double CI cost and additional work isn't just for core python but for the entire ecosystem.
- forrestthewoods 5y ago> You can already use multiple cores by writing C extensions Don’t use Python is also my preferred recommendation when I encounter Python.
- andybak 5y agoThanks but: 1. Learning a new language is non-trivial for many people (and don't sneer - it's about time not competency) 2. The ecosystem matters. If the code you want to interface with is in Python then "don't use Python" is just glib. I'm mainly proficient in Javascript, Python and C#. My choice of language is rarely based on "which is best for this task?" but mostly "want do I need to run on and interface with?"
- Someone 5y agoFTA: Overall, there was still a large amount of excitement and curiosity about nogil mode from the attendees. ⇒ I don’t see this as a team of worn-down maintainers, but rather as a team of seasoned developers, who know you don’t turn a project with millions of users around on a dime. One question I miss is whether the set of locks this change introduces is the optimal one. I would think that, if it were to be improved upon in the future, C extensions would have to be changed again, so you’d rather not do that often.
- throwaway894345 5y ago> who know you don’t turn a project with millions of users around on a dime Hasn’t this gil/nogil saga been going on for ~decades? What about fixing packaging or limiting the C-extension interface? I have enormous sympathy of what the Python maintainers are up against, but “on a dime” seems hard to justify.
- mumblemumble 5y agoPython nerfing C extensions would be like McDonald's deciding to de-emphasize burgers on their menu. Sure, other foods are healthier and get more positive buzz on Twitter. But it turns out that the burgers are still the main thing bringing people to the restaurant.
- throwaway894345 5y agoI wasn’t talking about nerfing C-extensions. You can get the same power out of a smaller, well-defined interface that would give the interpreter a lot more leeway for optimization. This optimizations would therefore make native Python packages more feasible, making the ecosystem less dependent on C extensions. Anyway, I don’t think people use Python because of the C-extensions packages (although there are certainly some exceptions e.g., pandas), I think people use C extensions because the native performance is so poor.
- pwang 5y ago
- sshlocalhost98 5y agoCan anyone elin (explain like I am noob), why should the Gil be removed? Why need concurrency in normal python scripts let’s days for web scraping or machine learning ? I don’t really get it.
- josefx 5y agoGoing by the paper it also improves performance even for single threaded code[1]. The GIL is just dated design, there isn't anything good about it. [1]https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsDFosB5e6BfnXLlejd9l0/edit#heading=h.6i2dwx2chdq9 https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsD... > Why need concurrency in normal python scripts let’s days for web scraping or machine learning ? I had a few scripts where I had to process over a hundred files, just running that in parallel can reduce a job that takes minutes to seconds.
- ptx 5y agoAs some other commenters have mentioned, the GIL removal branch also makes some unrelated optimizations and the performance improvement comes from those rather than from the GIL removal itself, as I understand it. From your link: "Stripping out some of the GIL-removal related changes would result in even faster performance on the single-threaded pyperformance benchmark suite. [...] The resulting interpreter is about 9% faster than the no-GIL proof-of-concept (or ~19% faster than CPython 3.9.0a3). That 9% difference between the “nogil” interpreter and the stripped-down “nogil” interpreter can be thought of as the “cost” of the major GIL-removal changes." So it seems removing the GIL has a negative impact on single-threaded code, with the version that has both the GIL and the unrelated optimizations being 9% faster.
- sshlocalhost98 5y agoThank you sir/ma’am for the explanation! The Google docs was an excellent resource.
- coldtea 5y ago>Can anyone elin (...) Why need concurrency in normal python scripts let’s days for web scraping or machine learning ? I don’t really get it. That's not a noob question. That's a loaded question, from a perspective of knowing what people use Python for. People use Python for all kinds of things, and a lot of them would be faster if they were able to take advantage of multithreading without the GIL.
- oxfordmale 5y agoYes, they need a lay out a plan on how to proceed. If they don't proceed with this, it will eventually result in the slow decline of Python and its eventual demise. Multi-threading in Python is an absolutely pain, and is something that needs to be addressed. We are at the end of Moore's law and CPU speeds have reached their limit. The only way forward performance wise is to kill the GIL.
- ghqmlp 5y agoIt is all political at this point. Python is run by representatives of big corporations, some of whom haven't really done anything in the past 10 years but are who great at promoting their personal brands. They are trying to strengthen corporate influence and discourage any power of traditional independent open source developers (who have contributed most of the useful features as opposed to corporate churn). We do not know what is going on in the background, it is proxy wars all over the place. It is possible that they want to reduce Facebook influence, that their code bases do not need this feature, etc. Just use C++.
- throwaway894345 5y agoKind of feels like Python is in a bit of a mess in no small part because the maintainers didn’t do a good job of blocking bad ideas. For example, exposing virtually the entire interpreter as the public C extension interface (can’t optimize much without breaking things, thus driving more dependence on C for performance which makes it even harder to change anything and leads to insanity in the Python packaging ecosystem).
- baobob 5y agoThis is asking for Guido to have predicted the multicore era and the success of Python in 1991, at a time when SMP architectures had only been commercially available on extremely expensive systems for 5 years. Much of what contributed to that success was in fact the rich C API, and the ease with which it could be embedded into larger applications, and I'm not sure what an alternative would look like that could produce efficient implementations of things like Numpy, etc.
- mikepurvis 5y agoI think a more reasonable timeframe for this discussion is the release of Python 3 in 2008. That was much more recently, clearly in the multicore era, and a major opportunity to do a breaking change.
- eesmith 5y agoThere were multiple discussions. The main issue was removing the GIL made single-threaded Python. Experiments in 1999 found single threaded performance was 50% slower - you would need 2 cores just to break even, assuming you had parallel code in the first place. Here's a 2007 essay by van Rossum on the topic: https://www.artima.com/weblogs/viewpost.jsp?thread=214235 https://www.artima.com/weblogs/viewpost.jsp?thread=214235 . > I'd welcome it if someone did another experiment along the lines of Greg's patch (which I haven't found online), and I'd welcome a set of patches into Py3k only if the performance for a single-threaded program (and for a multi-threaded but I/O-bound program) does not decrease.
- 5y ago
- RcouF1uZ4gsC 5y ago> It cannot be understated, removing the GIL will be a HUGE deal for Python! I am not sure about it. Python explicitly trades performance for simplicity and the GIL simplifies a whole lot of things and because of it, there are multi threading issues that will never be encountered by Python code. In addition, because of Python’s popularity, there is a massive amount of code written in Python (and other languages as extensions) and any change cannot introduce a bug. Finally, there are other alternatives to Python that are more performance focused. If you are having huge issues with Python performance, maybe you are using it in a matter it was not designed for.
- jka 5y ago> I wish Team Python would welcome GIL Eradication with open arms and a supportive attitude. This would look like focusing on helping identify and implement solutions to the impediments rather than simply pointing out problems and then helicoptering away. It looks like they are supportive and keen on the changes? From the article: > Gross's proposal was greeted with a mix of excitement and robust questioning from the assembled core developers. It's definitely a huge opportunity, no doubt. But it affects the C ABI[1] for Python itself, something that everyone in the conversation is aware of, and that would have implications for all distributions of Python. Add in the importance of effective code review for a 20k LoC changeset, and it seems there are good reasons to be cautious despite the optimism and excitement. [1] - https://docs.python.org/3/c-api/stable.html#stable-application-binary-interface https://docs.python.org/3/c-api/stable.html#stable-applicati...
- mumblemumble 5y ago> It cannot be understated, removing the GIL will be a HUGE deal for Python! It's an ugly wart which has existed for about 30 years I would argue the opposite: it's the secret to Python's success. It might even be my top example of how "worse is better" plays out in real life. I agree, the GIL feels like an ugly hack. The software engineer in me wants to hate it so much. And, now that I'm on the far side of a successful transition to a data science role, one might think that I hate it even more, yeah? Because the work I'm doing depends so very heavily on compute performance and parallelism. But it turns out, nah, I'm coming to like it. It's an ugly hack, but it's the best kind of ugly hack: one that gets the job done. Because I'm pretty sure that the GIL is the secret sauce that makes Python C extensions work so well. Without it, it would be much more difficult to write safe, correct C extensions. Doing it without introducing race conditions that destroy memory safety would be a black art. So people would probably do it less. And that probably means no robust Python scientific computing or data science ecosystem, because that stuff is all C extensions. We could instead use a C FFI, like it's done in other languages. Java, for example. But Java having to use an FFI and Python being able to use its C extension mechanism is exactly why Python has eaten all of Java's Wheaties in the data space. The marshaling costs of talking back and forth across that boundary are just too high. Copying goes up, locality of reference goes down, cache misses go up. You saturate the memory bus more quickly. Once you've done that, it doesn't matter how many threads you have running on how many cores. The bottleneck is happening outside the CPU. Top will happily tell you those cores are working hard, but it turns out that what they're working so hard at is sitting around and waiting. This isn't just theoretical. Last year I replaced a heavily parallelized compute-heavy batch process written in Java with a Python implementation that got the work done in less time despite being single-threaded. Sure, the Python implementation was basically a little scripting on top of a library written in C++, and the Java one was pure Java. But that's kind of the whole point. I also know that, back when I wrote the original Java implementation, I tried the same trick of farming the calculation out to a C++ library, and it actually made the throughput even worse. JNI is a harsh master. And besides, as others have said, numpy & friends give me most the parallelism I actually need, for free. Maybe it hurts other people more? Maybe Web developers? But there's a part of me that thinks, if you're trying to do that kind of work at scale, making Python go faster is perhaps barking up the wrong tree. There are plenty of other languages that are statically typed (reducing pointer chasing and branching can increase your throughput without giving Amazon more money in the process) and don't even need a global interpreter lock in the first place because they're not interpreted, either.
- stjohnswarts 5y agoI agree and have been complaining about it for ages. Mostly got around it via using processes and microservices, but this would make my life so much easier. With general processors having tons of cores and memory now it just doesn't make sense, especially if the new changes work well with little or no performance hit. Seems like a no brainer.
- omnicognate 5y agoNot everybody agrees with you that it's a worthy goal. Personally I have little desire to see it removed, doubt its removal will be achieved and tend to think the various attempts to do it are misguided. I could imagine getting pretty jaded as a python maintainer having to keep pointing out why the latest attempt won't work. I think the onus is on those who want this to demonstrate that it can be done successfully and lay out a plan as to how it can be put into the real world without causing chaos in a python ecosystem that has only relatively recently got over the trauma of python 3 (which is the bit I doubt is possible). Edit: To be clear, I don't think anything that consists of a set of proposed changes to the cpython interpreter is even beginning to attempt to think about the implications for the ecosystem, which is where the actual challenges are, and I'm assuming that's what the "Just creating a PR" comment is saying.