3 ms·

This is extremely naive, and you answer your own question: > [Par 2] ["Removing GIL creates performance costs that people don't want"] > [Par 3] ["Removing GI

This is extremely naive, and you answer your own question: > [Par 2] ["Removing GIL creates performance costs that people don't want"] > [Par 3] ["Removing GIL breaks CPython API which tons of codebases rely on and people will have to be paid to fix all that code"] In short, it's not possible to remove GIL without requiring code change. > There are no issues with the halting problem here. It's really easy to detect at runtime if python code is interacting with c/c++ code via the old cpython API. Remember, the halting problem only applies to static analysis, not dynamic runtime checks. This is not true and you're thinking of this wrong. When presented with a "will my code work after GIL removal" type of problem, the question becomes "will my code run C/C++ API". Whether it's detected at runtime is irrelevant because the question you want to answer is "does this piece of software require any change at all to work without GIL". To do that, you need a tool that statically analyzes whether known-bad functions (those that run C/C++ functions trivially/deterministically) are called from the rest of the codebase. Imagine you work for a company that has a package "xyz" written in C/C++ called in Python. Now that GIL is removed in an imaginary Python 4, my boss comes to me and asks "how much code needs to be changed for us to port to Python 4". Now, you can apply heuristics such as "how many modules import xyz". But since the correctness implications are co-infective, any module that doesn't import xyz but imports a module that import xyz will be affected as well. You can be more granular and verify whether individual functions use objects in xyz. Which brings me to my original point that there are two cases, either this is a static analysis problem (which we lack) or it's work someone has to crawl the entire codebase, inspect function by function, run unittest by unittest to determine if GIL removal breaks anything. I know because people did exactly this in 2 -> 3 change and it's an extraordinarily expensive ($$$) transition. Your view is naive beyond comprehension and it's hard to think of GIL removal as anything easier than the 2 -> 3 transition which was a shitshow of the magnitude software industry hasn't seen before.

phire 3y ago
It's super easy to check if any C/C++ code has been imported. Not too hard to get a reasonably good result with static analysis, just find all import statements and see if it maps to a python file or a dll/so file, recusing as needed. You don't need to look at any other code except the import statements. For an even better result the main python interpreter can be modified with a flag to just list all imported dll/so files. Run your application with that flag and see what it imported and it will catch weird edge cases (like, python code that messes with import paths) There is no need to check the entire codebase. It's only the C/C++ code that directly uses the cpython api that needs to be checked. Any python code can be ignored, along with C/C++ code that doesn't interact with the cpython API. This isn't like the 2 -> 3 transition where you needed to touch almost every line of python code. The amount of touched code for most companies should be tiny. Also, the proposal isn't for a python 4 were everyone is required to upgrade to this new paradigm. It's for an optional mode that allows running without the GIL. The only people who actually need to worry about it are people who want multithreadding. Your theoretical company can do a quick check to see what c/c++ libraries are imported and estimate how much code would need to be checked. If it's too daunting, they can just keep operating with the GIL enabled.
- gnulinux 3y ago
  > Your theoretical company can do a quick check to see what c/c++ libraries are imported and estimate how much code would need to be checked. If it's too daunting, they can just keep operating with the GIL enabled. Once again, this doesn't make sense and you're simply naively glossing over things. This won't work because likely one of your dependencies use no-GIL mode (such as numpy or something like that). This essentially forks the ecosystem into two, and if you have a 5 million LoC GIL python code you have no way of importing a no-GIL dependency. Nobody will sign up for that kind of mess, python package management is already a huge mess. What happens if the library goes from GIL to no-GIL? Runtime exception. Yeah, thanks but no thanks. Beyond that, it's unclear how best to respond to your comment about static analysis being easy and that you don't need to inspect python code line by line. It needs to be understood that there are companies who have millions of lines of code where code is mixed with C/C++ FFI calls all over the place. I have a codebase like that where almost everything is FFI/Python mixed. I know many companies do. In our case we do need to go line by line to see what's inspected. Even if we don't, our boss will ask us to because if anything fails it'll be our responsibility so we at least need to do the busy work of confirming nothing is affected. It seems like you suffer from the problem that you identify a small subset of reality, then determine the solution you propose is practical, when in real life it causes tons of codebases to be almost rebuilt. Knowing Python folks, I know no one has any interest in such a thing.

phire 3y ago

It's super easy to check if any C/C++ code has been imported. Not too hard to get a reasonably good result with static analysis, just find all import statements and see if it maps to a python file or a dll/so file, recusing as needed. You don't need to look at any other code except the import statements. For an even better result the main python interpreter can be modified with a flag to just list all imported dll/so files. Run your application with that flag and see what it imported and it will catch weird edge cases (like, python code that messes with import paths) There is no need to check the entire codebase. It's only the C/C++ code that directly uses the cpython api that needs to be checked. Any python code can be ignored, along with C/C++ code that doesn't interact with the cpython API. This isn't like the 2 -> 3 transition where you needed to touch almost every line of python code. The amount of touched code for most companies should be tiny. Also, the proposal isn't for a python 4 were everyone is required to upgrade to this new paradigm. It's for an optional mode that allows running without the GIL. The only people who actually need to worry about it are people who want multithreadding. Your theoretical company can do a quick check to see what c/c++ libraries are imported and estimate how much code would need to be checked. If it's too daunting, they can just keep operating with the GIL enabled.

gnulinux 3y ago
> Your theoretical company can do a quick check to see what c/c++ libraries are imported and estimate how much code would need to be checked. If it's too daunting, they can just keep operating with the GIL enabled. Once again, this doesn't make sense and you're simply naively glossing over things. This won't work because likely one of your dependencies use no-GIL mode (such as numpy or something like that). This essentially forks the ecosystem into two, and if you have a 5 million LoC GIL python code you have no way of importing a no-GIL dependency. Nobody will sign up for that kind of mess, python package management is already a huge mess. What happens if the library goes from GIL to no-GIL? Runtime exception. Yeah, thanks but no thanks. Beyond that, it's unclear how best to respond to your comment about static analysis being easy and that you don't need to inspect python code line by line. It needs to be understood that there are companies who have millions of lines of code where code is mixed with C/C++ FFI calls all over the place. I have a codebase like that where almost everything is FFI/Python mixed. I know many companies do. In our case we do need to go line by line to see what's inspected. Even if we don't, our boss will ask us to because if anything fails it'll be our responsibility so we at least need to do the busy work of confirming nothing is affected. It seems like you suffer from the problem that you identify a small subset of reality, then determine the solution you propose is practical, when in real life it causes tons of codebases to be almost rebuilt. Knowing Python folks, I know no one has any interest in such a thing.