6 ms·

The GIL removal by that guy reverted some of the improvement done by other optimisations, so the overall improvement was much smaller. And most people do care

The GIL removal by that guy reverted some of the improvement done by other optimisations, so the overall improvement was much smaller. And most people do care for single-threaded speed, because the vast majority of Python software is written as single-threaded.

metadat 4y ago
> the vast majority of Python software is written as single-threaded. This is a self-fulfilling prophecy, as the GIL makes Python's (and Ruby's) concurrency story pretty rough compared to nearly all other widely used languages: C, C++, Java, Go, Rust, and even Javascript (as of late).
- SiempreViernes 4y ago
  Pretty sure it is actually a statement that comes from the "python is a scripting language" school, and not because the huge horde of programmers that craves concurrency when they write a script to reformat a log file to csv keeps being put off by the python multiprocessing story.
  - metadat 4y ago
    Not sure I understand your point, can you clarify? Python is used across many different domains, being able to really take advantage of multiple cores would be a big deal. I'd really appreciate if python included concurrency or parallelism capabilities that didn't disappoint and frustrate me. If you've tried using the thread module, multiprocessing module, or async function coloring feature, you probably can relate. They can sort of work but are about as appealing as being probed internally in a medical setting.
    - pclmulqdq 4y ago
      I'm not the person you responded to, but I think the gist of it is: what it is is not defined by how it is used. Python, at its core, is a scripting language, like awk and bash. The other uses don't change that. Occasionally, a technology breaks out of its intended domain. Python is one of these - it plays host to lots of webservers, and even a filesystem (dropbox). Similarly, HTML is a text markup language, but that doesn't stop people from making video games using it. The developers of the technology now have to make a philosophical decision about what that technology is. They can decide to re-optimize for the current use cases and risk losing the heart of their technology as tradeoffs get made, or they can decide to keep the old vision. All of the design choices flow down from the vision. They have decided that Python is a scripting language. Personally, I agree (https://specbranch.com/posts/python-and-asm/ https://specbranch.com/posts/python-and-asm/). In turn, the people using the language for something other than its intended purpose have choices to make - including whether to abandon the technology. If instead Python moves toward competing with languages like go, it is going to need to make a lot of different tradeoffs. Ditching the GIL comes with tradeoffs of making Python slower for those who just want to write scripts. Adding more rigorous scoping would make it easier to have confidence in a production server, but harder to tinker. Everything comes with tradeoffs, and decisions on those tradeoffs come from the values of the development team. Right now, they value scripting.
      - metadat 4y ago
        Python has already become a lot more than a scripting language. To say today that scripting is it's core identity seems naive at best. Yes, it has roots but has object oriented and functional facets which do not exist in awk or bash. Pandas, numpy, scipy, tensorflow. All of these go way beyond what is possible with a scripting language. Since when is the runtime performance of a script a serious concern? Why is it a problem if this aspect gets slightly slower if it brings real concurrency support?
        pclmulqdq 4y ago
        Awk is a functional language with a surprising number of features. Bash, not so much. Developers really do care about runtime performance of scripting languages: They often wait for scripts to finish running before doing other work, and if the sum of the time it takes to write and execute a script is too long, they will look for an alternative. All of the libraries you have cited are from the ML and stats communities, and they are not core language features. From what I understand, ML folks like Python because it is fast to play with and get results. In other words, they like it because it is a scripting language. Personally, I like that Python has kept the GIL so far because I would never run a 24/7 server in Python and I am happy to use it very frequently for single-threaded scripting tasks. Edit: I didn't decide that Python was a scripting language. The Python maintainers did. The point is that the identity of a project doesn't flow down from its use cases. Edit 2: I should have said "the identity of a project doesn't flow down from its users."
        Supermancho 4y ago
        >> To say today that scripting is it's core identity seems naive at best. Yes, it has roots but has object oriented and functional facets which do not exist in awk or bash. Pandas, numpy, scipy, tensorflow. All of these go way beyond what is possible with a scripting language. No, it is not. It's what you can do with a scripting language, which is a fairly straightforward functional categorization. If you are very familiar with it and really want to just ignore the performance issues and don't mind strange workarounds (eg elaborate libraries like Twisted), you probably use python. > The point is that the identity of a project doesn't flow down from its use cases. Use cases are what determines the identity of a language. It takes a massive and ongoing marketing campaign to convince people otherwise, with limited success without use-cases. Python is popular because it's a scripting language with a relatively low learning curve (one true way to do things philosophy). That's it. It will improve over time, but it's been slow going...and that's in comparison to the new Java features! Haskell is a shining example of how the language identity is foisted upon the public until you are convinced enough to try it. It doesn't take long to learn it's a nightmare of incomplete features and overloaded idioms for common tasks, so it isn't used. There aren't good use-cases for a language with haskell's problems, so the developer community and industry-at-large avoids it.
- tyingq 4y ago
  Getting rid of the GIL will also immediately expose all the not-thread-safe stuff that currently exists, so there's a couple of waves you would need before it would be broadly usable.
  - tempest_ 4y ago
    Cool, they should start now. As a python dev, pythons multiprocess/multithreading story is one the largest pain points in the language. Single threaded performance is not that useful while processors have been growing sideways for 10 years. I often look at elixir with jealousy.
    - oofbey 4y ago
      On the flip side, if your workload can be parallelized across thousands of cores, python has about the best CUDA support anywhere.
      - pjmlp 4y ago
        That would be C++ and Fortran actually.
        p10_user 4y ago
        But the python bindings are great and useful to many, so Python gets to be added to the list.
        pjmlp 4y ago
        Just like any language with FFI capabilities to call the same libraries.
    - Nimitz14 4y ago
      As another python dev, it has basically never been a painpoint for me. Your anecdote adds little.
    - 4y ago
- citrin_ru 4y ago
  IMHO majority of Python software doesn't use threads because it is easier to write single threaded code (for many reasons), not because of GIL.
  - metadat 4y ago
    In Golang you can spawn a green thread on any function call with a single keyword: `go'. The ergonomics are such that it's not difficult to use. Why can't or shouldn't we have a mechanism comparably fantastic and easy to use in Python?
    - Twirrim 4y ago
      https://docs.python.org/3/library/concurrent.futures.html https://docs.python.org/3/library/concurrent.futures.html sort of gives you that, syntax works with threads or processes.
    - metadat 4y ago
      @Twirrim How would you rate the user experience of the concurrent.futures package compared to the golang `go' keyword? It is architecturally comparable to async Javascript programming, which imho is a shoehorn solution.
    - lmm 4y ago
      Because making it easy to write C/C++ extensions that work the way you expect (including for things like passing a Python callback to a C/C++ library) has always been a priority for Python in a way that it isn't for Golang?
      - liuliu 4y ago
        Any C/C++ extensions that wants to enable more efficient Python has to learn GIL and how to manipulate that as well. Including not limited to: how to give up GIL (so that other Python code can progress); how to prepare your newly initiated threads to be Python GIL friendly etc. Personally, GIL is more surprising to me when interop with Python.
- dahart 4y ago
  I agree with your point, but the vast majority of C, C++, Java, and Javacript code is also written as single-threaded. It’s fair to acknowledge that the primary use case of most languages is single threaded programming, and also that improving the ergonomics of concurrency in Python would be a huge boon.
magic_hamster 4y ago
A lot of python runs in celery and possibly even more python runs parallelized cuda code. Not sure at all the majority of python code is single threaded, especially for more serious projects.

metadat 4y ago

> the vast majority of Python software is written as single-threaded. This is a self-fulfilling prophecy, as the GIL makes Python's (and Ruby's) concurrency story pretty rough compared to nearly all other widely used languages: C, C++, Java, Go, Rust, and even Javascript (as of late).

SiempreViernes 4y ago
Pretty sure it is actually a statement that comes from the "python is a scripting language" school, and not because the huge horde of programmers that craves concurrency when they write a script to reformat a log file to csv keeps being put off by the python multiprocessing story.
- metadat 4y ago
  Not sure I understand your point, can you clarify? Python is used across many different domains, being able to really take advantage of multiple cores would be a big deal. I'd really appreciate if python included concurrency or parallelism capabilities that didn't disappoint and frustrate me. If you've tried using the thread module, multiprocessing module, or async function coloring feature, you probably can relate. They can sort of work but are about as appealing as being probed internally in a medical setting.
  - pclmulqdq 4y ago
    I'm not the person you responded to, but I think the gist of it is: what it is is not defined by how it is used. Python, at its core, is a scripting language, like awk and bash. The other uses don't change that. Occasionally, a technology breaks out of its intended domain. Python is one of these - it plays host to lots of webservers, and even a filesystem (dropbox). Similarly, HTML is a text markup language, but that doesn't stop people from making video games using it. The developers of the technology now have to make a philosophical decision about what that technology is. They can decide to re-optimize for the current use cases and risk losing the heart of their technology as tradeoffs get made, or they can decide to keep the old vision. All of the design choices flow down from the vision. They have decided that Python is a scripting language. Personally, I agree (https://specbranch.com/posts/python-and-asm/ https://specbranch.com/posts/python-and-asm/). In turn, the people using the language for something other than its intended purpose have choices to make - including whether to abandon the technology. If instead Python moves toward competing with languages like go, it is going to need to make a lot of different tradeoffs. Ditching the GIL comes with tradeoffs of making Python slower for those who just want to write scripts. Adding more rigorous scoping would make it easier to have confidence in a production server, but harder to tinker. Everything comes with tradeoffs, and decisions on those tradeoffs come from the values of the development team. Right now, they value scripting.
    - metadat 4y ago
      Python has already become a lot more than a scripting language. To say today that scripting is it's core identity seems naive at best. Yes, it has roots but has object oriented and functional facets which do not exist in awk or bash. Pandas, numpy, scipy, tensorflow. All of these go way beyond what is possible with a scripting language. Since when is the runtime performance of a script a serious concern? Why is it a problem if this aspect gets slightly slower if it brings real concurrency support?
      - pclmulqdq 4y ago
        Awk is a functional language with a surprising number of features. Bash, not so much. Developers really do care about runtime performance of scripting languages: They often wait for scripts to finish running before doing other work, and if the sum of the time it takes to write and execute a script is too long, they will look for an alternative. All of the libraries you have cited are from the ML and stats communities, and they are not core language features. From what I understand, ML folks like Python because it is fast to play with and get results. In other words, they like it because it is a scripting language. Personally, I like that Python has kept the GIL so far because I would never run a 24/7 server in Python and I am happy to use it very frequently for single-threaded scripting tasks. Edit: I didn't decide that Python was a scripting language. The Python maintainers did. The point is that the identity of a project doesn't flow down from its use cases. Edit 2: I should have said "the identity of a project doesn't flow down from its users."
        Supermancho 4y ago
        >> To say today that scripting is it's core identity seems naive at best. Yes, it has roots but has object oriented and functional facets which do not exist in awk or bash. Pandas, numpy, scipy, tensorflow. All of these go way beyond what is possible with a scripting language. No, it is not. It's what you can do with a scripting language, which is a fairly straightforward functional categorization. If you are very familiar with it and really want to just ignore the performance issues and don't mind strange workarounds (eg elaborate libraries like Twisted), you probably use python. > The point is that the identity of a project doesn't flow down from its use cases. Use cases are what determines the identity of a language. It takes a massive and ongoing marketing campaign to convince people otherwise, with limited success without use-cases. Python is popular because it's a scripting language with a relatively low learning curve (one true way to do things philosophy). That's it. It will improve over time, but it's been slow going...and that's in comparison to the new Java features! Haskell is a shining example of how the language identity is foisted upon the public until you are convinced enough to try it. It doesn't take long to learn it's a nightmare of incomplete features and overloaded idioms for common tasks, so it isn't used. There aren't good use-cases for a language with haskell's problems, so the developer community and industry-at-large avoids it.
tyingq 4y ago
Getting rid of the GIL will also immediately expose all the not-thread-safe stuff that currently exists, so there's a couple of waves you would need before it would be broadly usable.
- tempest_ 4y ago
  Cool, they should start now. As a python dev, pythons multiprocess/multithreading story is one the largest pain points in the language. Single threaded performance is not that useful while processors have been growing sideways for 10 years. I often look at elixir with jealousy.
  - oofbey 4y ago
    On the flip side, if your workload can be parallelized across thousands of cores, python has about the best CUDA support anywhere.
    - pjmlp 4y ago
      That would be C++ and Fortran actually.
      - p10_user 4y ago
        But the python bindings are great and useful to many, so Python gets to be added to the list.
        pjmlp 4y ago
        Just like any language with FFI capabilities to call the same libraries.
  - Nimitz14 4y ago
    As another python dev, it has basically never been a painpoint for me. Your anecdote adds little.
  - 4y ago
citrin_ru 4y ago
IMHO majority of Python software doesn't use threads because it is easier to write single threaded code (for many reasons), not because of GIL.
- metadat 4y ago
  In Golang you can spawn a green thread on any function call with a single keyword: `go'. The ergonomics are such that it's not difficult to use. Why can't or shouldn't we have a mechanism comparably fantastic and easy to use in Python?
  - Twirrim 4y ago
    https://docs.python.org/3/library/concurrent.futures.html https://docs.python.org/3/library/concurrent.futures.html sort of gives you that, syntax works with threads or processes.
  - metadat 4y ago
    @Twirrim How would you rate the user experience of the concurrent.futures package compared to the golang `go' keyword? It is architecturally comparable to async Javascript programming, which imho is a shoehorn solution.
  - lmm 4y ago
    Because making it easy to write C/C++ extensions that work the way you expect (including for things like passing a Python callback to a C/C++ library) has always been a priority for Python in a way that it isn't for Golang?
    - liuliu 4y ago
      Any C/C++ extensions that wants to enable more efficient Python has to learn GIL and how to manipulate that as well. Including not limited to: how to give up GIL (so that other Python code can progress); how to prepare your newly initiated threads to be Python GIL friendly etc. Personally, GIL is more surprising to me when interop with Python.
dahart 4y ago
I agree with your point, but the vast majority of C, C++, Java, and Javacript code is also written as single-threaded. It’s fair to acknowledge that the primary use case of most languages is single threaded programming, and also that improving the ergonomics of concurrency in Python would be a huge boon.

magic_hamster 4y ago

A lot of python runs in celery and possibly even more python runs parallelized cuda code. Not sure at all the majority of python code is single threaded, especially for more serious projects.