6 ms·

This was FUD when it was posted and now it is age-old FUD which there's no point in re-posting. Jython and IronPython do not have a GIL. Multiprocessing avoids

by pekk 13y ago

This was FUD when it was posted and now it is age-old FUD which there's no point in re-posting. Jython and IronPython do not have a GIL. Multiprocessing avoids the GIL. Blocking on I/O gives up the GIL. There are all kinds of techniques used instead of throwing threads naively at every problem. And, conveniently, none of this is mentioned in the article. Either the author was not aware of these basic facts, or suppressed them. It is blatantly false that "no single issue has caused more frustration or curiosity for Python novices and experts alike than the Global Interpreter Lock." The author may consider it important, but this does not mean that author is speaking for everyone else. Novices would have good reason to avoid shared-everything threading, which introduces piles of race conditions and difficulty controlling runaway threads, and should try simpler tools first and see whether they can get good results instead of prematurely optimizing with techniques they don't know how to use. Experts will know that the GIL is often not a primary concern, and where it actually is a concern they'll be conversant with other tools like multiprocessing and task queues. The people with the most to say about the GIL are mediocre programmers who want to show off that they are so good Python is limiting them, and people not very familiar with Python (possibly with background in languages which try to make threads the answer to everything) who have an axe to grind. Instead of asking how it is to do what they want to do, they just assume that the problem is the GIL and there is no solution, then expect to be praised for their technical acumen. People with technical acumen just solve the problem in any of the available ways instead of bitching in public about how it's the tool's fault they can't solve the problem because they have defined the problem incorrectly and insist on some arbitrary way of doing it.

jknupp 13y ago
I'm the original author (and didn't post this) but I fail to see how the article is FUD. In fact, the article makes the exact same point as your fourth paragraph ("Novices would..."). Controlling access to shared data often does lead to issues for many programmers, and just throwing threading at a problem is rarely a good idea. I'm sorry you didn't find the article useful, but perhaps you're not in the article's target audience. I simply wanted to give Python novices some information and background about the GIL. That said, the fact that other implementations do not have a GIL isn't relevant to the article; it specifically refers to the cPython implementation. And your observation that multiprocessing avoids the GIL is explicitly mentioned in the article. To say "blocking on I/O gives up the GIL" is true in a very narrow sense but not very interesting. Import any third party package using C extensions and you now need to worry about how well the author manages the GIL.
- markdown 13y ago
  As a newcomer to Python (and programming in general), your article was very informative. Thank you for writing it.
- goostavos 13y ago
  Ignore the negativity train on Hacker News as of late. I thought it was a very well written and interesting article. I was once one of the very noobies that your article describes. In fact, I even made the same Stack Overflow post asking why threading made my program slow to a crawl. I did then (thanks to a friendly answer) learn about multiprocessing rather than threading, but I never actually got around to learning -- or even thinking about!-- what exactly the GIL is and why it makes threading terrible for CPU bound tasks. Point being, I enjoyed it.
  - Luyt 13y ago
    "I never actually got around to learning -- or even thinking about!-- what exactly the GIL is" David Beazley does a good job on explaining the GIL. There are multiple videos of his GIL talks on YouTube. This one, for instance: http://www.youtube.com/watch?v=Obt-vMVdM8s http://www.youtube.com/watch?v=Obt-vMVdM8s He has a page about this at http://www.dabeaz.com/GIL/ http://www.dabeaz.com/GIL/
- rdtsc 13y ago
  There has been talk of criticism on HN about how comments are negative and that is true. So I hope this doesn't add to it and I am trying to honest here and say how I see things. First of all thanks for putting the effort into writing that. There is good information in the article describing how the GIL works and how threads work and things to watch out for. But unfortunately I have to agree with the gp as well, a part of it is FUD. The FUD is not in what was said (most what was said about the GIL is true) the FUD is in what was not said, and that is that threads are there for a reason, and they do provide good speedup in a large number of applications, namely those that are IO bound. I don't think that fact is too hard for novices to grasp so it should be me mentioned. The perception otherwise is that the creators have simply gone insane and decide to add threading to the language but you should never use it because it doesn't work (so why didn't they just remove it then?). Well it does work in large class of problems. Think about nodejs. Like it or not it has become popular recently. It doesn't by default support or take advantage of multiple cores yet it is often found to be performant enough to handle a decent number of concurrent clients connecting. Granted it doesn't pretend to have threads in the first place but it is an example of a modern, useful ecosystem that does not take advantage of all the cores. > To say "blocking on I/O gives up the GIL" is true in a very narrow sense Not it is not. It is true in a very general and wide sense. Try it out! Spawn 50 threads and try to download 50 different pages, you'll notice a speedup relative to doing it all sequentially in a while loop. > Import any third party package using C extensions and you now need to worry about how well the author manages the GIL. Also misrepresents the facts a bit. GIL actually is supposed to make it easier to write C extensions. If the C extension doesn't mess with the GIL it is straight forward to write because the GIL is there. If the GIL wasn't there calling C extensions and returning or having a callback in Python from C would be a pretty complicated affair. Now you can play with the GIL in C and release it to achieve parallel speedup, and I have done that, but that was in a few cases already over the years.
diminoten 13y ago
You're conflating the engineer/scientist role here pretty severely. The engineer routes around the problem, the scientist tries to understand the problem better. An engineer with a deadline will not be held up in any significant way by the GIL. The engineer will use one of the many ways you've already listed to solve the problem and move on. The scientist, not bound as strictly by deadlines or by "just being done", will explore the issue more leisurely, focusing on the whys and what the problem means. The scientist will explore the implications. I think only you are attempting to "blame" Python for this problem - the author's tone in no way suggests that the problem is in Python itself or even the CPython implementation. There's a distinct, "here's the lay of the land on this issue" about this article, rather than a "Python is at fault for these reasons..." that you seem to be interpreting.
- dekhn 13y ago
  I'm an ex-scientist (>10 years) who is now an software engineer (>5 years). Everything below is anecdotal, but based on a pretty large sample of engineers and scientists combined with a highly skilled anti-bias corrector. While I agree superficially in many ways with your characterization of engineers on deadlines versus scientists who will explore certain issues more closely, I think there are a number of things wrong with your comment: 1) As a scientist I frequently had to deal with the GIL. A major part of my time was taking C libraries, making SWIG wrappers for them, and using the Python SWIG libraries to do productive science. Once SWIG-wrapped, a good C library is very quick to do prototyping but it requires close attention and a good understanding of CPython API threading details to ensure your SWIG libraries are safe for other scientists. 2) Nor are "engineers on deadlines" going to accept the GIL in CPython. I'm an engineer on deadlines and after 18 years of programming in Python, I've switched to go because frankly, I can't see Python having much of a future anymore. 3) Ultimately, engineers and scientists are the same thing. 24% of the time of every scientist is spent coming up with some engineering solution in a hurry so you can get your experiment to work, (the other 75% of the time is spent writing grant proposals begging for money for the engineering and writing papers so that your grant proposals get approved, and 1% actually feeling true scientific inspiration). The whole "leisurely" word you use doesn't apply to how scientists anywhere in the real world work (except perhaps somebody with awesome funding nearing retirement, or perhaps some wealthy citizen-scientist with a home lab).
jwarkentin 13y ago
This kind of makes it sound like you have a very limited variety of experience programming. Two things: 1) Race conditions exist with any sort of asynchronous or concurrent software development, even when there is no parallelism or threading. Have you ever written a moderately complex JavaScript app? There's no shortage of race conditions and no threads either. 2) There are in fact cases where forking isn't a reasonable solution. For one of my previous jobs I developed a multi-threaded server with Python (before I learned about the GIL). The problem with forking came in the fact that there was one particular variable that needed to be shared between threads for processing. It happened to contain about 5GB of data, memory which I didn't want duplicated and which would have been a pain in the butt to keep synced between processes. For most of these shared data structures I could do multiple processes and just use Redis as a shared data store. However, this piece of data was accessed far too frequently and that would have just been a major performance bottleneck. To say the least, I was terribly frustrated when I learned about the GIL. I wish they would put a big warning up on their home page or at least in the docs relating to threading.
- mzs 13y ago
  multiprocessing is just pretty broken in general. I've never been able to get it to work well in FreeBSD in particular. Essentially python calls all sorts of unsafe routines after the fork, here's an example bug report: http://bugs.python.org/issue3770 http://bugs.python.org/issue3770
yummyfajitas 13y ago
Multiprocessing is a terrible solution to the GIL. You gain parallelism, but you are then required to serialize/deserialize and duplicate every shared object. In Scala/Java, I might build a single immutable object (taking up e.g. 1kb) and transmit it to 10 actors. They use it as needed and let the GC deal with it when finished. In Python, I need to serialize it, transmit it 10 times and use 10kb of memory to store the copies. The GIL is a flaw in the language. We should accept that. There are workarounds and hacks, but the GIL is still a flaw. (Incidentally, my background is in python. My usage of Scala/Java is far more recent.)
- falcolas 13y ago
  Technically, the GIL is a "flaw" in the implementation. The language does not specify a GIL, it's just an implementation detail in cpython. I don't necessarily agree that it's a flaw, but that's another discussion entirely.
  - valdiorn 13y ago
    In this day an age, the inability to compute two things at once is a pretty major flaw, if you ask me.
    - falcolas 13y ago
      Using Python for high performance computation is also a flaw, if you ask me.
  - mzs 13y ago
    And in CPython it's less that the GIL is the flaw, but more the refcounted GC. Back in '96 there was a patch to remove the GIL, run of the mill single threaded python code ran 2-6x slower (largely dependent on threading implementation used) due to all the locking overhead around the refcount updates happening all the time. When you have a language that is perceived as slow already, making the vast majority of typical scripts of the time that much slower to allow MT python to be faster was going to be a very difficult thing to sell.
- 13y ago
leephillips 13y ago
Actual novices are more likely to get tripped up by such things as understanding mutable arguments to functions before they even try working with threads.

by pekk 13y ago

jknupp 13y ago

I'm the original author (and didn't post this) but I fail to see how the article is FUD. In fact, the article makes the exact same point as your fourth paragraph ("Novices would..."). Controlling access to shared data often does lead to issues for many programmers, and just throwing threading at a problem is rarely a good idea. I'm sorry you didn't find the article useful, but perhaps you're not in the article's target audience. I simply wanted to give Python novices some information and background about the GIL. That said, the fact that other implementations do not have a GIL isn't relevant to the article; it specifically refers to the cPython implementation. And your observation that multiprocessing avoids the GIL is explicitly mentioned in the article. To say "blocking on I/O gives up the GIL" is true in a very narrow sense but not very interesting. Import any third party package using C extensions and you now need to worry about how well the author manages the GIL.

markdown 13y ago
As a newcomer to Python (and programming in general), your article was very informative. Thank you for writing it.
goostavos 13y ago
Ignore the negativity train on Hacker News as of late. I thought it was a very well written and interesting article. I was once one of the very noobies that your article describes. In fact, I even made the same Stack Overflow post asking why threading made my program slow to a crawl. I did then (thanks to a friendly answer) learn about multiprocessing rather than threading, but I never actually got around to learning -- or even thinking about!-- what exactly the GIL is and why it makes threading terrible for CPU bound tasks. Point being, I enjoyed it.
- Luyt 13y ago
  "I never actually got around to learning -- or even thinking about!-- what exactly the GIL is" David Beazley does a good job on explaining the GIL. There are multiple videos of his GIL talks on YouTube. This one, for instance: http://www.youtube.com/watch?v=Obt-vMVdM8s http://www.youtube.com/watch?v=Obt-vMVdM8s He has a page about this at http://www.dabeaz.com/GIL/ http://www.dabeaz.com/GIL/
rdtsc 13y ago
There has been talk of criticism on HN about how comments are negative and that is true. So I hope this doesn't add to it and I am trying to honest here and say how I see things. First of all thanks for putting the effort into writing that. There is good information in the article describing how the GIL works and how threads work and things to watch out for. But unfortunately I have to agree with the gp as well, a part of it is FUD. The FUD is not in what was said (most what was said about the GIL is true) the FUD is in what was not said, and that is that threads are there for a reason, and they do provide good speedup in a large number of applications, namely those that are IO bound. I don't think that fact is too hard for novices to grasp so it should be me mentioned. The perception otherwise is that the creators have simply gone insane and decide to add threading to the language but you should never use it because it doesn't work (so why didn't they just remove it then?). Well it does work in large class of problems. Think about nodejs. Like it or not it has become popular recently. It doesn't by default support or take advantage of multiple cores yet it is often found to be performant enough to handle a decent number of concurrent clients connecting. Granted it doesn't pretend to have threads in the first place but it is an example of a modern, useful ecosystem that does not take advantage of all the cores. > To say "blocking on I/O gives up the GIL" is true in a very narrow sense Not it is not. It is true in a very general and wide sense. Try it out! Spawn 50 threads and try to download 50 different pages, you'll notice a speedup relative to doing it all sequentially in a while loop. > Import any third party package using C extensions and you now need to worry about how well the author manages the GIL. Also misrepresents the facts a bit. GIL actually is supposed to make it easier to write C extensions. If the C extension doesn't mess with the GIL it is straight forward to write because the GIL is there. If the GIL wasn't there calling C extensions and returning or having a callback in Python from C would be a pretty complicated affair. Now you can play with the GIL in C and release it to achieve parallel speedup, and I have done that, but that was in a few cases already over the years.

diminoten 13y ago

You're conflating the engineer/scientist role here pretty severely. The engineer routes around the problem, the scientist tries to understand the problem better. An engineer with a deadline will not be held up in any significant way by the GIL. The engineer will use one of the many ways you've already listed to solve the problem and move on. The scientist, not bound as strictly by deadlines or by "just being done", will explore the issue more leisurely, focusing on the whys and what the problem means. The scientist will explore the implications. I think only you are attempting to "blame" Python for this problem - the author's tone in no way suggests that the problem is in Python itself or even the CPython implementation. There's a distinct, "here's the lay of the land on this issue" about this article, rather than a "Python is at fault for these reasons..." that you seem to be interpreting.

dekhn 13y ago
I'm an ex-scientist (>10 years) who is now an software engineer (>5 years). Everything below is anecdotal, but based on a pretty large sample of engineers and scientists combined with a highly skilled anti-bias corrector. While I agree superficially in many ways with your characterization of engineers on deadlines versus scientists who will explore certain issues more closely, I think there are a number of things wrong with your comment: 1) As a scientist I frequently had to deal with the GIL. A major part of my time was taking C libraries, making SWIG wrappers for them, and using the Python SWIG libraries to do productive science. Once SWIG-wrapped, a good C library is very quick to do prototyping but it requires close attention and a good understanding of CPython API threading details to ensure your SWIG libraries are safe for other scientists. 2) Nor are "engineers on deadlines" going to accept the GIL in CPython. I'm an engineer on deadlines and after 18 years of programming in Python, I've switched to go because frankly, I can't see Python having much of a future anymore. 3) Ultimately, engineers and scientists are the same thing. 24% of the time of every scientist is spent coming up with some engineering solution in a hurry so you can get your experiment to work, (the other 75% of the time is spent writing grant proposals begging for money for the engineering and writing papers so that your grant proposals get approved, and 1% actually feeling true scientific inspiration). The whole "leisurely" word you use doesn't apply to how scientists anywhere in the real world work (except perhaps somebody with awesome funding nearing retirement, or perhaps some wealthy citizen-scientist with a home lab).

jwarkentin 13y ago

This kind of makes it sound like you have a very limited variety of experience programming. Two things: 1) Race conditions exist with any sort of asynchronous or concurrent software development, even when there is no parallelism or threading. Have you ever written a moderately complex JavaScript app? There's no shortage of race conditions and no threads either. 2) There are in fact cases where forking isn't a reasonable solution. For one of my previous jobs I developed a multi-threaded server with Python (before I learned about the GIL). The problem with forking came in the fact that there was one particular variable that needed to be shared between threads for processing. It happened to contain about 5GB of data, memory which I didn't want duplicated and which would have been a pain in the butt to keep synced between processes. For most of these shared data structures I could do multiple processes and just use Redis as a shared data store. However, this piece of data was accessed far too frequently and that would have just been a major performance bottleneck. To say the least, I was terribly frustrated when I learned about the GIL. I wish they would put a big warning up on their home page or at least in the docs relating to threading.

mzs 13y ago
multiprocessing is just pretty broken in general. I've never been able to get it to work well in FreeBSD in particular. Essentially python calls all sorts of unsafe routines after the fork, here's an example bug report: http://bugs.python.org/issue3770 http://bugs.python.org/issue3770

yummyfajitas 13y ago

Multiprocessing is a terrible solution to the GIL. You gain parallelism, but you are then required to serialize/deserialize and duplicate every shared object. In Scala/Java, I might build a single immutable object (taking up e.g. 1kb) and transmit it to 10 actors. They use it as needed and let the GC deal with it when finished. In Python, I need to serialize it, transmit it 10 times and use 10kb of memory to store the copies. The GIL is a flaw in the language. We should accept that. There are workarounds and hacks, but the GIL is still a flaw. (Incidentally, my background is in python. My usage of Scala/Java is far more recent.)

falcolas 13y ago
Technically, the GIL is a "flaw" in the implementation. The language does not specify a GIL, it's just an implementation detail in cpython. I don't necessarily agree that it's a flaw, but that's another discussion entirely.
- valdiorn 13y ago
  In this day an age, the inability to compute two things at once is a pretty major flaw, if you ask me.
  - falcolas 13y ago
    Using Python for high performance computation is also a flaw, if you ask me.
- mzs 13y ago
  And in CPython it's less that the GIL is the flaw, but more the refcounted GC. Back in '96 there was a patch to remove the GIL, run of the mill single threaded python code ran 2-6x slower (largely dependent on threading implementation used) due to all the locking overhead around the refcount updates happening all the time. When you have a language that is perceived as slow already, making the vast majority of typical scripts of the time that much slower to allow MT python to be faster was going to be a very difficult thing to sell.
13y ago

leephillips 13y ago

Actual novices are more likely to get tripped up by such things as understanding mutable arguments to functions before they even try working with threads.