5 ms·

The Python GIL Visualized

117 points by gthank 17y ago

tel 17y ago
Though this has nothing to do with GIL or Python, the presentation reminds me of threadscope: http://raintown.org/?page_id=132 http://raintown.org/?page_id=132 Take a look at it in action http://www.youtube.com/watch?v=qZXq8fxebKU http://www.youtube.com/watch?v=qZXq8fxebKU
jcl 17y ago
For readers in the future, here's a link to the article instead of the whole blog: http://www.dabeaz.com/blog/2010/01/python-gil-visualized.html http://www.dabeaz.com/blog/2010/01/python-gil-visualized.htm...
- gthank 17y ago
  Thanks. I must have mangled the link during the submission process. Can someone fix it?
- rbanffy 17y ago
  I found the post about block towers fascinating.
  - pradocchia 17y ago
    Indeed, I have some new ideas for playtime tonight.
  - ash 17y ago
    Yes! http://www.dabeaz.com/blog/2009/11/fun-with-block-towers.html http://www.dabeaz.com/blog/2009/11/fun-with-block-towers.htm...
dmaclay 17y ago
Just don't use threads in python, use multiprocessing!
- mattmcknight 17y ago
  The added benefit is that the multiprocessing approach scales more easily to multiple machines. Same rule applies to Ruby.
  - ajross 17y ago
    And the concomitant disadvantages are that multi-process solutions eat memory like there's no tomorrow, and can share cached data only via IPC. So you get your scalability at the cost of a very large constant factor drop in overall performance. Some problems don't care. Some do. But for those where threading is a more appropriate solution, the Python GIL is a significant limitation. And this is a great exposition as to why. Announcing that threading should never be used just tells the world that you only work in one problem domain (probably web content generation) and haven't thought through the details elsewhere.
    - apu 17y ago
      It's also not practical for many scenarios. Signals in python can only be received by the main thread, and since that's what multiprocessing uses for communication, it means many common scenarios are out: GUI programs where the main thread is the rendering loop, network programs where the main thread is the connection handler, etc. I've had much more luck using Python Remote Objects: http://pyro.sourceforge.net/ http://pyro.sourceforge.net/ It's trivial to make a network queue and use that for communication across different processes/machines/etc.
    - gte910h 17y ago
      I honestly would say MOST people who reach for threads do so instinctively, not after actual analysis of what type of concurrency would serve the project they're at best. Lots of devs come from the java world or windows world where threads are done all the time but multiprocess concurrency is rare. I'd say that out of the projects I've seen people grab for threads, only about 10-15% actually need the type of heavily shared data or marginally smaller memory footprint that threading gives you. Of the rest of them, it was a toss up, or leaned towards the simplification of data structures, easier porting to multiple processors and overall less complex components that IPC gives you. Additionally, I'd contend you're doing something wrong if you see a HUGE increase when you go from threads-> multiprocessed solutions. Usually you just see a small overhead increase unless you're doing something questionable like loading data structures over and over in all processes and the like. May I recommend Stevens Unix Network Programming Vol 2: IPC to see some common designs of MP programs? You may see some large memory footprint gains by just looking through some of the programs in that book and rethinking your processing/responsibilities of your different processes. I by no means think EVERYTHING should be multiprocessed, I just think MOST things can be equally well done with threads or processes, and MANY programmers have never seen multiprocess programs, especially large multiprocess programs, where most have worked in multithreaded programs (where there are more than 1 user threads). Having worked with both, the errors you see in multiprocess programs are MUCH more traceable and manageable. Multi-threading errors can take months to debug sometimes, and weeks to even get a good reproduction scenario for. Then again, for a SMALL amount of concurrency, threading is simpler to setup and do for sure.
diN0bot 17y ago
anyone know of a tool that will generate charts like that when i run my program? perhaps by instrumenting my code? i'd be very curious about this, but i wouldn't want to invest a lot of time into it. right now i use python for my web site serving (django).

117 points by gthank 17y ago

tel 17y ago

Though this has nothing to do with GIL or Python, the presentation reminds me of threadscope: http://raintown.org/?page_id=132 http://raintown.org/?page_id=132 Take a look at it in action http://www.youtube.com/watch?v=qZXq8fxebKU http://www.youtube.com/watch?v=qZXq8fxebKU

jcl 17y ago

For readers in the future, here's a link to the article instead of the whole blog: http://www.dabeaz.com/blog/2010/01/python-gil-visualized.html http://www.dabeaz.com/blog/2010/01/python-gil-visualized.htm...

gthank 17y ago
Thanks. I must have mangled the link during the submission process. Can someone fix it?
rbanffy 17y ago
I found the post about block towers fascinating.
- pradocchia 17y ago
  Indeed, I have some new ideas for playtime tonight.
- ash 17y ago
  Yes! http://www.dabeaz.com/blog/2009/11/fun-with-block-towers.html http://www.dabeaz.com/blog/2009/11/fun-with-block-towers.htm...

dmaclay 17y ago

Just don't use threads in python, use multiprocessing!

mattmcknight 17y ago
The added benefit is that the multiprocessing approach scales more easily to multiple machines. Same rule applies to Ruby.
- ajross 17y ago
  And the concomitant disadvantages are that multi-process solutions eat memory like there's no tomorrow, and can share cached data only via IPC. So you get your scalability at the cost of a very large constant factor drop in overall performance. Some problems don't care. Some do. But for those where threading is a more appropriate solution, the Python GIL is a significant limitation. And this is a great exposition as to why. Announcing that threading should never be used just tells the world that you only work in one problem domain (probably web content generation) and haven't thought through the details elsewhere.
  - apu 17y ago
    It's also not practical for many scenarios. Signals in python can only be received by the main thread, and since that's what multiprocessing uses for communication, it means many common scenarios are out: GUI programs where the main thread is the rendering loop, network programs where the main thread is the connection handler, etc. I've had much more luck using Python Remote Objects: http://pyro.sourceforge.net/ http://pyro.sourceforge.net/ It's trivial to make a network queue and use that for communication across different processes/machines/etc.
  - gte910h 17y ago
    I honestly would say MOST people who reach for threads do so instinctively, not after actual analysis of what type of concurrency would serve the project they're at best. Lots of devs come from the java world or windows world where threads are done all the time but multiprocess concurrency is rare. I'd say that out of the projects I've seen people grab for threads, only about 10-15% actually need the type of heavily shared data or marginally smaller memory footprint that threading gives you. Of the rest of them, it was a toss up, or leaned towards the simplification of data structures, easier porting to multiple processors and overall less complex components that IPC gives you. Additionally, I'd contend you're doing something wrong if you see a HUGE increase when you go from threads-> multiprocessed solutions. Usually you just see a small overhead increase unless you're doing something questionable like loading data structures over and over in all processes and the like. May I recommend Stevens Unix Network Programming Vol 2: IPC to see some common designs of MP programs? You may see some large memory footprint gains by just looking through some of the programs in that book and rethinking your processing/responsibilities of your different processes. I by no means think EVERYTHING should be multiprocessed, I just think MOST things can be equally well done with threads or processes, and MANY programmers have never seen multiprocess programs, especially large multiprocess programs, where most have worked in multithreaded programs (where there are more than 1 user threads). Having worked with both, the errors you see in multiprocess programs are MUCH more traceable and manageable. Multi-threading errors can take months to debug sometimes, and weeks to even get a good reproduction scenario for. Then again, for a SMALL amount of concurrency, threading is simpler to setup and do for sure.

diN0bot 17y ago

anyone know of a tool that will generate charts like that when i run my program? perhaps by instrumenting my code? i'd be very curious about this, but i wouldn't want to invest a lot of time into it. right now i use python for my web site serving (django).