3 ms·

The added benefit is that the multiprocessing approach scales more easily to multiple machines. Same rule applies to Ruby.

ajross 17y ago
And the concomitant disadvantages are that multi-process solutions eat memory like there's no tomorrow, and can share cached data only via IPC. So you get your scalability at the cost of a very large constant factor drop in overall performance. Some problems don't care. Some do. But for those where threading is a more appropriate solution, the Python GIL is a significant limitation. And this is a great exposition as to why. Announcing that threading should never be used just tells the world that you only work in one problem domain (probably web content generation) and haven't thought through the details elsewhere.
- apu 17y ago
  It's also not practical for many scenarios. Signals in python can only be received by the main thread, and since that's what multiprocessing uses for communication, it means many common scenarios are out: GUI programs where the main thread is the rendering loop, network programs where the main thread is the connection handler, etc. I've had much more luck using Python Remote Objects: http://pyro.sourceforge.net/ http://pyro.sourceforge.net/ It's trivial to make a network queue and use that for communication across different processes/machines/etc.
- gte910h 17y ago
  I honestly would say MOST people who reach for threads do so instinctively, not after actual analysis of what type of concurrency would serve the project they're at best. Lots of devs come from the java world or windows world where threads are done all the time but multiprocess concurrency is rare. I'd say that out of the projects I've seen people grab for threads, only about 10-15% actually need the type of heavily shared data or marginally smaller memory footprint that threading gives you. Of the rest of them, it was a toss up, or leaned towards the simplification of data structures, easier porting to multiple processors and overall less complex components that IPC gives you. Additionally, I'd contend you're doing something wrong if you see a HUGE increase when you go from threads-> multiprocessed solutions. Usually you just see a small overhead increase unless you're doing something questionable like loading data structures over and over in all processes and the like. May I recommend Stevens Unix Network Programming Vol 2: IPC to see some common designs of MP programs? You may see some large memory footprint gains by just looking through some of the programs in that book and rethinking your processing/responsibilities of your different processes. I by no means think EVERYTHING should be multiprocessed, I just think MOST things can be equally well done with threads or processes, and MANY programmers have never seen multiprocess programs, especially large multiprocess programs, where most have worked in multithreaded programs (where there are more than 1 user threads). Having worked with both, the errors you see in multiprocess programs are MUCH more traceable and manageable. Multi-threading errors can take months to debug sometimes, and weeks to even get a good reproduction scenario for. Then again, for a SMALL amount of concurrency, threading is simpler to setup and do for sure.
  - ajross 17y ago
    That's a fair case, but it's not exactly a rebuttal to what I said either. If you can fit your problem into a multi-process solution (and most web stuff qualifies), then yes, there are many advantages there. But some problems benefit from both CPU parallelism and fast shared memory (think numerics work, graphics, games, data stores...). And this is an area that (due to the GIL contention visualized in the linked article) Python serves poorly. But don't fool yourself that "MOST" of anything works with any one architecture. It's a big world. And I assure you I've read Stevens.
    - illumen 17y ago
      pygame does a lot of its threaded graphics, and sound work entirely in C world, because python threads are so limited. Some of us have been using mmapd surfaces with python/pygame for years though. For graphical work, it's not all that hard to share data with python. You can use the same approaches with numpy. See this cookbook example: http://www.pygame.org/wiki/MmapSurfaces http://www.pygame.org/wiki/MmapSurfaces it's only a dozen lines of code to share data via mmap. There are lots of issues with the python, and numpy mmap modules though... and mmap is very different across platforms (eg, linux, windows, macosx have quite different behaviour). Another approach good for some graphics problems is forking. Forking is pretty fast, and it allows you to share memory in a fairly nice way. However mixing forking and threading is a quite tricky with python. This is also not so good on windows - mainly its a good method on *nix. cu,
      - gte910h 17y ago
        I'd love a good windows fork.....win7 hasn't implemented it yet I'm guessing?
    - gte910h 17y ago
      >But don't fool yourself that "MOST" of anything works with any one architecture. It's a big world. Of methods of concurrency that people use, there are two main approaches, threads, and multiprocessing. I just contend only about 15% of solutions are super clear winners for one or the other, leaving 65-70% of apps that can go either way. And I'm not a web developer, I'm an embedded software developer with years of experience in embedded video, streaming video, and hardware emulation. Lots of THAT stuff also fits just fine in either system too. As does the test systems, large analytics programs, etc that I've seen unrelated to the field. Actually python largely keeping to the inconsistency of the underlying OS with regards to mmap and other shared memory solutions is the biggest obstacles there. Oh, and the fact that hardcore numerics are still callouts to C libs in python. >And I assure you I've read Stevens. Lots and lots of people read Vol1, it seems like 1/10th as many read Vol2. Hell, I'm pretty sure most people who read Vol1 haven't a clue what's in Vol2. I'm by no means saying I want the GIL to say the same, I just think numerics work, graphics, games need threads, after that, you get to use either one (aka, 15% need threads, the other 85 need processes or can use either).

ajross 17y ago

And the concomitant disadvantages are that multi-process solutions eat memory like there's no tomorrow, and can share cached data only via IPC. So you get your scalability at the cost of a very large constant factor drop in overall performance. Some problems don't care. Some do. But for those where threading is a more appropriate solution, the Python GIL is a significant limitation. And this is a great exposition as to why. Announcing that threading should never be used just tells the world that you only work in one problem domain (probably web content generation) and haven't thought through the details elsewhere.

apu 17y ago
It's also not practical for many scenarios. Signals in python can only be received by the main thread, and since that's what multiprocessing uses for communication, it means many common scenarios are out: GUI programs where the main thread is the rendering loop, network programs where the main thread is the connection handler, etc. I've had much more luck using Python Remote Objects: http://pyro.sourceforge.net/ http://pyro.sourceforge.net/ It's trivial to make a network queue and use that for communication across different processes/machines/etc.
gte910h 17y ago
I honestly would say MOST people who reach for threads do so instinctively, not after actual analysis of what type of concurrency would serve the project they're at best. Lots of devs come from the java world or windows world where threads are done all the time but multiprocess concurrency is rare. I'd say that out of the projects I've seen people grab for threads, only about 10-15% actually need the type of heavily shared data or marginally smaller memory footprint that threading gives you. Of the rest of them, it was a toss up, or leaned towards the simplification of data structures, easier porting to multiple processors and overall less complex components that IPC gives you. Additionally, I'd contend you're doing something wrong if you see a HUGE increase when you go from threads-> multiprocessed solutions. Usually you just see a small overhead increase unless you're doing something questionable like loading data structures over and over in all processes and the like. May I recommend Stevens Unix Network Programming Vol 2: IPC to see some common designs of MP programs? You may see some large memory footprint gains by just looking through some of the programs in that book and rethinking your processing/responsibilities of your different processes. I by no means think EVERYTHING should be multiprocessed, I just think MOST things can be equally well done with threads or processes, and MANY programmers have never seen multiprocess programs, especially large multiprocess programs, where most have worked in multithreaded programs (where there are more than 1 user threads). Having worked with both, the errors you see in multiprocess programs are MUCH more traceable and manageable. Multi-threading errors can take months to debug sometimes, and weeks to even get a good reproduction scenario for. Then again, for a SMALL amount of concurrency, threading is simpler to setup and do for sure.
- ajross 17y ago
  That's a fair case, but it's not exactly a rebuttal to what I said either. If you can fit your problem into a multi-process solution (and most web stuff qualifies), then yes, there are many advantages there. But some problems benefit from both CPU parallelism and fast shared memory (think numerics work, graphics, games, data stores...). And this is an area that (due to the GIL contention visualized in the linked article) Python serves poorly. But don't fool yourself that "MOST" of anything works with any one architecture. It's a big world. And I assure you I've read Stevens.
  - illumen 17y ago
    pygame does a lot of its threaded graphics, and sound work entirely in C world, because python threads are so limited. Some of us have been using mmapd surfaces with python/pygame for years though. For graphical work, it's not all that hard to share data with python. You can use the same approaches with numpy. See this cookbook example: http://www.pygame.org/wiki/MmapSurfaces http://www.pygame.org/wiki/MmapSurfaces it's only a dozen lines of code to share data via mmap. There are lots of issues with the python, and numpy mmap modules though... and mmap is very different across platforms (eg, linux, windows, macosx have quite different behaviour). Another approach good for some graphics problems is forking. Forking is pretty fast, and it allows you to share memory in a fairly nice way. However mixing forking and threading is a quite tricky with python. This is also not so good on windows - mainly its a good method on *nix. cu,
    - gte910h 17y ago
      I'd love a good windows fork.....win7 hasn't implemented it yet I'm guessing?
  - gte910h 17y ago
    >But don't fool yourself that "MOST" of anything works with any one architecture. It's a big world. Of methods of concurrency that people use, there are two main approaches, threads, and multiprocessing. I just contend only about 15% of solutions are super clear winners for one or the other, leaving 65-70% of apps that can go either way. And I'm not a web developer, I'm an embedded software developer with years of experience in embedded video, streaming video, and hardware emulation. Lots of THAT stuff also fits just fine in either system too. As does the test systems, large analytics programs, etc that I've seen unrelated to the field. Actually python largely keeping to the inconsistency of the underlying OS with regards to mmap and other shared memory solutions is the biggest obstacles there. Oh, and the fact that hardcore numerics are still callouts to C libs in python. >And I assure you I've read Stevens. Lots and lots of people read Vol1, it seems like 1/10th as many read Vol2. Hell, I'm pretty sure most people who read Vol1 haven't a clue what's in Vol2. I'm by no means saying I want the GIL to say the same, I just think numerics work, graphics, games need threads, after that, you get to use either one (aka, 15% need threads, the other 85 need processes or can use either).