10 ms·

Having ported Ruby to IBM's Blue Gene/L my advice is to forget about the GIL. Run one Python process per core. Use something like MPI2 for message passing commu

• kevingadd 9y ago
  The post addresses this strategy and describes why they consider it insufficient. You can already do this in Python, anyway.
  • sametmax 9y ago
    Yes but they discard multiple interpreters as having no real advantages. This is dishonest since it should be able to share objects with much, much less overhead as multiprocessing, while allowing to use multiple CPU. It's not perfect, but honestly it seems like a very good deal for Python.
• metalliqaz 9y ago
  I agree wholeheartedly. Almost every time I hear from someone who is upset about the GIL, I find that they would be much better suited to using multiprocessing instead of multithreading. With 80% of the developers out there, they are basically assured of producing better, more stable code this way.
  • pslam 9y ago
    Absolutely agree. Almost all tasks will perform very well when using multiprocessing. It also has a nice side-effect of steering you towards explicitly coding data flows without fine-grained sharing. If you need to close that gap between the performance of multiprocessing, and multithreading, then you probably shouldn't be using Python, or any language of the same shape, in the first place. There is one other option I'd like to see: multiprocessing style, but with multiple Python interpreter instances in the same process — one per thread. There would still be the hard delineation of data boundaries between instances, but less overhead for pushing data between them.
    • metalliqaz 9y ago
      What would that get you?
    • weberc2 9y ago
      > If you need to close that gap between the performance of multiprocessing, and multithreading, then you probably shouldn't be using Python, or any language of the same shape, in the first place. Unfortunately, these performance concerns often manifest well after the "rewrite it in a different language" date has expired. There are a lot of people in that boat, and they need better options. > There is one other option I'd like to see: multiprocessing style, but with multiple Python interpreter instances in the same process — one per thread. There would still be the hard delineation of data boundaries between instances, but less overhead for pushing data between them. If I understand correctly, the article discusses this ("subinterpreters"), but claims that there is no advantage to this approach vs multiprocessing. Presumably any overhead savings are eaten by GIL contention or some such?
      • dom0 9y ago
        > There are a lot of people in that boat, and they need better options. Land isn't coming to you, folks, you must start rowing if you want to get there. Rewrite bit for bit. Module for module. Package for package.
        • weberc2 9y ago
          Sounds like you're saying this is infeasible; care to explain why?
          • funkymike 9y ago
            I took it to mean that it is feasible. Instead of saying "well we used the wrong language, I guess we're screwed," you rewrite one component at a time, piece by piece, until the whole has been replaced. This is the approach I try to use myself. It's nearly impossible to replace an entire system all at once. But replacing one part at a time is doable and you can see the improvements much sooner.
  • bb101 9y ago
    I can see how using multiprocessing trumps threads for smaller programs. However it can become memory inefficient to have larger programs running in multiple processes, especially on servers with less resources.
    • njharman 9y ago
      If I run N versions of program that occupies 8mb of memory the memory footprint of the code is much less than N*8mb due to shared libraries/memory pages. It's a factor, sure. But, one you should weigh with other factors to determine what is best.
  • optimusclimb 9y ago
    Except when your use case requires a massive shared data cache that needs to be atomically updated.
    • zepolen 9y ago
      Of which there are plenty well defined ones to do the job already, and as a plus they can communicate with any language not just Python.
    • omarforgotpwd 9y ago
      Redis could help. Obviously not perfect for every use case but covers many of them.
      • optimusclimb 9y ago
        It doesn't if you need to manage atomic data across the processes, as there's no way to lock and block the other cache consumers (think the data you need to handle cache evictions, etc.) Also, you're describing multiple python processes + an extra server (redis) process - as a "simpler" solution for the limitation that Python doesn't do multi-threads well. Of course there are a ton of use cases out there where you can scale in other ways, but threads and shared memory exist for a reason - there's no reason not to call a spade a spade and say the GIL is still a limitation.
        • tinix 9y ago
          Blocking workers in a Redis queue is not hard... You can simply put them all on a pubsub control channel and then orchestrate them that way you need to do shit. Or literally just take down the processes, or the network, so they disconnect and stop BLPOPing the queue. Cache evictions can be handled by Redis natively with TTL. For retries and failure mitigation, you can still lean on Redis via BRPOPLPUSH/RPOPLPUSH. If you want to scale beyond one machine, you can't rely on threading to help you. So why not just do it right to begin with, and use a parallel worker queue? It's not a matter of the GIL being a limitation, a single machine is a limitation too. Don't blame your tools because you're misusing them. As for threading in Python... on a single machine, for one reason or another... I would still rather use multiple processes, or at the very least, would just simply use eventlet and greenthreads. Not saying it covers all use cases, it's not a silver bullet, and it doesn't replace threading natively, but damnit, it scales better, and it's the right way to do the task at hand.
  • agumonkey 9y ago
    hettinger said that multiprocess used pickle for every communication and that it must be accounted for when optimizing
  • Animats 9y ago
    Python's "multiprocessing" means launching another Python interpreter in a subprocess. Each process has a full copy of the Python environment. They may share the base interpreter, but there's a separate copy of every package loaded and all data. Memory consumption is bloated and the CPU caches thrash. Launching a subprocess is expensive; it means a full interpreter launch and a recompile/reload. "Multiprocessing" is useful when you have a lot of work to do concurrently and not too much data to pass between processes. I've used Python subprocesses that way. Parallelizing your number crunching is probably not going to work very well.
    • peterkelly 9y ago
      If CPU load is an issue, why would you be using an interpreter in the first place?
      • dsfyu404ed 9y ago
        Just because those resources exist does not mean you get to park your 1997 Chevy Cavalier diagonal across three parking spaces. I'm not going to run your code on my server if your code uses resources so poorly that I can't run other things I want to run on my server.
        • DigitalJack 9y ago
          You are getting downvotes, I suspect, because your comment makes no sense in the context of the post you replied to. Perhaps you meant to reply to GP?
      • zerkten 9y ago
        It seems like people never make this assessment, or use the GIL argument to put interpreted languages down. I personally run into I/O bound problems way more often than CPU bound ones. That said, I'm mainly doing things in the realm of a Python web developer. Scientists probably hit CPU bound problems more often with Python, but seem to drop down to C/C++ extensions without needing to complain about the problems.
      • 9y ago
  • glic3rinu 9y ago
    Lately I've found out that multiprocessing will not help you if your program is multithreaded. There is no sane way of forking a multithreaded program. For one, the child process will inherit a copy of all locks in the state they where at forking time, possibly causing random crashes and deadlocks.
    • lstyls 9y ago
      > There is no sane way of forking a multithreaded program The sane way of forking a multithreaded process is to exec immediately after.
      • dom0 9y ago
        It is possible to do more after a fork (cf. async-signal-safe), but it's hairy enough to just say — don't, always exec (similar to how doing actual work in a signal handler is generally a very bad idea).
    • d0mine 9y ago
      that is why 'forkserver' start method exists. https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods https://docs.python.org/3/library/multiprocessing.html#conte...
    • hyperbovine 9y ago
      If the child program is multithreaded then it's almost certainly not pure Python in the first place. So, wrap it up in `with nogil:` Cython statements and use the threading module (or concurrent.futures.ThreadPoolExecutor).
  • slantedview 9y ago
    > multiprocessing instead of multithreading There's a reason threads exist.
    • gnaritas 9y ago
      Those reasons aren't what they used to be, resources aren't nearly as limited these days and we now have the hindsight to see that threads lead to very buggy code due to shared state. Processes are better.
  • ericfrederich 9y ago
    The GIL is a legit pain when dealing with GUIs. When you're jumping between C/C++ code and Python code you don't care much about the GIL... until you have a GUI which needs to be kept responsive and needs the GIL to do so.
    • simonh 9y ago
      Ive done a fair bit of GUI development in Python, mainly using Qt and not hit any significant responsiveness issues. The multi-threading support in Python is perfectly fine for providing responsive switching between activities and event loops, as long as you don't have anything that locks hard for too long. But in that case you can always split that off into a separate process e.g. The way browsers nowadays run a process per tab.
  • kerkeslager 9y ago
    Your parent comment gives good advice, because the GIL is probably here to stay and so there's no use complaining about it. But the idea that multiprocessing gives better results than multithreading is ridiculous. In languages which don't have a GIL, threads are almost as capable as processes, but lighter weight. Threads are almost always preferable to processes in most languages. I understand why the GIL is still around, and don't necessarily support removing it, but it's definitely not there because it produces "better, more stable [Python] code".
    • gnaritas 9y ago
      > In languages which don't have a GIL, threads are almost as capable as processes, but lighter weight. But also plagued with shared state concurrency bugs, something multi-processing completely avoids so... > Threads are almost always preferable to processes in most languages. No, they aren't. It's too easy to write buggy code with threads, it's a flawed model. Now it's certainly true that more people choose threads than processes but that's because they vastly overestimate their ability to write bug free lock based code. Processes are better.
      • kerkeslager 9y ago
        1. In Python, Python provides mechanisms for communicating between processes. Literally the exact same mechanisms can be used to communicate between threads. So I'm not sure why you think processes are inherently safer than threads. 2. If we're taking about all languages, I'm really just not sure why you would assume threads imply locks. There are a ton of threading models out there which don't rely on explicit locking, and there are even some that don't use locking, period.
        • gnaritas 9y ago
          > So I'm not sure why you think processes are inherently safer than threads. Because they remove the unsafe way of sharing state from the programmer. The issue isn't that state can be shared correct in threads, it's that it doesn't have to be done correctly and programmers are simply terrible at doing it right. > There are a ton of threading models out there which don't rely on explicit locking, and there are even some that don't use locking, period. It's not about locks, it's about shared mutable state. Programmers are bad at dealing with shared mutable state, regardless of how access is synchronized.
  • captainmuon 9y ago
    If you have long running, computationally intensive code, with simple interactions, sure. Then multiple processes is the right thing. But sometimes you are writing a GUI app, or some "real time" code [1]. You put blocking calls onto a different thread to keep the UI responsive. But then you find that still the blocking calls freeze the UI across threads due to the GIL. Pure Python code is not the problem in this case - the GIL gets released between statements often enough. It is long running C code. You could release the GIL manually in there, but it is not done everywhere. Also, there are often calls that are supposed to be instant (like opening a file, or starting an async operation), that take seconds under bad conditions (when the network is down). ---- [1] well, with Python probably not in the strict definition of real time, but say you are controlling some external device
• omarforgotpwd 9y ago
  Yes, multi-processing is much easier anyway. Not to mention how complicated, not backwards compatible and thorny trying to get rid of the GIL is...
• carapace 9y ago
  I can't tell you how happy I was to see your comment at the top of this discussion. Relevant: "Python is Only Slow If You Use it Wrong" http://apenwarr.ca/diary/2011-10-pycodeconf-apenwarr.pdf http://apenwarr.ca/diary/2011-10-pycodeconf-apenwarr.pdf
• sbeckeriv 9y ago
  I would like to know more.
• dguaraglia 9y ago
  Whoa, thanks for bringing up MPI2, you might have just saved me a lot of painstaking development with the mmap and multiprocessing libraries.
• zeptomu 9y ago
  I am a heavy user of Python and its scientific libraries (numpy, etc.), and although I know about the GIL, I have to add, that for us (we do a lot of scientific code-prototyping to evaluate remote sensing processing methods) the GIL hasn't been a problem so far. E.g. in the remote sensing and earth observation domain you can simply divide your problem (e.g. semantic segmentation) into (maybe over-lapping) subproblems (via e.g. tiling) and start separate processes for each image processing tool-chain. Granted you may not utilize your resources to the full extent by only applying multiprocessing (and ignoring threading), but in my experience you can solve a lot of problems by simply applying map-reduce-like programs and optimizing for throughput.
• astrodust 9y ago
  Counterpoint: Threads in JRuby work as threads should work. No GIL. No grinding of gears. Multi-process is just one form of concurrency, and it's not always the best one.
• cshenton 9y ago
  Thats not an option when you want to do something like reinforcement learning with lock free updates. In that case, the networks are small enough that you want to use the CPU, but learning sensitive enough that you don't want multiple copies of the network getting out of sync. Then you absolutely need multiple cores sharing memory.