4 ms·

multiprocessing needs to run copies of your program that are sufficiently initialised that they can execute the function, yet no initialisation code must be run

multiprocessing needs to run copies of your program that are sufficiently initialised that they can execute the function, yet no initialisation code must be run that should not be run multiple times. That means you either use fork - which is a major can of worms for a reusable library to use. Or you write something like this in your entry point module: if __name__=='__main__': multiprocessing.freeze_support() once_only_application_code() Suppose I don't realise that your library is using multiprocessing, and I carelessly call it from this two-line script: import library_that_uses_multiprocessing_internally library_that_uses_multiprocessing_internally.do_stuff() That's basically a fork bomb. And where do you put the multiprocessing.set_start_method call? Surely not in the library.

coldtea 3y ago
>multiprocessing needs to run copies of your program that are sufficiently initialised that they can execute the function, yet no initialisation code must be run that should not be run multiple times. Huh? As far as I remember multiprocessing just sends pickled versions of the function to run and any of its dependencies (other functions, closures, etc). As long as the function doesn't use global state that's not available when pickled, it's fine. But it doesn't re-initialize your whole program for each process in the pool. >That means you either use fork - which is a major can of worms for a reusable library to use. How did we get into a reusable library authoring? Yes, multiprocessing is not just turnkey to use inside a reusable library you make. But the context were programs here, or not? >Or you write something like this in your entry point module: Hmmm? This is to have it support freezing the script (that is, using a tool to make it a distributable, like PyInstaller). That's not necessarily a use case most have.
- munch117 3y ago
  > How did we get into a reusable library authoring? That was always my premise. Maybe I didn't make it clear enough, because I tend to just take it for granted that that's how you write code, in a style that's suited for reuse. > But the context were programs here, or not? That's the "global" I was talking about: Code that's using multiprocessing needs to know the context that it's embedded in. Any moment I might grab that piece of code and transfer it to a library of reusable components, because that's how I work - code that starts out as part of a standalone program doesn't necessarily stay that way. Multiprocessing gets in the way of that.
  - coldtea 3y ago
    >That was always my premise. Maybe I didn't make it clear enough, because I tend to just take it for granted that that's how you write code, in a style that's suited for reuse. That's somewhat condescending. You can write code "in a style that's suitable for reuse" without being a library author - well, without publishing public packages anyway. Re-use is not only about some totally generic package that can run under any arbitrary context willy nilly. And of course there are tons of programs where the parts don't make sense as libraries, because they're tied to the specific functionality and overall design (whether because of the domain logic required or due to optimization or other constraints). You write them to be modular and clean, but not with "arbitrary people running my code in whatever context" in mind. Not to mention the mountains of purpose-specific throwaway scripts, e.g. in the scientific community especially, where Python is big, there's little regard for reuse (even less so for library building), and it's not because multiprocess is stopping them :) So, yeah, I'd say, even if not 100% suitable for generic reusable library-style code, it doesn't mean multiprocess can't be applied in a huge number of specific people's problems and codebases. >Code that's using multiprocessing needs to know the context that it's embedded in. If you want to speed up your Python program and there's something that can run in parallel with no shared state, you can use multiprocess to run it. If having it as a "reusable component" that hides away the fact multiprocess is used, and that can be called in any arbitrary context, is your concern, it's a valid one, but then perhaps a specific Python program and its performance is not your main priority. Library writing is, instead :) Else, it's enough that the user calling multiprocessing knows the function that is to be passed and its dependencies (or lack thereof). Other than that, they don't have to change their top level program's architecture.
    - munch117 3y ago
      I didn't mean to say that no one should ever use multiprocessing. I was laying out the reasons why I don't. I'm really looking forward to subinterpreters. I think they have great potential for supporting a style of multiprocessing that is both faster and better isolated.

coldtea 3y ago

>multiprocessing needs to run copies of your program that are sufficiently initialised that they can execute the function, yet no initialisation code must be run that should not be run multiple times. Huh? As far as I remember multiprocessing just sends pickled versions of the function to run and any of its dependencies (other functions, closures, etc). As long as the function doesn't use global state that's not available when pickled, it's fine. But it doesn't re-initialize your whole program for each process in the pool. >That means you either use fork - which is a major can of worms for a reusable library to use. How did we get into a reusable library authoring? Yes, multiprocessing is not just turnkey to use inside a reusable library you make. But the context were programs here, or not? >Or you write something like this in your entry point module: Hmmm? This is to have it support freezing the script (that is, using a tool to make it a distributable, like PyInstaller). That's not necessarily a use case most have.

munch117 3y ago
> How did we get into a reusable library authoring? That was always my premise. Maybe I didn't make it clear enough, because I tend to just take it for granted that that's how you write code, in a style that's suited for reuse. > But the context were programs here, or not? That's the "global" I was talking about: Code that's using multiprocessing needs to know the context that it's embedded in. Any moment I might grab that piece of code and transfer it to a library of reusable components, because that's how I work - code that starts out as part of a standalone program doesn't necessarily stay that way. Multiprocessing gets in the way of that.
- coldtea 3y ago
  >That was always my premise. Maybe I didn't make it clear enough, because I tend to just take it for granted that that's how you write code, in a style that's suited for reuse. That's somewhat condescending. You can write code "in a style that's suitable for reuse" without being a library author - well, without publishing public packages anyway. Re-use is not only about some totally generic package that can run under any arbitrary context willy nilly. And of course there are tons of programs where the parts don't make sense as libraries, because they're tied to the specific functionality and overall design (whether because of the domain logic required or due to optimization or other constraints). You write them to be modular and clean, but not with "arbitrary people running my code in whatever context" in mind. Not to mention the mountains of purpose-specific throwaway scripts, e.g. in the scientific community especially, where Python is big, there's little regard for reuse (even less so for library building), and it's not because multiprocess is stopping them :) So, yeah, I'd say, even if not 100% suitable for generic reusable library-style code, it doesn't mean multiprocess can't be applied in a huge number of specific people's problems and codebases. >Code that's using multiprocessing needs to know the context that it's embedded in. If you want to speed up your Python program and there's something that can run in parallel with no shared state, you can use multiprocess to run it. If having it as a "reusable component" that hides away the fact multiprocess is used, and that can be called in any arbitrary context, is your concern, it's a valid one, but then perhaps a specific Python program and its performance is not your main priority. Library writing is, instead :) Else, it's enough that the user calling multiprocessing knows the function that is to be passed and its dependencies (or lack thereof). Other than that, they don't have to change their top level program's architecture.
  - munch117 3y ago
    I didn't mean to say that no one should ever use multiprocessing. I was laying out the reasons why I don't. I'm really looking forward to subinterpreters. I think they have great potential for supporting a style of multiprocessing that is both faster and better isolated.