4 ms·

Good to get advanced notice, if I read all the way down, that they will silently completely change the behavior of multiprocessing in 3.14 (only on Unix/Linux,

Good to get advanced notice, if I read all the way down, that they will silently completely change the behavior of multiprocessing in 3.14 (only on Unix/Linux, in case other people wonder what’s going on), which is going to break a bunch of programs I work with. I really like using Python, but I can’t keep using it when they just keep breaking things like this. Most people don’t read all the release notes.

icegreentea2 2y ago
Not defending their specific course of action here, but you should probably try to wade into the linked discussion (https://github.com/python/cpython/issues/84559 https://github.com/python/cpython/issues/84559). Looks like the push to disable warnings (in 3.13) is mostly coming from one guy.
- CJefferson 2y ago
  I think should have a dig. While it’s not perfect, I know a few other people people who do “set up lots of data structures, including in libraries, then make use of the fact multiprocessing uses fork to duplicate them”. While fork always has sharp edges, it’s also long been clearly documented that’s the behavior on Linux.
  - int_19h 2y ago
    I'm pretty sure that significantly more people were burned by fork being the default with no actual benefit to their code, whether because of the deadlocks etc that it triggers in multithreaded non-fork-aware code, or because their code wouldn't work correctly on other platform. Keeping it there as an option that one can explicitly enable for those few cases where it's actually useful and with full understanding of consequences is surely the better choice for something as high-level as Python.
    - CJefferson 2y ago
      I agree that fork was an awful default. However, changing the default silently just means people's code is going to change behaviour between versions, or silently break if someone with an older version runs their code. At this point, it's probably better to just require people give an explicit choice (they can even make one of the choice names be 'default' or something, to make life easy for people who don't really care).
      - int_19h 2y ago
        I'm with you on undesirability of silent change of behavior. But requiring people to make an explicit choice would immediately break a lot more code, because now all the (far more numerous) instances of code that genuinely doesn't care one way or another won't run at all without changes - and note that for packages, this also breaks anyone depending on them, requiring a fix that is not even in their code. So it's downsides either way, and which one is more disruptive to the ecosystem depends on the proportion of code affected in different ways. I assume that they did look at existing Python code out in the wild to get at least an eyeball estimate of that when making the decision.
  - nyrikki 2y ago
    > posix_spawn() now accepts None for the env argument, which makes the newly spawned process use the current process environment That is the thing about fork(), spawn(), and even system() being essential wrappers around clone() in glibc and musl. You can duplicate the behavior of fork() without making the default painful for everyone else. In musl systems() calls posix_spawn() which calls clone(). All that changes is replacing a legacy call fork() that is nothing more than a legacy convenience alias with real issues and foot guns with multiple threads.
almostgotcaught 2y ago
> I really like using Python, but I can’t keep using it when they just keep breaking things like this. So much perl clutching. Just curious, since I guess you've made up your mind, what's your plan to migrate away? Or are you hoping maintainers see your comment and reconsider the road-map?
- fmajid 2y ago
  Not the person you are responding to, but my Python 3 migration plan was to move to Go for all new projects.
  - almostgotcaught 2y ago
    I'm sure your departure from the community will be the tectonic shift that'll finally get the PSF to change their course.
- CJefferson 2y ago
  Rust. I've already re-written several of my decent sized research programs to Rust, and plan to finish converting what's left soon. Rust isn't perfect (no language is), but they do seem to try much harder to not break backwards compatability.
mixmastamyk 2y ago
Unfortunately, they learned the wrong lesson from the 2->3 transition. Break things constantly instead of all at once. :p Still, this one doesn’t seem too bad. Add method=FORK now and forget about it.
nyrikki 2y ago
You are complaining about spawn()? both fork() and spawn() are just wrappers around clone() on most libc types anyway. spawn() was introduced to POSIX in the last century to address some of the problems with fork() especially related to multi threading, so I an curious how your code is so dependent on UTM, yet multi threading.
- CJefferson 2y ago
  My code isn't dependant on multi-threading at all. It use fork in Python multiprocess, because many packages can't be "pickled" (the standard way of copying data structures between processes), so instead my code looks like: * Set up big complicated data-structures. * Use fork to make a bunch of copies of my running program, and all my datastructures * Use multiprocessing to make all those python programs talk to each other and share work, thereby using all my CPU cores.
  - nyrikki 2y ago
    'Threading' is an overload term. And while I didn't know, I was wondering if at the library level, the fact that posix_spawn() pauses the parent, while fork() doesn't, that is what you were leveraging. The python multiprocessing module has been problematic for a while, as the platform abstractions are leaky and to be honest the POSIX version of spawn() was poorly implemented and mostly copied the limits of Windows. I am sure that some of the recent deadlocks are due to this pull request as an example that calls out how risky this is. https://github.com/python/cpython/pull/114279 https://github.com/python/cpython/pull/114279 Personally knowing the pain of fork() in the way you are using it, I have moved on. But I would strongly encourage you to look into how clone() and the CLONE_VM and CLONE_VFORK options interact, document your use case and file an actionable issue against the multiprocessing module. Go moved away from fork in 1.9 which may explain the issues with it better than the previous linked python discussion. But looking at the git blame, all the 'fixes' have been about people trading known problems and focusing on the happy path. My reply was intended for someone to address that tech debt and move forward with an intentional designed refactoring. As I just focus on modern Linux, I avoid the internal submodule and just call clone() in a custom module or use python as glue to languages that have better concurrency.

icegreentea2 2y ago

Not defending their specific course of action here, but you should probably try to wade into the linked discussion (https://github.com/python/cpython/issues/84559 https://github.com/python/cpython/issues/84559). Looks like the push to disable warnings (in 3.13) is mostly coming from one guy.

CJefferson 2y ago
I think should have a dig. While it’s not perfect, I know a few other people people who do “set up lots of data structures, including in libraries, then make use of the fact multiprocessing uses fork to duplicate them”. While fork always has sharp edges, it’s also long been clearly documented that’s the behavior on Linux.
- int_19h 2y ago
  I'm pretty sure that significantly more people were burned by fork being the default with no actual benefit to their code, whether because of the deadlocks etc that it triggers in multithreaded non-fork-aware code, or because their code wouldn't work correctly on other platform. Keeping it there as an option that one can explicitly enable for those few cases where it's actually useful and with full understanding of consequences is surely the better choice for something as high-level as Python.
  - CJefferson 2y ago
    I agree that fork was an awful default. However, changing the default silently just means people's code is going to change behaviour between versions, or silently break if someone with an older version runs their code. At this point, it's probably better to just require people give an explicit choice (they can even make one of the choice names be 'default' or something, to make life easy for people who don't really care).
    - int_19h 2y ago
      I'm with you on undesirability of silent change of behavior. But requiring people to make an explicit choice would immediately break a lot more code, because now all the (far more numerous) instances of code that genuinely doesn't care one way or another won't run at all without changes - and note that for packages, this also breaks anyone depending on them, requiring a fix that is not even in their code. So it's downsides either way, and which one is more disruptive to the ecosystem depends on the proportion of code affected in different ways. I assume that they did look at existing Python code out in the wild to get at least an eyeball estimate of that when making the decision.
- nyrikki 2y ago
  > posix_spawn() now accepts None for the env argument, which makes the newly spawned process use the current process environment That is the thing about fork(), spawn(), and even system() being essential wrappers around clone() in glibc and musl. You can duplicate the behavior of fork() without making the default painful for everyone else. In musl systems() calls posix_spawn() which calls clone(). All that changes is replacing a legacy call fork() that is nothing more than a legacy convenience alias with real issues and foot guns with multiple threads.

almostgotcaught 2y ago

> I really like using Python, but I can’t keep using it when they just keep breaking things like this. So much perl clutching. Just curious, since I guess you've made up your mind, what's your plan to migrate away? Or are you hoping maintainers see your comment and reconsider the road-map?

fmajid 2y ago
Not the person you are responding to, but my Python 3 migration plan was to move to Go for all new projects.
- almostgotcaught 2y ago
  I'm sure your departure from the community will be the tectonic shift that'll finally get the PSF to change their course.
CJefferson 2y ago
Rust. I've already re-written several of my decent sized research programs to Rust, and plan to finish converting what's left soon. Rust isn't perfect (no language is), but they do seem to try much harder to not break backwards compatability.

mixmastamyk 2y ago

Unfortunately, they learned the wrong lesson from the 2->3 transition. Break things constantly instead of all at once. :p Still, this one doesn’t seem too bad. Add method=FORK now and forget about it.

nyrikki 2y ago

You are complaining about spawn()? both fork() and spawn() are just wrappers around clone() on most libc types anyway. spawn() was introduced to POSIX in the last century to address some of the problems with fork() especially related to multi threading, so I an curious how your code is so dependent on UTM, yet multi threading.

CJefferson 2y ago
My code isn't dependant on multi-threading at all. It use fork in Python multiprocess, because many packages can't be "pickled" (the standard way of copying data structures between processes), so instead my code looks like: * Set up big complicated data-structures. * Use fork to make a bunch of copies of my running program, and all my datastructures * Use multiprocessing to make all those python programs talk to each other and share work, thereby using all my CPU cores.
- nyrikki 2y ago
  'Threading' is an overload term. And while I didn't know, I was wondering if at the library level, the fact that posix_spawn() pauses the parent, while fork() doesn't, that is what you were leveraging. The python multiprocessing module has been problematic for a while, as the platform abstractions are leaky and to be honest the POSIX version of spawn() was poorly implemented and mostly copied the limits of Windows. I am sure that some of the recent deadlocks are due to this pull request as an example that calls out how risky this is. https://github.com/python/cpython/pull/114279 https://github.com/python/cpython/pull/114279 Personally knowing the pain of fork() in the way you are using it, I have moved on. But I would strongly encourage you to look into how clone() and the CLONE_VM and CLONE_VFORK options interact, document your use case and file an actionable issue against the multiprocessing module. Go moved away from fork in 1.9 which may explain the issues with it better than the previous linked python discussion. But looking at the git blame, all the 'fixes' have been about people trading known problems and focusing on the happy path. My reply was intended for someone to address that tech debt and move forward with an intentional designed refactoring. As I just focus on modern Linux, I avoid the internal submodule and just call clone() in a custom module or use python as glue to languages that have better concurrency.