7 ms·

Python multiprocessing is different under Linux and Windows

36 points by gthank 16y ago

deleted 16y ago
[deleted]
- pmjordan 16y ago
  I suspect it's not exactly trivial. I haven't worked with IPC in Python, but on the C level, it's a bit easier to communicate between processes related via a fork() than separately spawned processes. I suspect adding exec() support on the linux side would affect a large number of small details in the existing code. Not something you'd complete in the duration of writing a short blog post.
- tdmackey 16y ago
  Writing a patch for this is an incredible undertaking, writing a blog to draw attention to the issue is not. The author is a user of python that is aware of the issue, but is not one who has contributed to the language. This is probably well outside of his expertise as the task is systems programming in C and he is a python web developer.
  - kingkilr 16y ago
    Brandon may be a professional web developer, but at PyCon he gave a talk about the internal algorithms of the dict, he knows his stuff. It's just, as Jesse points out, a lot of work.
- gthank 16y ago
  Whether you meant it to be rude or not, it pretty much came off that way. When somebody takes the time to point out a legitimate shortcoming, much less do so in a way that isn't snarky or mean-spirited, criticizing that person for not also solving the problem is a net-negative response. There are any number of reasons that someone might not solve the problem. In no particular order, here are some off the top of my head: * they don't have the resources for testing (not everyone has a copy of Windows, much less every supported version of Windows) * they don't have the expertise * they don't have the time * they don't care enough, but thought it would be useful to write it down someplace for the next time they come across the problem Asking for a patch every time somebody provides useful feedback is a fantastic way to make sure you don't maximize the community around your project.
  - jnoller 16y ago
    Hi, I am that jnoller. And no, I wasn't trying to be rude. Yes, I want it too, I agree with Brandon, however I do not have the time to do so. He asked for a feature; I told him I agree, but that I don't personally have the time for a patch. No drama here.
    - gthank 16y ago
      Sorry - that was aimed at the comment that appears to have been deleted, not at you. Based on your blog, etc., I was pretty sure you actually meant "Hey, if you get me a working patch, it will get in". The comment, OTOH, was pretty clearly an indictment of Brandon for not writing a patch.
      - jnoller 16y ago
        Yup, looks like it was deleted. I totally understand it when people don't have the time to submit patches :). I have almost no time lately, and way too many projects I want to be part of so I'm the last person to ask people to spend time they might not have.
    - deleted 16y ago
      [deleted]
- mattyb 16y ago
  * Noller. From the post: By figuring out how to get locks, queues, and shared data structures all working cleanly on such different architectures, the multiprocessing authors save Python programmers out on the street like me from reinventing a dozen wheels when we need to support multi-platform concurrency. From the ticket: Therefore, the "multiprocessing" module should offer an option under Linux that ignores the advantage of being able to do a bare fork() and instead spins up a new interpreter instance just like Windows does. Maybe he doesn't know how to write such a patch? Also, damn, the Windows implementation is waaaaaaaay longer. Unix: http://code.google.com/p/python-multiprocessing/source/browse/trunk/Lib/multiprocessing/forking.py?spec=svn73&r=25#72 http://code.google.com/p/python-multiprocessing/source/brows... Windows: http://code.google.com/p/python-multiprocessing/source/browse/trunk/Lib/multiprocessing/forking.py?spec=svn73&r=25#144 http://code.google.com/p/python-multiprocessing/source/brows...
  - jnoller 16y ago
    The python-multiprocessing source on google code is not canonical fwiw, you're better viewing the python.org svn version.
    - mattyb 16y ago
      Yea just realized it's a backport, I'm looking at the python.org version now.
    - mattyb 16y ago
      http://svn.python.org/view/python/tags/r265/Modules/_multiprocessing/ http://svn.python.org/view/python/tags/r265/Modules/_multipr... Am I looking in the right place?
      - jnoller 16y ago
        yup, close enough, though here's trunk: http://svn.python.org/view/python/trunk/Lib/multiprocessing/ http://svn.python.org/view/python/trunk/Lib/multiprocessing/
- deleted 16y ago
  [deleted]
deleted 16y ago
[deleted]
jnoller 16y ago
I said this in a reply to another comment, but it's worth repeating. Yes, brandon is right, there is a difference, albeit an intentional one. I agree with him that having an option to force this behavior to match across platforms is desirable, however, I do not currently have the time to implement it myself. So, in short; I agree, patch welcome or we wait until I or someone else has the time to do it.
rbanffy 16y ago
Someone should call Microsoft and report a bug on how Windows handles child processes. This seems serious ;-) Can anyone check if this problem also exists in the Cygwin version of Python? It's closer to the Unix source and should behave in a more unix-ish fashion. I am glad I no longer can ;-) I always install Cygwin when I work from a Windows box. It gives me a varnish layer of civility on what I feel is a mostly barbaric environment. That said, the POSIX subsystem NT had should be able to handle proper process forking. I am not sure how easy (or not) would it be to run Python on it, but that should also work.
- alextingle 16y ago
  NT's POSIX subsystem never supported fork(). They, um, overlooked that bit...
  - daeken 16y ago
    Actually, that's incorrect. Both the old-school POSIX subsystem and the Interix (Services For Unix) subsystem support a fork system call with copy-on-write semantics. In fact, the NT kernel itself can fork processes trivially, but this isn't exposed in the Win32 subsystem, like most of the fun stuff on NT.
    - rbanffy 16y ago
      I always found it weird having a bunch of Windows kernel operations that are not exposed to Windows apps. How would you call those?
      - daeken 16y ago
        Ntdll.dll contains the userspace side of the NT API. Specifically, Zw* and Nt* contain most of the APIs that map directly into the kernel. Then there's the Rtl* stuff which provides a lot of fun functions for manipulating processes, and the Ldr* stuff for loading binaries; Ldr is a lot of fun to hook into, if you want to manipulate binaries with a high degree of transparency. Edit: One thing to note, though, is that it's easy to subvert the Win32 subsystem by using these APIs directly. I.e. if you want to fork a Win32 process, you're going to need to not only fork it the standard NT way, but then tell CSRSS what you're actually doing. It's easy to run into a situation where the Win32 subsystem won't play nicely with you, because it just doesn't have enough information.
blub 16y ago
He should stop trying to test for Windows behaviour on Linux and use a VM. Having the multiprocessing module expose an option for turning off forking would be useless, as Windows is not UNIX and there are many other subtle or not so subtle differences.
- neilc 16y ago
  If multiprocessing exposed such an option, it would effectively be harder to write non-portable code, which seems like a good thing (to do optionally). You could then switch on "no, do a sensible fork(2)" purely as a performance optimization. Seems like a pretty obviously good thing to add.
demosthe 16y ago
Shared resources across multiple concurrent processes is not a good way to ensure a full night's sleep. I would follow the documentation's advice and pass arguments to the child process and not try and depend on the state of global objects.
- mkramlich 16y ago
  Agreed. I think the #1 rule of multi-platform support is to NOT support multiple platforms. Been there, done that, you won't like the tee-shirt. But if you "must" do it, let the implementation/agent for each platform be as different as they need to be, with no shared mutable memory, and just pass messages.

36 points by gthank 16y ago

deleted 16y ago

[deleted]

pmjordan 16y ago
I suspect it's not exactly trivial. I haven't worked with IPC in Python, but on the C level, it's a bit easier to communicate between processes related via a fork() than separately spawned processes. I suspect adding exec() support on the linux side would affect a large number of small details in the existing code. Not something you'd complete in the duration of writing a short blog post.
tdmackey 16y ago
Writing a patch for this is an incredible undertaking, writing a blog to draw attention to the issue is not. The author is a user of python that is aware of the issue, but is not one who has contributed to the language. This is probably well outside of his expertise as the task is systems programming in C and he is a python web developer.
- kingkilr 16y ago
  Brandon may be a professional web developer, but at PyCon he gave a talk about the internal algorithms of the dict, he knows his stuff. It's just, as Jesse points out, a lot of work.
gthank 16y ago
Whether you meant it to be rude or not, it pretty much came off that way. When somebody takes the time to point out a legitimate shortcoming, much less do so in a way that isn't snarky or mean-spirited, criticizing that person for not also solving the problem is a net-negative response. There are any number of reasons that someone might not solve the problem. In no particular order, here are some off the top of my head: * they don't have the resources for testing (not everyone has a copy of Windows, much less every supported version of Windows) * they don't have the expertise * they don't have the time * they don't care enough, but thought it would be useful to write it down someplace for the next time they come across the problem Asking for a patch every time somebody provides useful feedback is a fantastic way to make sure you don't maximize the community around your project.
- jnoller 16y ago
  Hi, I am that jnoller. And no, I wasn't trying to be rude. Yes, I want it too, I agree with Brandon, however I do not have the time to do so. He asked for a feature; I told him I agree, but that I don't personally have the time for a patch. No drama here.
  - gthank 16y ago
    Sorry - that was aimed at the comment that appears to have been deleted, not at you. Based on your blog, etc., I was pretty sure you actually meant "Hey, if you get me a working patch, it will get in". The comment, OTOH, was pretty clearly an indictment of Brandon for not writing a patch.
    - jnoller 16y ago
      Yup, looks like it was deleted. I totally understand it when people don't have the time to submit patches :). I have almost no time lately, and way too many projects I want to be part of so I'm the last person to ask people to spend time they might not have.
  - deleted 16y ago
    [deleted]
mattyb 16y ago
* Noller. From the post: By figuring out how to get locks, queues, and shared data structures all working cleanly on such different architectures, the multiprocessing authors save Python programmers out on the street like me from reinventing a dozen wheels when we need to support multi-platform concurrency. From the ticket: Therefore, the "multiprocessing" module should offer an option under Linux that ignores the advantage of being able to do a bare fork() and instead spins up a new interpreter instance just like Windows does. Maybe he doesn't know how to write such a patch? Also, damn, the Windows implementation is waaaaaaaay longer. Unix: http://code.google.com/p/python-multiprocessing/source/browse/trunk/Lib/multiprocessing/forking.py?spec=svn73&r=25#72 http://code.google.com/p/python-multiprocessing/source/brows... Windows: http://code.google.com/p/python-multiprocessing/source/browse/trunk/Lib/multiprocessing/forking.py?spec=svn73&r=25#144 http://code.google.com/p/python-multiprocessing/source/brows...
- jnoller 16y ago
  The python-multiprocessing source on google code is not canonical fwiw, you're better viewing the python.org svn version.
  - mattyb 16y ago
    Yea just realized it's a backport, I'm looking at the python.org version now.
  - mattyb 16y ago
    http://svn.python.org/view/python/tags/r265/Modules/_multiprocessing/ http://svn.python.org/view/python/tags/r265/Modules/_multipr... Am I looking in the right place?
    - jnoller 16y ago
      yup, close enough, though here's trunk: http://svn.python.org/view/python/trunk/Lib/multiprocessing/ http://svn.python.org/view/python/trunk/Lib/multiprocessing/
deleted 16y ago
[deleted]

deleted 16y ago

[deleted]

jnoller 16y ago

I said this in a reply to another comment, but it's worth repeating. Yes, brandon is right, there is a difference, albeit an intentional one. I agree with him that having an option to force this behavior to match across platforms is desirable, however, I do not currently have the time to implement it myself. So, in short; I agree, patch welcome or we wait until I or someone else has the time to do it.

rbanffy 16y ago

Someone should call Microsoft and report a bug on how Windows handles child processes. This seems serious ;-) Can anyone check if this problem also exists in the Cygwin version of Python? It's closer to the Unix source and should behave in a more unix-ish fashion. I am glad I no longer can ;-) I always install Cygwin when I work from a Windows box. It gives me a varnish layer of civility on what I feel is a mostly barbaric environment. That said, the POSIX subsystem NT had should be able to handle proper process forking. I am not sure how easy (or not) would it be to run Python on it, but that should also work.

alextingle 16y ago
NT's POSIX subsystem never supported fork(). They, um, overlooked that bit...
- daeken 16y ago
  Actually, that's incorrect. Both the old-school POSIX subsystem and the Interix (Services For Unix) subsystem support a fork system call with copy-on-write semantics. In fact, the NT kernel itself can fork processes trivially, but this isn't exposed in the Win32 subsystem, like most of the fun stuff on NT.
  - rbanffy 16y ago
    I always found it weird having a bunch of Windows kernel operations that are not exposed to Windows apps. How would you call those?
    - daeken 16y ago
      Ntdll.dll contains the userspace side of the NT API. Specifically, Zw* and Nt* contain most of the APIs that map directly into the kernel. Then there's the Rtl* stuff which provides a lot of fun functions for manipulating processes, and the Ldr* stuff for loading binaries; Ldr is a lot of fun to hook into, if you want to manipulate binaries with a high degree of transparency. Edit: One thing to note, though, is that it's easy to subvert the Win32 subsystem by using these APIs directly. I.e. if you want to fork a Win32 process, you're going to need to not only fork it the standard NT way, but then tell CSRSS what you're actually doing. It's easy to run into a situation where the Win32 subsystem won't play nicely with you, because it just doesn't have enough information.

blub 16y ago

He should stop trying to test for Windows behaviour on Linux and use a VM. Having the multiprocessing module expose an option for turning off forking would be useless, as Windows is not UNIX and there are many other subtle or not so subtle differences.

neilc 16y ago
If multiprocessing exposed such an option, it would effectively be harder to write non-portable code, which seems like a good thing (to do optionally). You could then switch on "no, do a sensible fork(2)" purely as a performance optimization. Seems like a pretty obviously good thing to add.

demosthe 16y ago

Shared resources across multiple concurrent processes is not a good way to ensure a full night's sleep. I would follow the documentation's advice and pass arguments to the child process and not try and depend on the state of global objects.

mkramlich 16y ago
Agreed. I think the #1 rule of multi-platform support is to NOT support multiple platforms. Been there, done that, you won't like the tee-shirt. But if you "must" do it, let the implementation/agent for each platform be as different as they need to be, with no shared mutable memory, and just pass messages.