4 ms·

>However, in CPython the GIL means that we don't have parallelism, except through multiprocessing which requires trade-offs. I can't speak for anybody else, bu

>However, in CPython the GIL means that we don't have parallelism, except through multiprocessing which requires trade-offs. I can't speak for anybody else, but I personally haven't felt that inconvenienced by these trade offs at all. The major one is needing to pickle objects before sending them back and forth between processes. Since I prefer to keep the thread interfaces as tight as possible anyway, this only ended up being a major problem once - when I was trying to pickle a stacktrace before sending it to the parent process.

ant6n 11y ago
multiprocessing can be fickle. In ipython, ctrl-c during a computation using multiprocessing results in general unhappiness, and will probably be followed by a 'killall python'
- vosper 11y ago
  Oh, yeah - and this one! Much more so than the inconvenience I posted!
- crdoconnor 11y ago
  I've had this problem too, but after doing the (quite considerable) leg-work to fix it involving messing around with process groups and the ridiculous number of different types of signal, I'm more inclined to blame the horrible 30 year old UNIX APIs than python. I've had this signal handling problem in single process/single threaded code too, and in other languages. True, it would be nice if python layered on a nicer set of APIs to deal with the nastiness.
  - logicchains 11y ago
    >I'm more inclined to blame the horrible 30 year old UNIX APIs than python. Datapoint: I encountered the same thing on Windows, having to use the Powershell equivalent of `killall python` to deal with it.
  - easytiger 11y ago
    No, this is pretty easy in c. i think the python docs even apologise for it somewhere.
vosper 11y ago
I was inconvenienced by multiprocessing just today: We were using it to spin up a bunch of workers, and have spent some time debugging an issue with SQLAlchemy, which requires special management when forking: "it’s usually required that a separate Engine be used for each child process. This is because the Engine maintains a reference to a connection pool that ultimately references DBAPI connections - these tend to not be portable across process boundaries." [1] So you have to be careful when dealing with forking and databases, which has lead us to subclass multiprocessing.Process specifically for our (common) case of wanting to continue to use the session object in the child process without having to think about recycling the Engine object. It was that code we have been trying to debug today, because in some specific cases we still run into issues (yes, we read the docs). Also, when people (usually new engineers) don't know about this they usually blindly use the multiprocessing module directly (and who can blame them) and end up spend some time debugging intermittent connectivity issues until someone says "oh, you should use the xyz module, it handles the DB stuff for you". [1] http://docs.sqlalchemy.org/en/rel_0_9/core/connections.html#basic-usage http://docs.sqlalchemy.org/en/rel_0_9/core/connections.html#...
- crdoconnor 11y ago
  >it’s usually required that a separate Engine be used for each child process. Yeah, provided you do this there should be no headache. >So you have to be careful when dealing with forking and databases, which has lead us to subclass multiprocessing.Process specifically for our (common) case of wanting to continue to use the session object in the child process without having to think about recycling the Engine object. What use case led to this being a non-negotiable requirement?
  - vosper 11y ago
    It's not non-negotiable, it's for safety and convenience and DRY: We want our engineers to be able to use the database in their parallel processing jobs without having to understand the internals of SQLAlchemy, Engines, and forking.
- jperras 11y ago
  You probably know this already, but if you're using (or are amenable to using) uwsgi for managing your workers, there's an option[1] that can control whether your application code is loaded before or after fork() is called. I've run into the exact issue you've described; the --lazy-apps solution may be slightly more resource intensive (since you maintain N copies of the application in memory), but ends up being much nicer to reason about. 1. http://uwsgi-docs.readthedocs.org/en/latest/articles/TheArtOfGracefulReloading.html#preforking-vs-lazy-apps-vs-lazy http://uwsgi-docs.readthedocs.org/en/latest/articles/TheArtO...
Kalium 11y ago
I have. I've personally worked on a project that had to be significantly refactored because the original author didn't understand the implications of GIL.
yummyfajitas 11y ago
Other major inconveniences I've run into: Collecting metrics require using an external collector process (statsd) rather than just an in-proc lib (e.g., codahale metrics on the JVM). Issues generating random numbers - I ran 8M simulations on 8 cores, but because the random seed was forked, I actually only ran 1M simulations 8 times. (Easy enough to fix, but annoying.) Deduplication is really handy in reducing memory usage - why would you want the same (immutable) object duplicated N times? And of course various issues with resources - you need to be careful only to connect to the DB/etc after forking otherwise stuff gets weird. And of course, this means that instead of sharing a single connection pool, you've now got one DB connection per proc, which may or may not be a big waste of resources.
- semi-extrinsic 11y ago
  > Issues generating random numbers - I ran 8M simulations on 8 cores, but because the random seed was forked, I actually only ran 1M simulations 8 times. (Easy enough to fix, but annoying.) IIRC, even if you do your easy fix of giving the 8 threads different seeds, your results are not trustworthy since the parallell random number streams are likely to exhibit significant correlations. There's quite a bit of algorithm research behind quality PRNGs.
mzs 11y ago
I gave-up on multiprocessing a long time ago. I was happily writing code that used it on linux and then tried running it on FreeBSD. Turned-out that it worked basically because of nuances of glibc but was totally not fork safe. The most simple things broke. I don't know if they have ever been addressed.

ant6n 11y ago

multiprocessing can be fickle. In ipython, ctrl-c during a computation using multiprocessing results in general unhappiness, and will probably be followed by a 'killall python'

vosper 11y ago
Oh, yeah - and this one! Much more so than the inconvenience I posted!
crdoconnor 11y ago
I've had this problem too, but after doing the (quite considerable) leg-work to fix it involving messing around with process groups and the ridiculous number of different types of signal, I'm more inclined to blame the horrible 30 year old UNIX APIs than python. I've had this signal handling problem in single process/single threaded code too, and in other languages. True, it would be nice if python layered on a nicer set of APIs to deal with the nastiness.
- logicchains 11y ago
  >I'm more inclined to blame the horrible 30 year old UNIX APIs than python. Datapoint: I encountered the same thing on Windows, having to use the Powershell equivalent of `killall python` to deal with it.
- easytiger 11y ago
  No, this is pretty easy in c. i think the python docs even apologise for it somewhere.

vosper 11y ago

I was inconvenienced by multiprocessing just today: We were using it to spin up a bunch of workers, and have spent some time debugging an issue with SQLAlchemy, which requires special management when forking: "it’s usually required that a separate Engine be used for each child process. This is because the Engine maintains a reference to a connection pool that ultimately references DBAPI connections - these tend to not be portable across process boundaries." [1] So you have to be careful when dealing with forking and databases, which has lead us to subclass multiprocessing.Process specifically for our (common) case of wanting to continue to use the session object in the child process without having to think about recycling the Engine object. It was that code we have been trying to debug today, because in some specific cases we still run into issues (yes, we read the docs). Also, when people (usually new engineers) don't know about this they usually blindly use the multiprocessing module directly (and who can blame them) and end up spend some time debugging intermittent connectivity issues until someone says "oh, you should use the xyz module, it handles the DB stuff for you". [1] http://docs.sqlalchemy.org/en/rel_0_9/core/connections.html#basic-usage http://docs.sqlalchemy.org/en/rel_0_9/core/connections.html#...

crdoconnor 11y ago
>it’s usually required that a separate Engine be used for each child process. Yeah, provided you do this there should be no headache. >So you have to be careful when dealing with forking and databases, which has lead us to subclass multiprocessing.Process specifically for our (common) case of wanting to continue to use the session object in the child process without having to think about recycling the Engine object. What use case led to this being a non-negotiable requirement?
- vosper 11y ago
  It's not non-negotiable, it's for safety and convenience and DRY: We want our engineers to be able to use the database in their parallel processing jobs without having to understand the internals of SQLAlchemy, Engines, and forking.
jperras 11y ago
You probably know this already, but if you're using (or are amenable to using) uwsgi for managing your workers, there's an option[1] that can control whether your application code is loaded before or after fork() is called. I've run into the exact issue you've described; the --lazy-apps solution may be slightly more resource intensive (since you maintain N copies of the application in memory), but ends up being much nicer to reason about. 1. http://uwsgi-docs.readthedocs.org/en/latest/articles/TheArtOfGracefulReloading.html#preforking-vs-lazy-apps-vs-lazy http://uwsgi-docs.readthedocs.org/en/latest/articles/TheArtO...

Kalium 11y ago

I have. I've personally worked on a project that had to be significantly refactored because the original author didn't understand the implications of GIL.

yummyfajitas 11y ago

Other major inconveniences I've run into: Collecting metrics require using an external collector process (statsd) rather than just an in-proc lib (e.g., codahale metrics on the JVM). Issues generating random numbers - I ran 8M simulations on 8 cores, but because the random seed was forked, I actually only ran 1M simulations 8 times. (Easy enough to fix, but annoying.) Deduplication is really handy in reducing memory usage - why would you want the same (immutable) object duplicated N times? And of course various issues with resources - you need to be careful only to connect to the DB/etc after forking otherwise stuff gets weird. And of course, this means that instead of sharing a single connection pool, you've now got one DB connection per proc, which may or may not be a big waste of resources.

semi-extrinsic 11y ago
> Issues generating random numbers - I ran 8M simulations on 8 cores, but because the random seed was forked, I actually only ran 1M simulations 8 times. (Easy enough to fix, but annoying.) IIRC, even if you do your easy fix of giving the 8 threads different seeds, your results are not trustworthy since the parallell random number streams are likely to exhibit significant correlations. There's quite a bit of algorithm research behind quality PRNGs.

mzs 11y ago

I gave-up on multiprocessing a long time ago. I was happily writing code that used it on linux and then tried running it on FreeBSD. Turned-out that it worked basically because of nuances of glibc but was totally not fork safe. The most simple things broke. I don't know if they have ever been addressed.