3 ms·
I was inconvenienced by multiprocessing just today: We were using it to spin up a bunch of workers, and have spent some time debugging an issue with SQLAlchemy,
by vosper 11y ago
I was inconvenienced by multiprocessing just today: We were using it to spin up a bunch of workers, and have spent some time debugging an issue with SQLAlchemy, which requires special management when forking: "it’s usually required that a separate Engine be used for each child process. This is because the Engine maintains a reference to a connection pool that ultimately references DBAPI connections - these tend to not be portable across process boundaries." [1]
So you have to be careful when dealing with forking and databases, which has lead us to subclass multiprocessing.Process specifically for our (common) case of wanting to continue to use the session object in the child process without having to think about recycling the Engine object. It was that code we have been trying to debug today, because in some specific cases we still run into issues (yes, we read the docs). Also, when people (usually new engineers) don't know about this they usually blindly use the multiprocessing module directly (and who can blame them) and end up spend some time debugging intermittent connectivity issues until someone says "oh, you should use the xyz module, it handles the DB stuff for you".
[1] http://docs.sqlalchemy.org/en/rel_0_9/core/connections.html#basic-usage http://docs.sqlalchemy.org/en/rel_0_9/core/connections.html#...
- crdoconnor 11y ago>it’s usually required that a separate Engine be used for each child process. Yeah, provided you do this there should be no headache. >So you have to be careful when dealing with forking and databases, which has lead us to subclass multiprocessing.Process specifically for our (common) case of wanting to continue to use the session object in the child process without having to think about recycling the Engine object. What use case led to this being a non-negotiable requirement?
- vosper 11y agoIt's not non-negotiable, it's for safety and convenience and DRY: We want our engineers to be able to use the database in their parallel processing jobs without having to understand the internals of SQLAlchemy, Engines, and forking.
- jperras 11y agoYou probably know this already, but if you're using (or are amenable to using) uwsgi for managing your workers, there's an option[1] that can control whether your application code is loaded before or after fork() is called. I've run into the exact issue you've described; the --lazy-apps solution may be slightly more resource intensive (since you maintain N copies of the application in memory), but ends up being much nicer to reason about. 1. http://uwsgi-docs.readthedocs.org/en/latest/articles/TheArtOfGracefulReloading.html#preforking-vs-lazy-apps-vs-lazy http://uwsgi-docs.readthedocs.org/en/latest/articles/TheArtO...