3 ms·
Absolutely agree. Almost all tasks will perform very well when using multiprocessing. It also has a nice side-effect of steering you towards explicitly coding d
by pslam 9y ago
Absolutely agree. Almost all tasks will perform very well when using multiprocessing. It also has a nice side-effect of steering you towards explicitly coding data flows without fine-grained sharing.
If you need to close that gap between the performance of multiprocessing, and multithreading, then you probably shouldn't be using Python, or any language of the same shape, in the first place.
There is one other option I'd like to see: multiprocessing style, but with multiple Python interpreter instances in the same process — one per thread. There would still be the hard delineation of data boundaries between instances, but less overhead for pushing data between them.
- metalliqaz 9y agoWhat would that get you?
- weberc2 9y ago> If you need to close that gap between the performance of multiprocessing, and multithreading, then you probably shouldn't be using Python, or any language of the same shape, in the first place. Unfortunately, these performance concerns often manifest well after the "rewrite it in a different language" date has expired. There are a lot of people in that boat, and they need better options. > There is one other option I'd like to see: multiprocessing style, but with multiple Python interpreter instances in the same process — one per thread. There would still be the hard delineation of data boundaries between instances, but less overhead for pushing data between them. If I understand correctly, the article discusses this ("subinterpreters"), but claims that there is no advantage to this approach vs multiprocessing. Presumably any overhead savings are eaten by GIL contention or some such?
- dom0 9y ago> There are a lot of people in that boat, and they need better options. Land isn't coming to you, folks, you must start rowing if you want to get there. Rewrite bit for bit. Module for module. Package for package.
- weberc2 9y agoSounds like you're saying this is infeasible; care to explain why?
- funkymike 9y agoI took it to mean that it is feasible. Instead of saying "well we used the wrong language, I guess we're screwed," you rewrite one component at a time, piece by piece, until the whole has been replaced. This is the approach I try to use myself. It's nearly impossible to replace an entire system all at once. But replacing one part at a time is doable and you can see the improvements much sooner.
- weberc2 9y agoBy "it is infeasible", I meant, "removing the GIL is infeasible"; not "rewriting is infeasible".