4 ms·

I do care about the CPython GIL. I develop a program with a lot of analysis and output to custom formats. The things that can go to the underlying C extension g

I do care about the CPython GIL. I develop a program with a lot of analysis and output to custom formats. The things that can go to the underlying C extension go there, but the development cost is far higher for migrating stuff there. I'd like to be able to parallelize the different sections, but multiprocessing is a mess (only pickable data can be used, no shared state), threading doesn't give me any benefit due to the GIL, and PyPy hasn't worked the last three times I tried. Also, improvements can stack. If I get threading working, I can still move more things to the C extension to improve performance, or migrate finally to PyPy (if they manage to remove the GIL).

clipradiowallet 5y ago
> but multiprocessing is a mess Could you share a little more about your experiences with mp? I'm in the other camp - I use multiprocess in several python applications and am more or less happy with it. My real curiosity I suppose is what data you are trying to pass around that isn't a good candidate for mp. My own use cases normally resemble something like multiple worker-style processes consuming their workloads from Queue objects, and emitting their results to another Queue object. Usually my initial process is responsible for consuming and aggregating those results, but in some cases the initial process does nothing but coordinate the worker and aggregator processes.
- gjulianm 5y ago
  My main issue with multiprocessing is that the pickle module is incredibly unhelpful when something can't be picked. Given that there's quite a lot of context shared between types of analysis, and that context tends to change a lot, using pickle means I'm looking at quite some time debugging every now and then to see what's failing. Regarding the data itself in this case, there are some objects that are references to in-memory structures of a C extension. Last time I checked I didn't see simple ways to share those structures via pickle. Shared memory was an option, but it required me to implement and change quite a lot of things just to get things working, not to mention that whenever the C extension gets new features I need to invest extra time in making those compatible with pickle. It wasn't worth it, specially knowing that I would still run into problems every now and then due to pickling the context.
xscott 5y ago
> threading doesn't give me any benefit due to the GIL If your heavy lifting is done in C extensions, and those extensions release the GIL before doing that work, you can still get the benefit of parallelism within CPython.
- gjulianm 5y ago
  Part of the heavy lifting is in C extensions, but part is in Python code that's hard to migrate to C, and that's the part that takes more time now after migrating the low hanging fruit. I'd get a bit of benefit, but far from what I'd get if the GIL wasn't a thing.
  - xscott 5y ago
    Depending on the nature of the interpreted Python part of the code, you could get a 60-100 times speedup by using PyPy (no rewriting needed). Removing the GIL in CPython would only beat that if you could get 60-100 times parallelism.
    - gjulianm 5y ago
      I tried using PyPy several times, but I always run into problems due to the C extension and other libraries I use. Also, last time I checked it wasn't very up-to-date with respect to the mainstream Python version.

clipradiowallet 5y ago

> but multiprocessing is a mess Could you share a little more about your experiences with mp? I'm in the other camp - I use multiprocess in several python applications and am more or less happy with it. My real curiosity I suppose is what data you are trying to pass around that isn't a good candidate for mp. My own use cases normally resemble something like multiple worker-style processes consuming their workloads from Queue objects, and emitting their results to another Queue object. Usually my initial process is responsible for consuming and aggregating those results, but in some cases the initial process does nothing but coordinate the worker and aggregator processes.

gjulianm 5y ago
My main issue with multiprocessing is that the pickle module is incredibly unhelpful when something can't be picked. Given that there's quite a lot of context shared between types of analysis, and that context tends to change a lot, using pickle means I'm looking at quite some time debugging every now and then to see what's failing. Regarding the data itself in this case, there are some objects that are references to in-memory structures of a C extension. Last time I checked I didn't see simple ways to share those structures via pickle. Shared memory was an option, but it required me to implement and change quite a lot of things just to get things working, not to mention that whenever the C extension gets new features I need to invest extra time in making those compatible with pickle. It wasn't worth it, specially knowing that I would still run into problems every now and then due to pickling the context.

xscott 5y ago

> threading doesn't give me any benefit due to the GIL If your heavy lifting is done in C extensions, and those extensions release the GIL before doing that work, you can still get the benefit of parallelism within CPython.

gjulianm 5y ago
Part of the heavy lifting is in C extensions, but part is in Python code that's hard to migrate to C, and that's the part that takes more time now after migrating the low hanging fruit. I'd get a bit of benefit, but far from what I'd get if the GIL wasn't a thing.
- xscott 5y ago
  Depending on the nature of the interpreted Python part of the code, you could get a 60-100 times speedup by using PyPy (no rewriting needed). Removing the GIL in CPython would only beat that if you could get 60-100 times parallelism.
  - gjulianm 5y ago
    I tried using PyPy several times, but I always run into problems due to the C extension and other libraries I use. Also, last time I checked it wasn't very up-to-date with respect to the mainstream Python version.