3 ms·
> is there any use case of No-GIL which is not solved by multiprocessing ? Tons: - A web server that responds using shared state to multiple clients at the sa
by gjulianm 3y ago
> is there any use case of No-GIL which is not solved by multiprocessing ?
Tons:
- A web server that responds using shared state to multiple clients at the same time.
- multiprocessing uses pickle to send and receive data, which is a massive overhead in terms of performance. Let’s say you want to do some parallel computations on a data structure, and said structure is 1GB in memory. Multiprocessing won’t be able to deal with it with good performance.
- Another consequence of using pickle is that you can’t share all types of objects. To make matters worse, errors due to unpickable objects are really hard to debug in any non-trivial data structure. This means that sharing certain objects (specially those created by native libraries) can be impossible.
- Any processing where state needs to be shared during process execution is really hard to do with the multiprocessing module. For example, the Prometheus exporter for Flask, which only outputs some basic stats for response times/paths/etc, needs a weird hack with a temporary directory if you want to collect stats for all processes.
I could go on, honestly. But the GIL is a massive problem when trying to do parallelism in Python.