3 ms·

The fallacy here is assuming this is a necessary trade off. You can have good single threaded performance without a GIL. Rust has this notion of zero cost abstr

The fallacy here is assuming this is a necessary trade off. You can have good single threaded performance without a GIL. Rust has this notion of zero cost abstractions. It does Threads just fine. Java does single threaded stuff just fine as well. Lots of other languages do. This is not science fiction. The locking gets optimized away. Or you can choose to have code without locks at all. Or fine-grained/structured concurrency. Something being thread safe or not is basically just an API contract. And of course optimistic locking is also a thing. There are no good reasons why python wouldn't be able to do similar things. But you need to get rid of the GIL first. A lot of the performance reduction in python without the Gil is basically just unaddressed technical debt. That should be fixable over time. Adding a lot of locks is a stop gap solution. That indeed makes things slower. But the proper fix is probably rethinking how that stuff works internally in a lot of places or having API contracts that document the thread safety or lack thereof. And also having faster python runtimes and compilers actually enables re-implementing a lot of things that currently depend on native libraries in python. A lot of native code interactions are precisely where you need locks. Unless you change how that works. The point of removing the GIL is getting systematic about finding and fixing those things. It will get better over time.

logicchains 3y ago
>The fallacy here is assuming this is a necessary trade off. You can have good single threaded performance without a GIL. It depends what you're doing. Classic example: I have a huge dataframe, 10s-100s of GB. I want to process it in multiple threads, with each thread handling a different part of it. If I just use multiprocessing, it has to copy the memory to the new processes and then copy the results back, which is super slow. Sure there are hacky work-arounds and alternative approaches, but in a language without the GIL I don't need to fiddle around with those, I can just do the equivalent of Pool.map() and it doesn't need to copy anything.
- Galanwe 3y ago
  1) GIL or no GIL does not make a single visible difference in terms of your Python code, how it works, what it can or cannot do. It's only a matter of whether the interpreter internally use locks or not. 2) There is no more copy in multiprocessing that in multithreading, to the very minor exception of the reference counting structures used internally by the interpreter which will get copied on write. 3) The problem you describe is not related to multiprocessing or multithreading, it's just related to a misunderstanding of the Pool.map() API. What you provide to Pool.map() is sent over a queue (and thus pickled) to worker threads - or processes. You don't _have_ to use this queue, so long that your function has a way to access the variable you want to use. That is, the following code will have your subprocesses share, without copying, your 100GB dataframe: DF = pd.read_parquet('100GB.pq') def worker_process(id): # Do something with DF return 42 with multiprocessing.Pool(10): pool.map(work_process, [1, 2, 3, 4, 5])

logicchains 3y ago

>The fallacy here is assuming this is a necessary trade off. You can have good single threaded performance without a GIL. It depends what you're doing. Classic example: I have a huge dataframe, 10s-100s of GB. I want to process it in multiple threads, with each thread handling a different part of it. If I just use multiprocessing, it has to copy the memory to the new processes and then copy the results back, which is super slow. Sure there are hacky work-arounds and alternative approaches, but in a language without the GIL I don't need to fiddle around with those, I can just do the equivalent of Pool.map() and it doesn't need to copy anything.

Galanwe 3y ago
1) GIL or no GIL does not make a single visible difference in terms of your Python code, how it works, what it can or cannot do. It's only a matter of whether the interpreter internally use locks or not. 2) There is no more copy in multiprocessing that in multithreading, to the very minor exception of the reference counting structures used internally by the interpreter which will get copied on write. 3) The problem you describe is not related to multiprocessing or multithreading, it's just related to a misunderstanding of the Pool.map() API. What you provide to Pool.map() is sent over a queue (and thus pickled) to worker threads - or processes. You don't _have_ to use this queue, so long that your function has a way to access the variable you want to use. That is, the following code will have your subprocesses share, without copying, your 100GB dataframe: DF = pd.read_parquet('100GB.pq') def worker_process(id): # Do something with DF return 42 with multiprocessing.Pool(10): pool.map(work_process, [1, 2, 3, 4, 5])