4 ms·

This would be great if it means we can run the C portions of Python in threads without performance hits. I recently started a little project that is a cross-pla

This would be great if it means we can run the C portions of Python in threads without performance hits. I recently started a little project that is a cross-platform GUI for batch bzip2 compression, and Python did it quite well with its built-in bzip2 module. But, once I tried to do it parallel, the performance impacts of GIL were obvious. Yes, you can work around that with multi-process, but I'd rather not be spamming the running processes list and have to actually handle seperate processes that should be threads. In the end I settled for C++ and QT with the native bzip2 library with a few modifications.

chocolatebunny 9y ago
Were you running on Windows or Linux? It's my understanding that multiple processes doesn't have a big performance penalty on Linux compared to multiple threads.
- devwastaken 9y ago
  I was running under Windows, but the application is meant for Windows linux and mac.
orf 9y ago
Did you investigate the multiprocessing library? Also this kind of thing should be relatively light on the GIL if done correctly. The bzip2 module releases the GIL (I assume?), as does file IO, which is most of the workload in your use case?
apenwarr 9y ago
In normal CPython, you can design your C extension (such as bzip2) to release the GIL while it runs. This is one of the few times when threads are useful in Python. It's also why scipy etc are as fast as they are. I don't know if the bzip2 module does this, but it probably should.
- akx 9y ago
  Yeah, the stdlib bz2 module does release the GIL for the time of the compression (though it locks the compressor object simultaneously): https://github.com/python/cpython/blob/d4b93e21c2664d6a78e0656e7a7be0807be1c352/Modules/_bz2module.c#L242 https://github.com/python/cpython/blob/d4b93e21c2664d6a78e06...
- cjbillington 9y ago
  This. Any part of my numerical code that is a bottleneck, is either already coming from scipy or numpy, or I'm going to write in Cython if possible. Rewriting in Cython is already the opimisation you would do before going to multithreaded, because it can get you factors of 10, 100, etc, whereas multithreaded gets me a factor of 4 to 8 depending on how many cores I have and how independent the workload is. So by the time it comes to consider multiple threads, the bottlenecks that I want to paralellise are already non-GIL-holding. I wrote a tool to measure what proportion of the time the GIL is held in a program: https://github.com/chrisjbillington/gil_load https://github.com/chrisjbillington/gil_load I encourange people to measure what fraction of the time the GIL is actually held in their multithreaded programs. Unless it's approaching 100%, go ahead and use more threads! You will get a speedup. It's my experience that this is true more often than not. The biggest exception is poorly written C extensions that do not release the GIL even though they have no need for it. But if you're writing your own in Cython it's a matter of just typing `with nogil:`.
deleted 9y ago
[deleted]
dkersten 9y ago
If you're calling into C, you can disable the GIL from C for the duration of the call. You have to re-enable it again before the call returns and you have to be careful not to call any of the Python C API calls that rely on the GIL (reference counting and such for sure). Of course, if you want to do this, you can't simply call into a random C library directly but have to write a C stub.
monkmartinez 9y ago
> I'd rather not be spamming the running processes list and have to actually handle seperate processes that should be threads. I may be a bit naive asking this... but why would you care that much? Looking at activity monitor on my Mac, I count 14 Google Chrome Helper Process instances each spawning upwards of 13 threads. Adobe does something similar, as do several other programs/applications on my machine. Yet, my machine is mostly idle. I can only speak for myself here. If I want something done on my computer... I don't care if it spams my process list if that is what it takes to complete the task. Don't crash my machine, but do what you have to do to get it done quickly.
- devwastaken 9y ago
  This is a parallel compression application that uses all cores of a system by default. On some systems, it may use 100% HDD, others near 100% CPU. Its meant to take up as much resources as it can unless its core usage is lowered. But, with any program that has a high workload, the potential exists that the programs UI will not respond, or perhaps your desktop won't even allow you to get to the UI to stop the process. This is where task manager saves the day. Along with that, I like it to be a single process so its easily wrappable in whatever monitoring or process-throttling application you want. I will admit I'm completely assuming that multiple processes is harder than a single process to do that with. Also, when you get up to the 16 thread count, seeing that many processes pop up at the top of your process list is both annoying and doesn't let you know how much the application overall is using easily. It could also be scary to some users who have never seen that before and think its trying to run a whole bunch of programs. Yes, some of those are clearly nitpicks and not good technical reasons, but this is a problem that is fixed with a good framework anyways.
- darpa_escapee 9y ago
  There's more overhead communicating between processes than there would be if threads were just modifying shared state.
- robotresearcher 9y ago
  A process swap completely wipes the cache. Once swapped in, your process is not up to top speed for a while, until the working set has been copied into cache. You'd like to keep it that way for as long as possible. Best case scenario: one process per core.
cjhanks 9y ago
That doesn't seem quite right. C extensions can release the GIL and still continue running. So long as they are not operating on Python objects directly it is safe.

chocolatebunny 9y ago

Were you running on Windows or Linux? It's my understanding that multiple processes doesn't have a big performance penalty on Linux compared to multiple threads.

devwastaken 9y ago
I was running under Windows, but the application is meant for Windows linux and mac.

orf 9y ago

Did you investigate the multiprocessing library? Also this kind of thing should be relatively light on the GIL if done correctly. The bzip2 module releases the GIL (I assume?), as does file IO, which is most of the workload in your use case?

apenwarr 9y ago

In normal CPython, you can design your C extension (such as bzip2) to release the GIL while it runs. This is one of the few times when threads are useful in Python. It's also why scipy etc are as fast as they are. I don't know if the bzip2 module does this, but it probably should.

akx 9y ago
Yeah, the stdlib bz2 module does release the GIL for the time of the compression (though it locks the compressor object simultaneously): https://github.com/python/cpython/blob/d4b93e21c2664d6a78e0656e7a7be0807be1c352/Modules/_bz2module.c#L242 https://github.com/python/cpython/blob/d4b93e21c2664d6a78e06...
cjbillington 9y ago
This. Any part of my numerical code that is a bottleneck, is either already coming from scipy or numpy, or I'm going to write in Cython if possible. Rewriting in Cython is already the opimisation you would do before going to multithreaded, because it can get you factors of 10, 100, etc, whereas multithreaded gets me a factor of 4 to 8 depending on how many cores I have and how independent the workload is. So by the time it comes to consider multiple threads, the bottlenecks that I want to paralellise are already non-GIL-holding. I wrote a tool to measure what proportion of the time the GIL is held in a program: https://github.com/chrisjbillington/gil_load https://github.com/chrisjbillington/gil_load I encourange people to measure what fraction of the time the GIL is actually held in their multithreaded programs. Unless it's approaching 100%, go ahead and use more threads! You will get a speedup. It's my experience that this is true more often than not. The biggest exception is poorly written C extensions that do not release the GIL even though they have no need for it. But if you're writing your own in Cython it's a matter of just typing `with nogil:`.

deleted 9y ago

[deleted]

dkersten 9y ago

If you're calling into C, you can disable the GIL from C for the duration of the call. You have to re-enable it again before the call returns and you have to be careful not to call any of the Python C API calls that rely on the GIL (reference counting and such for sure). Of course, if you want to do this, you can't simply call into a random C library directly but have to write a C stub.

monkmartinez 9y ago

> I'd rather not be spamming the running processes list and have to actually handle seperate processes that should be threads. I may be a bit naive asking this... but why would you care that much? Looking at activity monitor on my Mac, I count 14 Google Chrome Helper Process instances each spawning upwards of 13 threads. Adobe does something similar, as do several other programs/applications on my machine. Yet, my machine is mostly idle. I can only speak for myself here. If I want something done on my computer... I don't care if it spams my process list if that is what it takes to complete the task. Don't crash my machine, but do what you have to do to get it done quickly.

devwastaken 9y ago
This is a parallel compression application that uses all cores of a system by default. On some systems, it may use 100% HDD, others near 100% CPU. Its meant to take up as much resources as it can unless its core usage is lowered. But, with any program that has a high workload, the potential exists that the programs UI will not respond, or perhaps your desktop won't even allow you to get to the UI to stop the process. This is where task manager saves the day. Along with that, I like it to be a single process so its easily wrappable in whatever monitoring or process-throttling application you want. I will admit I'm completely assuming that multiple processes is harder than a single process to do that with. Also, when you get up to the 16 thread count, seeing that many processes pop up at the top of your process list is both annoying and doesn't let you know how much the application overall is using easily. It could also be scary to some users who have never seen that before and think its trying to run a whole bunch of programs. Yes, some of those are clearly nitpicks and not good technical reasons, but this is a problem that is fixed with a good framework anyways.
darpa_escapee 9y ago
There's more overhead communicating between processes than there would be if threads were just modifying shared state.
robotresearcher 9y ago
A process swap completely wipes the cache. Once swapped in, your process is not up to top speed for a while, until the working set has been copied into cache. You'd like to keep it that way for as long as possible. Best case scenario: one process per core.

cjhanks 9y ago

That doesn't seem quite right. C extensions can release the GIL and still continue running. So long as they are not operating on Python objects directly it is safe.