4 ms·

In normal CPython, you can design your C extension (such as bzip2) to release the GIL while it runs. This is one of the few times when threads are useful in Pyt

In normal CPython, you can design your C extension (such as bzip2) to release the GIL while it runs. This is one of the few times when threads are useful in Python. It's also why scipy etc are as fast as they are. I don't know if the bzip2 module does this, but it probably should.

akx 9y ago
Yeah, the stdlib bz2 module does release the GIL for the time of the compression (though it locks the compressor object simultaneously): https://github.com/python/cpython/blob/d4b93e21c2664d6a78e0656e7a7be0807be1c352/Modules/_bz2module.c#L242 https://github.com/python/cpython/blob/d4b93e21c2664d6a78e06...
cjbillington 9y ago
This. Any part of my numerical code that is a bottleneck, is either already coming from scipy or numpy, or I'm going to write in Cython if possible. Rewriting in Cython is already the opimisation you would do before going to multithreaded, because it can get you factors of 10, 100, etc, whereas multithreaded gets me a factor of 4 to 8 depending on how many cores I have and how independent the workload is. So by the time it comes to consider multiple threads, the bottlenecks that I want to paralellise are already non-GIL-holding. I wrote a tool to measure what proportion of the time the GIL is held in a program: https://github.com/chrisjbillington/gil_load https://github.com/chrisjbillington/gil_load I encourange people to measure what fraction of the time the GIL is actually held in their multithreaded programs. Unless it's approaching 100%, go ahead and use more threads! You will get a speedup. It's my experience that this is true more often than not. The biggest exception is poorly written C extensions that do not release the GIL even though they have no need for it. But if you're writing your own in Cython it's a matter of just typing `with nogil:`.

4 ms·

In normal CPython, you can design your C extension (such as bzip2) to release the GIL while it runs. This is one of the few times when threads are useful in Pyt

by apenwarr 9y ago

akx 9y ago
Yeah, the stdlib bz2 module does release the GIL for the time of the compression (though it locks the compressor object simultaneously): https://github.com/python/cpython/blob/d4b93e21c2664d6a78e0656e7a7be0807be1c352/Modules/_bz2module.c#L242 https://github.com/python/cpython/blob/d4b93e21c2664d6a78e06...
cjbillington 9y ago
This. Any part of my numerical code that is a bottleneck, is either already coming from scipy or numpy, or I'm going to write in Cython if possible. Rewriting in Cython is already the opimisation you would do before going to multithreaded, because it can get you factors of 10, 100, etc, whereas multithreaded gets me a factor of 4 to 8 depending on how many cores I have and how independent the workload is. So by the time it comes to consider multiple threads, the bottlenecks that I want to paralellise are already non-GIL-holding. I wrote a tool to measure what proportion of the time the GIL is held in a program: https://github.com/chrisjbillington/gil_load https://github.com/chrisjbillington/gil_load I encourange people to measure what fraction of the time the GIL is actually held in their multithreaded programs. Unless it's approaching 100%, go ahead and use more threads! You will get a speedup. It's my experience that this is true more often than not. The biggest exception is poorly written C extensions that do not release the GIL even though they have no need for it. But if you're writing your own in Cython it's a matter of just typing `with nogil:`.