5 ms·

In response to the multiple comments here complaining that multithreading is impossible in Python without using multiple processes, because of the GIL (global i

In response to the multiple comments here complaining that multithreading is impossible in Python without using multiple processes, because of the GIL (global interpreter lock): This is just not true, because C extension modules (i.e. libraries written to be used from Python but whose implementations are written in C) can release the global interpreter lock while inside a function call. Examples of these include numpy, scipy, pandas and tensorflow, and there are many others. Most Python processes that are doing CPU-intensive computation spend relatively little time actually executing Python, and are really just coordinating the C libraries (e.g. "mutiply these two matrices together"). The GIL is also released during IO operations like writing to a file or waiting for a subprocess to finish or send data down its pipe. So in most practical situations where you have a performance-critical application written in Python (or more precisely, the top layer is written in Python), multithreading works fine. If you are doing CPU intensive work in pure Python and you find things are unacceptably slow, then the simplest way to boost performance (and probably simplify your code) is to rewrite chunks of your code in terms of these C extension modules. If you can't do this for some reason then you will have to throw in the Python towel and re-write some or all of your code in a natively compiled language (if it's just a small fraction of your code then Cython is a good option). But this is the best course of action regardless of the threads situation, because pure Python code runs orders of magnitude slower than native code.

Rotareti 8y ago
Does anyone know how well Python and Rust team up compared to Python and C in practice?
- ikornaselur 8y ago
  I've yet to play with beyond just experimenting a little bit, but it seems it works very well. I've mainly been looking at these resources: https://github.com/rochacbruno/rust-python-example https://github.com/rochacbruno/rust-python-example https://github.com/PyO3/pyo3 https://github.com/PyO3/pyo3 Though I have not done rust <-> python in real practice
- jwandborg 8y ago
  Subjectively I'm really impressed by PyO3. If you care about speed, Rust is supposedly as fast as C. The Rust ecosystem also has a lot of supposedly safe(!) tools for parallelism.
- gpm 8y ago
  I've done it once, converting about 15 lines of python to rust. It was completely painless and resulted in a large speedup (changed a hotspot that was taking approximately 90% of execution time in a scientific simulation to approximately 0%). Type system and expressive macros seems like a big win over c to me.
  - quietbritishjim 8y ago
    Care to share a bit more detail on how you did this? Was there some interfacing library that you used analogous to Cython/SWIG/etc.? Presumably you didn't code directly against the C API (in python.h)?
    - gpm 8y ago
      The rust library interfacing with python is https://github.com/dgrunwald/rust-cpython https://github.com/dgrunwald/rust-cpython This library understand things like python arrays, objects, etc. and provides nice rust interfaces to them. Basically I just have to write a macro that specifies what functions I'm exposing to python, and other then that I'm writing normal rust. On the python side I'm importing them and calling them like any other python library. The build system is https://github.com/PyO3/setuptools-rust https://github.com/PyO3/setuptools-rust (which is linked at the bottom of the above readme).
charlescearl 8y ago
Also a nice short talk by Caleb Hattingh https://www.youtube.com/watch?v=NfnMJMkhDoQ https://www.youtube.com/watch?v=NfnMJMkhDoQ
- quietbritishjim 8y ago
  > talk [about Cython] That was interesting, thanks! I really wish he had shown his numpy code. He said at 13:46 "Numpy actually doesn't help you at all because the calculation is still getting done at the Python level". But his function could be vectorised with numpy using functions like numpy.maximum or numpy.where, in which case the main loop will be in C not Python. I can't figure out from what he said whether his numpy code did that or not. But either way, it's interesting that in this case the numpy version is arguably harder to write than the Cython version: rather than just adding a few bits of metadata (the types), you have to permute the whole control flow. If there's only a small amount of code you want to convert, I would still say it's better to use numpy though (if it actually is fast enough), because getting the build tools onto your computer for Cython can be a pain. And for some matrix computation there are speed inprovements above the fact that it's implemented in C e.g. matrix multiplication is faster than the naive O(n^3) version.
chrisseaton 8y ago
> complaining that multithreading is impossible in Python without using multiple processes, because of the GIL ... this is not true I think some people's opinions is that if you're writing in C then you're not really writing a Python program, so they think it is impossible in Python. Which seems a reasonable point to make to me. Your argument is that Python is fine for multithreading... as long as you actually write C instead of Python.
- ubernostrum 8y ago
  A whole lot depends on what exactly it is that someone wants to get out of using threading. The GIL means that a single Python interpreter process can execute at most one Python thread at a time, regardless of the number of CPUs or CPU cores available on the host machine. The GIL also introduces overhead which affects the performance of code using Python threads; how much you're affected by it will vary depending on what your code is doing. I/O-bound code tends to be much less affected, while CPU-bound code is much more affected. All of this dates back to design decisions made in the 1990s which presumably seemed reasonable for the time: most people using Python were running it on machines with one CPU which had one core, so being able to take advantage of multiple CPUs/cores to schedule multiple threads to execute simultaneously was not necessarily a high priority. And most people who wanted threading wanted it to use in things like network daemons, which are primarily I/O-bound. Hence, the GIL and the set of tradeoffs it makes. Now, of course, we carry multi-core computers in our pockets and people routinely use Python for CPU-bound data science tasks. Hindsight is great at spotting that, but hindsight doesn't give us a time machine to go back and change the decisions. Anyway. This is not the same thing as "multithreading is impossible". This is the same thing as "multithreading has some limitations, and for some cases the easiest way to work around them will be to use Python's C extension API". Which is what the parent comment seemed to be saying.
  - deleted 8y ago
    [deleted]
  - chrisseaton 8y ago
    Right I should have said parallel threads, not multithreading.
TheCondor 8y ago
The thread workers pool circumvent the GIL if you carefully follow some rules. I think the arguments and results need to be pickleable.
- quietbritishjim 8y ago
  I think you're thinking of the multiprocessing module, which uses separate processes to bypass the GIL. That's why the arguments and results need to be pickleable: pickle is a serialisation procotol, so it allows you to communicate the contents of objects between different processes. If you use threads within a single process, you don't need to pickle the objects; you just pass the object directly.
walterstucco 8y ago
that's not writing Python though
woolvalley 8y ago
Because we want first class python multithreading, like many other languages have. If we have to drop down into C, might as well use another language with first class multi-threading like java, kotlin, golang or swift and avoid all the other issues that come with slow GIL languages.

Rotareti 8y ago

Does anyone know how well Python and Rust team up compared to Python and C in practice?

ikornaselur 8y ago
I've yet to play with beyond just experimenting a little bit, but it seems it works very well. I've mainly been looking at these resources: https://github.com/rochacbruno/rust-python-example https://github.com/rochacbruno/rust-python-example https://github.com/PyO3/pyo3 https://github.com/PyO3/pyo3 Though I have not done rust <-> python in real practice
jwandborg 8y ago
Subjectively I'm really impressed by PyO3. If you care about speed, Rust is supposedly as fast as C. The Rust ecosystem also has a lot of supposedly safe(!) tools for parallelism.
gpm 8y ago
I've done it once, converting about 15 lines of python to rust. It was completely painless and resulted in a large speedup (changed a hotspot that was taking approximately 90% of execution time in a scientific simulation to approximately 0%). Type system and expressive macros seems like a big win over c to me.
- quietbritishjim 8y ago
  Care to share a bit more detail on how you did this? Was there some interfacing library that you used analogous to Cython/SWIG/etc.? Presumably you didn't code directly against the C API (in python.h)?
  - gpm 8y ago
    The rust library interfacing with python is https://github.com/dgrunwald/rust-cpython https://github.com/dgrunwald/rust-cpython This library understand things like python arrays, objects, etc. and provides nice rust interfaces to them. Basically I just have to write a macro that specifies what functions I'm exposing to python, and other then that I'm writing normal rust. On the python side I'm importing them and calling them like any other python library. The build system is https://github.com/PyO3/setuptools-rust https://github.com/PyO3/setuptools-rust (which is linked at the bottom of the above readme).

charlescearl 8y ago

Also a nice short talk by Caleb Hattingh https://www.youtube.com/watch?v=NfnMJMkhDoQ https://www.youtube.com/watch?v=NfnMJMkhDoQ

quietbritishjim 8y ago
> talk [about Cython] That was interesting, thanks! I really wish he had shown his numpy code. He said at 13:46 "Numpy actually doesn't help you at all because the calculation is still getting done at the Python level". But his function could be vectorised with numpy using functions like numpy.maximum or numpy.where, in which case the main loop will be in C not Python. I can't figure out from what he said whether his numpy code did that or not. But either way, it's interesting that in this case the numpy version is arguably harder to write than the Cython version: rather than just adding a few bits of metadata (the types), you have to permute the whole control flow. If there's only a small amount of code you want to convert, I would still say it's better to use numpy though (if it actually is fast enough), because getting the build tools onto your computer for Cython can be a pain. And for some matrix computation there are speed inprovements above the fact that it's implemented in C e.g. matrix multiplication is faster than the naive O(n^3) version.

chrisseaton 8y ago

> complaining that multithreading is impossible in Python without using multiple processes, because of the GIL ... this is not true I think some people's opinions is that if you're writing in C then you're not really writing a Python program, so they think it is impossible in Python. Which seems a reasonable point to make to me. Your argument is that Python is fine for multithreading... as long as you actually write C instead of Python.

ubernostrum 8y ago
A whole lot depends on what exactly it is that someone wants to get out of using threading. The GIL means that a single Python interpreter process can execute at most one Python thread at a time, regardless of the number of CPUs or CPU cores available on the host machine. The GIL also introduces overhead which affects the performance of code using Python threads; how much you're affected by it will vary depending on what your code is doing. I/O-bound code tends to be much less affected, while CPU-bound code is much more affected. All of this dates back to design decisions made in the 1990s which presumably seemed reasonable for the time: most people using Python were running it on machines with one CPU which had one core, so being able to take advantage of multiple CPUs/cores to schedule multiple threads to execute simultaneously was not necessarily a high priority. And most people who wanted threading wanted it to use in things like network daemons, which are primarily I/O-bound. Hence, the GIL and the set of tradeoffs it makes. Now, of course, we carry multi-core computers in our pockets and people routinely use Python for CPU-bound data science tasks. Hindsight is great at spotting that, but hindsight doesn't give us a time machine to go back and change the decisions. Anyway. This is not the same thing as "multithreading is impossible". This is the same thing as "multithreading has some limitations, and for some cases the easiest way to work around them will be to use Python's C extension API". Which is what the parent comment seemed to be saying.
- deleted 8y ago
  [deleted]
- chrisseaton 8y ago
  Right I should have said parallel threads, not multithreading.

TheCondor 8y ago

The thread workers pool circumvent the GIL if you carefully follow some rules. I think the arguments and results need to be pickleable.

quietbritishjim 8y ago
I think you're thinking of the multiprocessing module, which uses separate processes to bypass the GIL. That's why the arguments and results need to be pickleable: pickle is a serialisation procotol, so it allows you to communicate the contents of objects between different processes. If you use threads within a single process, you don't need to pickle the objects; you just pass the object directly.

walterstucco 8y ago

that's not writing Python though

woolvalley 8y ago

Because we want first class python multithreading, like many other languages have. If we have to drop down into C, might as well use another language with first class multi-threading like java, kotlin, golang or swift and avoid all the other issues that come with slow GIL languages.