3 ms·

That's a fair point: you can take advantage of multiple CPUs with Python, but you have to eat the overhead of multi-processing vs multi-threading. Multi-proces

That's a fair point: you can take advantage of multiple CPUs with Python, but you have to eat the overhead of multi-processing vs multi-threading. Multi-processing breaks down when you have to communicate with high frequency between various logical runners. I think there are vastly more problems that can be solved with multi-processing than those that require multi-threading but they do exist. There is also another layer of simulated parallelism via coroutines...but I was just trying to point out that there was potential for confusion in this piece.

dom0 10y ago
You can of course use multithreading to get parallelism, eg. if the expensive calculations are not done in Python itself but rather some other library, be it cryptography, compression, numpy, serdes'ing stuff and so on. I/O works in parallel as well.
- ddorian43 10y ago
  It's still 1 core for multithreading (on the python part).
  - gshulegaard 10y ago
    Just to add more context for others... C extensions called from Python are still subject to the GIL of the calling interpreter[1]. In fact, this guaranteed thread safety has led to the creation of many non-thread safe Python C extensions which is now a critical blocker to efforts to removing the GIL[2], although there are also a number of other issues as well. As noted in the StackOverflow, you can release the GIL but there are some tricks to doing so safely[3]. But threads of a C extensions library are not necessarily pinned to a single core by the GIL like regular Python byte code. Although many performance oriented libraries like numpy[4] forsake multi-threading by default in many cases and rely on parallelization by the user (programmer)[5]. [1] http://stackoverflow.com/questions/651048/concurrency-are-python-extensions-written-in-c-c-affected-by-the-global-inter http://stackoverflow.com/questions/651048/concurrency-are-py... [2] https://youtu.be/P3AyI_u66Bw https://youtu.be/P3AyI_u66Bw [3] https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock https://docs.python.org/3/c-api/init.html#thread-state-and-t... [4] http://www.numpy.org/ http://www.numpy.org/ [5] http://stackoverflow.com/questions/16617973/why-isnt-numpy-mean-multithreaded http://stackoverflow.com/questions/16617973/why-isnt-numpy-m...
- jMyles 10y ago
  I wish I had a thousand upvotes to give this comment. This is the single most misunderstood concept in async python. I started and still contribute to hendrix (a "Twisted Django") and I can't believe how bright a lightbulb this is for people when they realize it.

dom0 10y ago

You can of course use multithreading to get parallelism, eg. if the expensive calculations are not done in Python itself but rather some other library, be it cryptography, compression, numpy, serdes'ing stuff and so on. I/O works in parallel as well.

ddorian43 10y ago
It's still 1 core for multithreading (on the python part).
- gshulegaard 10y ago
  Just to add more context for others... C extensions called from Python are still subject to the GIL of the calling interpreter[1]. In fact, this guaranteed thread safety has led to the creation of many non-thread safe Python C extensions which is now a critical blocker to efforts to removing the GIL[2], although there are also a number of other issues as well. As noted in the StackOverflow, you can release the GIL but there are some tricks to doing so safely[3]. But threads of a C extensions library are not necessarily pinned to a single core by the GIL like regular Python byte code. Although many performance oriented libraries like numpy[4] forsake multi-threading by default in many cases and rely on parallelization by the user (programmer)[5]. [1] http://stackoverflow.com/questions/651048/concurrency-are-python-extensions-written-in-c-c-affected-by-the-global-inter http://stackoverflow.com/questions/651048/concurrency-are-py... [2] https://youtu.be/P3AyI_u66Bw https://youtu.be/P3AyI_u66Bw [3] https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock https://docs.python.org/3/c-api/init.html#thread-state-and-t... [4] http://www.numpy.org/ http://www.numpy.org/ [5] http://stackoverflow.com/questions/16617973/why-isnt-numpy-mean-multithreaded http://stackoverflow.com/questions/16617973/why-isnt-numpy-m...
jMyles 10y ago
I wish I had a thousand upvotes to give this comment. This is the single most misunderstood concept in async python. I started and still contribute to hendrix (a "Twisted Django") and I can't believe how bright a lightbulb this is for people when they realize it.