4 ms·

Practical threaded programming with Python

43 points by gklein 13y ago

japaget 13y ago
The article is from 2008, so it applies to Python 2. I don't know enough about threading to say whether the same code will work in Python 3.
- briancurtin 13y ago
  It will. If you just tweak some things in the examples like urllib2 imports and print-as-a-statement usage, the threading API is the same so it's still generally relevant.
- andrewcooke 13y ago
  it probably predates the multiprocessing package (introduced in 2.6, october 2008, although available 3rd party before that) and certainly doesn't mention it. multiprocessing does use multiple cores, but is a heavyweight solution (separate, communicating processes, wrapped in a thread-like api). if you need multiple threads for cpu-related performance, it can be very useful. python 2.6+, including 3. http://toastdriven.com/blog/2008/nov/11/brief-introduction-multiprocessing/ http://toastdriven.com/blog/2008/nov/11/brief-introduction-m... http://docs.python.org/2/library/multiprocessing.html http://docs.python.org/2/library/multiprocessing.html
jmspring 13y ago
Even old, the examples given are good for those not familiar with Python threading. IBM Developer Works has been a surprisingly decent site for assorted topics/examples over the years.
herge 13y ago
Please note that any use of python threads (in cpython, at least) will still use only one core at a time. So things datetime stuff, using Beautifulsoup, etc, will not be done in parallel. To do stuff on more than one core, look at the multiprocessing module.
- rdtsc 13y ago
  That is true. I think date was more of an example on how to use threads. But yeah, Beautifulsoup bit might actually run a bit slower if it is only doing parsing. (!unless there is a C extension underneath that does release the GIL!) Any discussion of Python and threads is always confusing and it seems to me there aren't many people who understand how it works (I know you do, not talking about your comment, just in general). In one camp you have people who like to write how Python is no good and threads are just broken. Never use them. In the other camp are people who say threads are fine, they work great, I never had any issues with them. A lot of time people in this camp are just reacting to the ones in the first camp but also without understanding the underlying mechanism. I think the first thing that should be mentioned in any introductory article on Python threads is that Python threads work great for IO concurrency but they won't help with CPU concurrency. Things like downloading files, sending data over a socket will work nicely. Thing like computing determinants won't. You can still structure your code in a threaded fashion so can use multiprocessing module in the future but you won't get a speed up. So threads are not completely unusable and broken but they also have a surprising limitation. Overall in Python in my career I probably deal more with IO concurrency and threads helped there quite a bit. Others will have a different experience depending on their area of expertise. Also it is worth mentioning that libraries like Numpy and C extensions in general have the option of releasing the GIL if they want to they can get a speedup. I have done this once by hand and it did help (with a hand written extension). Didn't personally test numpy's speedup. ADDITION: It is also worth mentioning that even though you not get a speedup for CPU related concurrency, you still have to deal with synchronization issues. So you get the worse of both worlds. Just something to keep in mind.
  - mrbrowning 13y ago
    > I think the first thing that should be mentioned in any introductory article on Python threads is that Python threads work great for IO concurrency but they won't help with CPU concurrency. Things like downloading files, sending data over a socket will work nicely. Python's threads certainly work acceptably for IO-bound purposes, but given the overhead of creating a real OS thread and the potential for GIL thrashing when using Python 2.x on a multicore machine, I'm not sure why you wouldn't favor a greenlet-based solution in most such cases, especially since you don't even really have to drop the threading idiom to do so.
iffycan 13y ago
The author mentions Twisted, and I wanted to show what it would look like to use Twisted: https://news.ycombinator.com/item?id=6220124 https://news.ycombinator.com/item?id=6220124

43 points by gklein 13y ago

The article is from 2008, so it applies to Python 2. I don't know enough about threading to say whether the same code will work in Python 3.

briancurtin 13y ago
It will. If you just tweak some things in the examples like urllib2 imports and print-as-a-statement usage, the threading API is the same so it's still generally relevant.
andrewcooke 13y ago
it probably predates the multiprocessing package (introduced in 2.6, october 2008, although available 3rd party before that) and certainly doesn't mention it. multiprocessing does use multiple cores, but is a heavyweight solution (separate, communicating processes, wrapped in a thread-like api). if you need multiple threads for cpu-related performance, it can be very useful. python 2.6+, including 3. http://toastdriven.com/blog/2008/nov/11/brief-introduction-multiprocessing/ http://toastdriven.com/blog/2008/nov/11/brief-introduction-m... http://docs.python.org/2/library/multiprocessing.html http://docs.python.org/2/library/multiprocessing.html

jmspring 13y ago

Even old, the examples given are good for those not familiar with Python threading. IBM Developer Works has been a surprisingly decent site for assorted topics/examples over the years.

herge 13y ago

Please note that any use of python threads (in cpython, at least) will still use only one core at a time. So things datetime stuff, using Beautifulsoup, etc, will not be done in parallel. To do stuff on more than one core, look at the multiprocessing module.

rdtsc 13y ago
That is true. I think date was more of an example on how to use threads. But yeah, Beautifulsoup bit might actually run a bit slower if it is only doing parsing. (!unless there is a C extension underneath that does release the GIL!) Any discussion of Python and threads is always confusing and it seems to me there aren't many people who understand how it works (I know you do, not talking about your comment, just in general). In one camp you have people who like to write how Python is no good and threads are just broken. Never use them. In the other camp are people who say threads are fine, they work great, I never had any issues with them. A lot of time people in this camp are just reacting to the ones in the first camp but also without understanding the underlying mechanism. I think the first thing that should be mentioned in any introductory article on Python threads is that Python threads work great for IO concurrency but they won't help with CPU concurrency. Things like downloading files, sending data over a socket will work nicely. Thing like computing determinants won't. You can still structure your code in a threaded fashion so can use multiprocessing module in the future but you won't get a speed up. So threads are not completely unusable and broken but they also have a surprising limitation. Overall in Python in my career I probably deal more with IO concurrency and threads helped there quite a bit. Others will have a different experience depending on their area of expertise. Also it is worth mentioning that libraries like Numpy and C extensions in general have the option of releasing the GIL if they want to they can get a speedup. I have done this once by hand and it did help (with a hand written extension). Didn't personally test numpy's speedup. ADDITION: It is also worth mentioning that even though you not get a speedup for CPU related concurrency, you still have to deal with synchronization issues. So you get the worse of both worlds. Just something to keep in mind.
- mrbrowning 13y ago
  > I think the first thing that should be mentioned in any introductory article on Python threads is that Python threads work great for IO concurrency but they won't help with CPU concurrency. Things like downloading files, sending data over a socket will work nicely. Python's threads certainly work acceptably for IO-bound purposes, but given the overhead of creating a real OS thread and the potential for GIL thrashing when using Python 2.x on a multicore machine, I'm not sure why you wouldn't favor a greenlet-based solution in most such cases, especially since you don't even really have to drop the threading idiom to do so.

iffycan 13y ago

The author mentions Twisted, and I wanted to show what it would look like to use Twisted: https://news.ycombinator.com/item?id=6220124 https://news.ycombinator.com/item?id=6220124