4 ms·
> Due to the design of the Python interpreter, using multiple threads to increase performance is at best a difficult task. At worst, it will decrease (sometime
by rdtsc 13y ago
> Due to the design of the Python interpreter, using multiple threads to increase performance is at best a difficult task. At worst, it will decrease (sometimes significantly) the speed of your program.
Nope. The writer sounds misinformed and is spreading FUD.
I have successfully used Python's threads to perform concurrent database fetches, http page getters, file uploads in parallel. Yes, there was almost linear speedup.
If you listen to this story it sounds like Guido and most other talented and smart Python contributors added threads to Python just to fuck with people heads -- "thread don't work but let's add them anyway! just to mess with them!". Nope they added them because there are many cases when they work.
The answer is if you handle concurrent I/O Python's threads will give you good speedup. Threads are real OS threads and come with nasty side-effects if using shared data structures, but make no mistake you will get the speedup.
Your mileage may very and everyone is probably biased and has a different perspective, but where I am coming from in the last 10+ years I have written mostly I/O bound concurrent code. There were very few cases where I hoped to use extra CPU concurrency.
Now I did have to do that a couple of times and if you do have that issue, most likely you'd want to descend down to C anyway. Which is what I did. Once in C you can release the lock so Python can process in parallel and your C extension/driver can process in parallel. This is exactly what I did.
Now wouldn't it be nice if Python had CPU level concurrency built in. Yes it would be great. But I don't think that is the #1 issue currently. We still don't have 16 cores on most machines.
#define RANT
What worries me is library fragmentation and the new Python 3 adoption (or lack of) now coupled with the new Async IO Future/Promise/Deferred framework introduction. That will harm Python faster and worse than GIL ever did. Adopting and standardizing a Twisted like approach to Async IO will put the nail in Python's coffin. And Guido is certainly marching in that direction. This will fragment existing (already rather fragmented) libraries. Now we'll have Twisted, Tornado, gevent, eventlet, asyncore, Threads, new Promise/Future thingie (anyone know of more?) ways of doing concurrent IO and every time you pick a library (unless you use threads, gevent or eventlet + monkey patching) you will end up choosing a whole new _ecosystem_ of frameworks.
I remember for years scouring the web for a Twisted version of an already existing library,because I had made the mistake of picking Twisted as the I/O concurrency framework. Regular library module is available, oh but I need to return a Deferred from it of course, in order for me to use it.
#undef RANT
- d0mine 13y agoHaving ioloop in the stdlib could make the cooperation between different async libraries easier.
- lmm 13y agoThe idea of including an async API (not an implementation) in the standard library is that it will enable compatibility by having twisted/tornado/gevent/etc. implement this interface, and async libraries to conform to this interface. It's trying to solve the same problem you identify. (Personally I'm not terribly hopeful that it will work, but what would you suggest?)
- raverbashing 13y ago"I have successfully used Python's threads to perform concurrent database fetches, http page getters, file uploads in parallel. Yes, there was almost linear speedup" Yes, for that, the Python threading model works fine. The problem is if you're doing a lot of processing on the threads and/or passing data between them. For CPU concurrency, go with Multiprocessing, works like a charm (maybe using ZeroMQ between the processes)
- simeonf 13y ago>Nope. The writer sounds misinformed and is spreading FUD. >I have successfully used Python's threads to perform concurrent database fetches, http page getters, file uploads in parallel. Yes, there was almost linear speedup. It's not quite that simple - especially for the novice. Not every IO oriented library is thread safe (urllib2) and not every C extension remembers to give up the GIL, so no - threads do not always automatically get you increased IO throughput. Worse - other languages do set you up to expect to be able to interleave IO and CPU bound code with threads. This frequently doesn't work well in Python. And finally - yes I have written CPU bound code and I think the techniques (processes, numerical libraries, Cython, alternative runtimes, etc) are sufficient to meet my performance needs it is a little annoying to feel that I'm giving up on 3/4 of the horsepower in my 4 core box by default...