4 ms·
Don't assume that threads will be useful, in CPython they are simply an abstraction and don't actually run in parallel. To make use of a multi core system you n
by Nursie 14y ago
Don't assume that threads will be useful, in CPython they are simply an abstraction and don't actually run in parallel. To make use of a multi core system you need to use processes.
I'd still like it if they could get rid of the GIL...
- dbaupp 14y agoThey aren't entirely a non-parallel abstraction. C extension modules can release the GIL[1] and continue calculating. A notable example of this is Numpy/Scipy[2]. [1]: http://docs.python.org/c-api/init.html#thread-state-and-the-global-interpreter-lock http://docs.python.org/c-api/init.html#thread-state-and-the-... [2]: http://www.scipy.org/ParallelProgramming#head-9e56edb190bf1ef0863981dcaf22371e1ae8bc7b http://www.scipy.org/ParallelProgramming#head-9e56edb190bf1e...
- Nursie 14y agoTrue, there are options, I guess I was just referring to straight python threads and interpreter calls, not what you can do with extensions. For heavy math workloads, which was what I was thinking of, you can use the techniques you linked to there to great effect. It is something that you need to put a little more thought into than you might in a non-locked situation like pthreads in C though.
- andrewcooke 14y agoalso multiprocessing. but i suspect you were voted down because these days hn favours tribal groupthink (you criticised python) over rational thought.
- d0mine 14y agoGIL or no GIL, threads run concurrently in Python. The difference might be in performance: - IO-bound tasks: GIL is released - CPU-bound tasks on N-core on a single host. To exploit multiple CPUs: a) no GIL (hypothetical): N times speed up (optimistically) (it suggests weak data dependency i.e., multiprocessing can be used to the same effect) b) option a) with multiple processes (shared memory or communication-based approach) Code complexity is the same on average (except on Windows) c) C extensions (existing or new): speed up 10*N or more on numerical code Cython makes it easy to write new extensions. Currently due to dynamic nature of Python, GIL or no GIL, C extensions might be necessary to exploit hardware fully (though C extensions might not be an option in some projects) - scalable to multiple hosts tasks: different processes i.e., GIL is not a problem Benefits of GIL: - C extensions (and the interpreter) are simpler to write correctly. Multithreaded programming is not trivial we need all the help we can get. - no performance penalty for single-threaded code - it encourages a synchronization through communication concurrent model (builtin in Go) Disadvantages: - some applications have no localized performance bottlenecks. So writing a small C extension won't help to get possible performance benefits of running on multicore in parallel. If performance is critical; Python might not be the right tool in this case - there are pathological cases when performance suffers greatly due to GIL (though other approaches would have their own pathological cases) - a (non-informed) perception that Python can't benefit from multicore
- Nursie 14y agoOK, sure, they run concurrently, but they can't be in the interpreter at the same time. The result is that using python threads to do python things, you won't utilise more than one core, most of the time. There are a myriad of ways around it, but it is another thing you need to take into account when writing python programs or deciding to use python for a project.
- d0mine 14y agoyes, writing efficient programs is different in different languages. "- Don't write Java (or C++, or Javascript, ...) in Python." and then you don't need to search for workarounds.
- Dylan16807 14y agoSo if someone writes code that makes heavy use of nonlibrary python code in multiple threads, you're automatically going to blame them as writing in the style of some other language? That's pretty No True Scotsman. I'm not arguing for or against the GIL here but it's something to keep in mind. "Threads run concurrently" is misleading. And it's not a matter of being pythonic, the GIL isn't even a python feature.
- Nursie 14y agoWhat I'm saying is that it's a misleading language feature and something to bear in mind. It's not really a case of just being pythonic.