6 ms·

This is only marginally related to the article in general, but python's implementation of concurrency and multi threading is fantastic in my experience. It took

This is only marginally related to the article in general, but python's implementation of concurrency and multi threading is fantastic in my experience. It took me literally a full 3 hours to get the basic hang of it, and I went from that to writing embarrassingly parallel code to do very large data operations in a matter of weeks. Not to sound ignorant, but I had zero idea about semaphores and locking even a month into using their implementation and my code worked perfectly. Big fan of that library since my work involves both querying REST APIs for data and doing computationally intensive operations on it. My cloud system is very low grade but with GIL, what now takes 12 minutes to complete on a good day would've taken literal hours to finish if I'd written it serially.

RossBencina 7y ago
I thought that Python's GIL (Global Interpreter Lock) precluded implementing parallel code in Python. Has something changed recently?
- hguant 7y ago
  TL;DR - implementing parallel code in Python means using multiprocessing as opposed to multithreading, but it depends on if you're CPU bound or I/O bound. The GIL prevents more than one instance of the python interpreter from running _per processor_. This effectively means you can only have one Python thread per processor running at a time, which is frustrating because that kinda defeats the point of threading. However, most python programs aren't CPU bound, but I/O bound. For those programs, using the python multithreading abstraction is fine; CPUs are fast enough to do context switches while one thread waits for data, etc. For CPU bound tasks, the solution is to write using the multiprocess module; each process spun up has its own interpreter and its own GIL. Things have changed somewhat with Python 3, where there are some more robust internal scheduling tools inside the multithreading module, but the rule of thumb still stays the same.
- BiteCode_dev 7y ago
  Python always could use multiprocessing to do parallel processing leveraging several CPU. However, it became especially easy with Python 3.2 (10 years ago) which introduced the ProcessPoolExecutor (https://docs.python.org/dev/library/concurrent.futures.html#concurrent.futures.ProcessPoolExecutor https://docs.python.org/dev/library/concurrent.futures.html#...): import random import time from concurrent.futures import ProcessPoolExecutor, as_completed def hello(): seconds = random.randint(0, 5) print(f"Start blocking for {seconds}s") time.sleep(seconds) print(f"Stopped blocking after {seconds}s") return seconds if __name__ == "__main__": with ProcessPoolExecutor(max_workers=2) as exec: a = exec.submit(hello) b = exec.submit(hello) for future in as_completed((a, b)): print(future.result()) The same exact same API exist for thread BTW. I don't think a tutorial should introduce you to concurrency using manual process/thread management any more. I makes no sense to me. You may note that multiprocessing still eats more RAM than with typical Go/Rust code since max_workers=n means n+1 Python VM spawing but on modern servers you don't really feel it. That's what most WSGI setup do anyway. Now, before 2019, there was one more use case that wasn't served well: how do you share some computation between isolated processes that need to communicate to make their work ? Typical use case was people using numpy or pandas crunching numbers that depended on each others. Indeed, communicating between processing using piping is expensive given the cost of message passing serialization. However, in the previous Python release (3.8), we introduced a mechanism to share memory for almost free (https://docs.python.org/3/library/multiprocessing.shared_memory.html https://docs.python.org/3/library/multiprocessing.shared_mem...): from multiprocessing.managers import SharedMemoryManager with SharedMemoryManager() as smm: sl = smm.ShareableList(name="unique_name", range(2000)) The sl object can then contain int, float, bool, str, bytes and None and its reference can be shared among processes. Each item can be deleted and replaced. You an also get an hold on sl by using the "unique_name" if you don't have the reference at hand. There is a raw ShareableMemory object for stuff like numpy/pandas array buffers if this is your main concern.
  - hnews_account_1 7y ago
    Thank you for this. I did not know of this change in implementation in Python 3.8. Are you a core dev btw? I have a complaint about single threads and processes and how they have absolutely no way to return values back to the main thread except through some shared memory object. Am I too ignorant to understand how big of a challenge it is to do this? from threading import Thread def func1(*args): #something def main(): new_thread, return_val = Thread(target=func1, args=(1, 2)) new_thread.start() new_thread.join() print(return_val) Instead to implement this, I keep having to use like single process pools that already have return mechanisms encoded. All I need is for a way of starting off a thread (or process) and joining it once my main thread / process is done and retrieving its return value for my use (assuming there is one).
    - BiteCode_dev 7y ago
      Not a core dev. The clean way to return a value is to pass it to a Queue (https://docs.python.org/fr/3/library/queue.html https://docs.python.org/fr/3/library/queue.html), and this is what the executor does, but for your use case, it's overkill. For very simple use cases, you can inherit from Thread and force join() to return the value: class SimpleReturningThread(Thread): result = None def run(self): try: if self._target: self.result = self._target(*self._args, **self._kwargs) finally: del self._target, self._args, self._kwargs def join(self, *args, **kwargs): super().join(*args, **kwargs) return self.result Personally, I'd stick to using a pool with only one worker in it. It's not worth the trouble of doing all this work. Wrap it in a few functions if you do the same thing repeatedly and call it a day, it's unlikely going to be what most your code is about anyway.
  - RossBencina 7y ago
    Thanks for the detailed answer. Correct me if I'm wrong, but from what you've said I take it that the GIL issue has not been resolved (i.e. the Python interpreter is not thread-safe, Python still can't have a single process, with a single Python interpreter, running multiple CPU-bound threads all doing useful compute in Python, running against a single unified local memory space containing all of the objects, as you would in say C or C++). The only option is to resort to more elaborate schemes involving multiple processes, multiple interpreters, and explicitly managed shared memory windows -- but all of these options are in place now, so there is at least a solution to most common use cases, even if it's not as simple as in other languages with proper support for threads and shared local memory.
- ndr 7y ago
  GIL is a problem only if you're CPU bound. For I/O bound problems Python can get you a long way.
  - RossBencina 7y ago
    That may be so. I was responding to the claim that "python's implementation of concurrency and multi threading is fantastic". Under such circumstances I'd expect to be able to do multi-threaded compute.

RossBencina 7y ago

I thought that Python's GIL (Global Interpreter Lock) precluded implementing parallel code in Python. Has something changed recently?

hguant 7y ago
TL;DR - implementing parallel code in Python means using multiprocessing as opposed to multithreading, but it depends on if you're CPU bound or I/O bound. The GIL prevents more than one instance of the python interpreter from running _per processor_. This effectively means you can only have one Python thread per processor running at a time, which is frustrating because that kinda defeats the point of threading. However, most python programs aren't CPU bound, but I/O bound. For those programs, using the python multithreading abstraction is fine; CPUs are fast enough to do context switches while one thread waits for data, etc. For CPU bound tasks, the solution is to write using the multiprocess module; each process spun up has its own interpreter and its own GIL. Things have changed somewhat with Python 3, where there are some more robust internal scheduling tools inside the multithreading module, but the rule of thumb still stays the same.
BiteCode_dev 7y ago
Python always could use multiprocessing to do parallel processing leveraging several CPU. However, it became especially easy with Python 3.2 (10 years ago) which introduced the ProcessPoolExecutor (https://docs.python.org/dev/library/concurrent.futures.html#concurrent.futures.ProcessPoolExecutor https://docs.python.org/dev/library/concurrent.futures.html#...): import random import time from concurrent.futures import ProcessPoolExecutor, as_completed def hello(): seconds = random.randint(0, 5) print(f"Start blocking for {seconds}s") time.sleep(seconds) print(f"Stopped blocking after {seconds}s") return seconds if __name__ == "__main__": with ProcessPoolExecutor(max_workers=2) as exec: a = exec.submit(hello) b = exec.submit(hello) for future in as_completed((a, b)): print(future.result()) The same exact same API exist for thread BTW. I don't think a tutorial should introduce you to concurrency using manual process/thread management any more. I makes no sense to me. You may note that multiprocessing still eats more RAM than with typical Go/Rust code since max_workers=n means n+1 Python VM spawing but on modern servers you don't really feel it. That's what most WSGI setup do anyway. Now, before 2019, there was one more use case that wasn't served well: how do you share some computation between isolated processes that need to communicate to make their work ? Typical use case was people using numpy or pandas crunching numbers that depended on each others. Indeed, communicating between processing using piping is expensive given the cost of message passing serialization. However, in the previous Python release (3.8), we introduced a mechanism to share memory for almost free (https://docs.python.org/3/library/multiprocessing.shared_memory.html https://docs.python.org/3/library/multiprocessing.shared_mem...): from multiprocessing.managers import SharedMemoryManager with SharedMemoryManager() as smm: sl = smm.ShareableList(name="unique_name", range(2000)) The sl object can then contain int, float, bool, str, bytes and None and its reference can be shared among processes. Each item can be deleted and replaced. You an also get an hold on sl by using the "unique_name" if you don't have the reference at hand. There is a raw ShareableMemory object for stuff like numpy/pandas array buffers if this is your main concern.
- hnews_account_1 7y ago
  Thank you for this. I did not know of this change in implementation in Python 3.8. Are you a core dev btw? I have a complaint about single threads and processes and how they have absolutely no way to return values back to the main thread except through some shared memory object. Am I too ignorant to understand how big of a challenge it is to do this? from threading import Thread def func1(*args): #something def main(): new_thread, return_val = Thread(target=func1, args=(1, 2)) new_thread.start() new_thread.join() print(return_val) Instead to implement this, I keep having to use like single process pools that already have return mechanisms encoded. All I need is for a way of starting off a thread (or process) and joining it once my main thread / process is done and retrieving its return value for my use (assuming there is one).
  - BiteCode_dev 7y ago
    Not a core dev. The clean way to return a value is to pass it to a Queue (https://docs.python.org/fr/3/library/queue.html https://docs.python.org/fr/3/library/queue.html), and this is what the executor does, but for your use case, it's overkill. For very simple use cases, you can inherit from Thread and force join() to return the value: class SimpleReturningThread(Thread): result = None def run(self): try: if self._target: self.result = self._target(*self._args, **self._kwargs) finally: del self._target, self._args, self._kwargs def join(self, *args, **kwargs): super().join(*args, **kwargs) return self.result Personally, I'd stick to using a pool with only one worker in it. It's not worth the trouble of doing all this work. Wrap it in a few functions if you do the same thing repeatedly and call it a day, it's unlikely going to be what most your code is about anyway.
- RossBencina 7y ago
  Thanks for the detailed answer. Correct me if I'm wrong, but from what you've said I take it that the GIL issue has not been resolved (i.e. the Python interpreter is not thread-safe, Python still can't have a single process, with a single Python interpreter, running multiple CPU-bound threads all doing useful compute in Python, running against a single unified local memory space containing all of the objects, as you would in say C or C++). The only option is to resort to more elaborate schemes involving multiple processes, multiple interpreters, and explicitly managed shared memory windows -- but all of these options are in place now, so there is at least a solution to most common use cases, even if it's not as simple as in other languages with proper support for threads and shared local memory.
ndr 7y ago
GIL is a problem only if you're CPU bound. For I/O bound problems Python can get you a long way.
- RossBencina 7y ago
  That may be so. I was responding to the claim that "python's implementation of concurrency and multi threading is fantastic". Under such circumstances I'd expect to be able to do multi-threaded compute.