3 ms·

Python always could use multiprocessing to do parallel processing leveraging several CPU. However, it became especially easy with Python 3.2 (10 years ago) whi

Python always could use multiprocessing to do parallel processing leveraging several CPU. However, it became especially easy with Python 3.2 (10 years ago) which introduced the ProcessPoolExecutor (https://docs.python.org/dev/library/concurrent.futures.html#concurrent.futures.ProcessPoolExecutor https://docs.python.org/dev/library/concurrent.futures.html#...): import random import time from concurrent.futures import ProcessPoolExecutor, as_completed def hello(): seconds = random.randint(0, 5) print(f"Start blocking for {seconds}s") time.sleep(seconds) print(f"Stopped blocking after {seconds}s") return seconds if __name__ == "__main__": with ProcessPoolExecutor(max_workers=2) as exec: a = exec.submit(hello) b = exec.submit(hello) for future in as_completed((a, b)): print(future.result()) The same exact same API exist for thread BTW. I don't think a tutorial should introduce you to concurrency using manual process/thread management any more. I makes no sense to me. You may note that multiprocessing still eats more RAM than with typical Go/Rust code since max_workers=n means n+1 Python VM spawing but on modern servers you don't really feel it. That's what most WSGI setup do anyway. Now, before 2019, there was one more use case that wasn't served well: how do you share some computation between isolated processes that need to communicate to make their work ? Typical use case was people using numpy or pandas crunching numbers that depended on each others. Indeed, communicating between processing using piping is expensive given the cost of message passing serialization. However, in the previous Python release (3.8), we introduced a mechanism to share memory for almost free (https://docs.python.org/3/library/multiprocessing.shared_memory.html https://docs.python.org/3/library/multiprocessing.shared_mem...): from multiprocessing.managers import SharedMemoryManager with SharedMemoryManager() as smm: sl = smm.ShareableList(name="unique_name", range(2000)) The sl object can then contain int, float, bool, str, bytes and None and its reference can be shared among processes. Each item can be deleted and replaced. You an also get an hold on sl by using the "unique_name" if you don't have the reference at hand. There is a raw ShareableMemory object for stuff like numpy/pandas array buffers if this is your main concern.

hnews_account_1 7y ago
Thank you for this. I did not know of this change in implementation in Python 3.8. Are you a core dev btw? I have a complaint about single threads and processes and how they have absolutely no way to return values back to the main thread except through some shared memory object. Am I too ignorant to understand how big of a challenge it is to do this? from threading import Thread def func1(*args): #something def main(): new_thread, return_val = Thread(target=func1, args=(1, 2)) new_thread.start() new_thread.join() print(return_val) Instead to implement this, I keep having to use like single process pools that already have return mechanisms encoded. All I need is for a way of starting off a thread (or process) and joining it once my main thread / process is done and retrieving its return value for my use (assuming there is one).
- BiteCode_dev 7y ago
  Not a core dev. The clean way to return a value is to pass it to a Queue (https://docs.python.org/fr/3/library/queue.html https://docs.python.org/fr/3/library/queue.html), and this is what the executor does, but for your use case, it's overkill. For very simple use cases, you can inherit from Thread and force join() to return the value: class SimpleReturningThread(Thread): result = None def run(self): try: if self._target: self.result = self._target(*self._args, **self._kwargs) finally: del self._target, self._args, self._kwargs def join(self, *args, **kwargs): super().join(*args, **kwargs) return self.result Personally, I'd stick to using a pool with only one worker in it. It's not worth the trouble of doing all this work. Wrap it in a few functions if you do the same thing repeatedly and call it a day, it's unlikely going to be what most your code is about anyway.
RossBencina 7y ago
Thanks for the detailed answer. Correct me if I'm wrong, but from what you've said I take it that the GIL issue has not been resolved (i.e. the Python interpreter is not thread-safe, Python still can't have a single process, with a single Python interpreter, running multiple CPU-bound threads all doing useful compute in Python, running against a single unified local memory space containing all of the objects, as you would in say C or C++). The only option is to resort to more elaborate schemes involving multiple processes, multiple interpreters, and explicitly managed shared memory windows -- but all of these options are in place now, so there is at least a solution to most common use cases, even if it's not as simple as in other languages with proper support for threads and shared local memory.
- BiteCode_dev 7y ago
  There are been many attempt to create a different design without the GIL. None of them made it to cPython. The whole language assume it's safe to mutate things around. E.G: list.append() is assumed to be thread safe.

hnews_account_1 7y ago

Thank you for this. I did not know of this change in implementation in Python 3.8. Are you a core dev btw? I have a complaint about single threads and processes and how they have absolutely no way to return values back to the main thread except through some shared memory object. Am I too ignorant to understand how big of a challenge it is to do this? from threading import Thread def func1(*args): #something def main(): new_thread, return_val = Thread(target=func1, args=(1, 2)) new_thread.start() new_thread.join() print(return_val) Instead to implement this, I keep having to use like single process pools that already have return mechanisms encoded. All I need is for a way of starting off a thread (or process) and joining it once my main thread / process is done and retrieving its return value for my use (assuming there is one).

BiteCode_dev 7y ago
Not a core dev. The clean way to return a value is to pass it to a Queue (https://docs.python.org/fr/3/library/queue.html https://docs.python.org/fr/3/library/queue.html), and this is what the executor does, but for your use case, it's overkill. For very simple use cases, you can inherit from Thread and force join() to return the value: class SimpleReturningThread(Thread): result = None def run(self): try: if self._target: self.result = self._target(*self._args, **self._kwargs) finally: del self._target, self._args, self._kwargs def join(self, *args, **kwargs): super().join(*args, **kwargs) return self.result Personally, I'd stick to using a pool with only one worker in it. It's not worth the trouble of doing all this work. Wrap it in a few functions if you do the same thing repeatedly and call it a day, it's unlikely going to be what most your code is about anyway.

RossBencina 7y ago

Thanks for the detailed answer. Correct me if I'm wrong, but from what you've said I take it that the GIL issue has not been resolved (i.e. the Python interpreter is not thread-safe, Python still can't have a single process, with a single Python interpreter, running multiple CPU-bound threads all doing useful compute in Python, running against a single unified local memory space containing all of the objects, as you would in say C or C++). The only option is to resort to more elaborate schemes involving multiple processes, multiple interpreters, and explicitly managed shared memory windows -- but all of these options are in place now, so there is at least a solution to most common use cases, even if it's not as simple as in other languages with proper support for threads and shared local memory.

BiteCode_dev 7y ago
There are been many attempt to create a different design without the GIL. None of them made it to cPython. The whole language assume it's safe to mutate things around. E.G: list.append() is assumed to be thread safe.