3 ms·

Yes, used for decades - and for good reason and benefit. No, no the same thing. Sharing objects between processes is not easily achieved for various reasons (a

Yes, used for decades - and for good reason and benefit. No, no the same thing. Sharing objects between processes is not easily achieved for various reasons (at least in Python). It would be easier to get it in multithreading with an arena based allocation model where objects live in a shared or non-shared area of memory. Also it's not my idea. I am just advocating it as the better model for Python to advance to. It would also not take years to implement and the risks are minimized as there is full compatibility with existing code.

Galanwe 3y ago
> No, no the same thing. Well to me it is the same thing. Two threads sharing state through a common heap, or two processes sharing state through a shared memory is pretty much indistinguishable, at least on Linux. The question is not multithreading or multiprocessing anymore, the difference to me is more semantic than real. The question is then just how these threads/processes communicate. I would argue that shared mutable state is rarely a good idea, and an equivalent message based system is often preferable. For the few use cases that remain where you would want shared mutable states, as mentioned in my original answer, Python has shared memory support, though with non-built-in types. Improving these shared types should be the only thing you advocate for.
- Too 3y ago
  Right now, you often get stuck on using multiprocessing because you need to share one object that is unpickleable and can't be sent over queues. Improving shared types would be a great addition. Problem is the reason for objects being unpicklable in the first place, is often a more underlying thing that can't be shared across processes, like open file handles.
- miraculixx 3y ago
  I agree, actually. It's just that (some) people really want multithreading. And with that Python had the rare chance to say, ok you'll get it, but we will make it safe by default. Now they said ok you get it, even though it's a big risk, we hope it'll work. If not, we'll take it back. In what world is "taking back" even an option?
- thayne 3y ago
  > two processes sharing state through a shared memory The point is that isn't very practical, especially in python. In practice, any data passed between processes must be serialized and deserialized over a pipe or in shared memory. With a thread per interpreter, native python objecta could potentially be shared across threads/interpreters as long as there was adequate synchronization. Technically, you could also do that with shared memory with multiprocessing, but it would be harder, and you would need to specify the object is intended to be shared from the beginning. > pretty much indistinguishable, at least on Linux. Linux isn't the only OS people use python on. > I would argue that shared mutable state is rarely a good idea, and an equivalent message based system is often preferable. I agree, but even for message passing, having to serialize and deserialize the object across process boundaries hurts performance.
usrbinbash 3y ago
> It would be easier to get it in multithreading with an arena based allocation model where objects live in a shared or non-shared area of memory. That still relies on IPC, and the associated kernel overhead to set, ctl and access the shared memory, IN ADDITION to the kernel overhead of having to do a context switch between processes. Threads only rely on the context switch. > I am just advocating it as the better model for Python to advance to https://discuss.python.org/t/poll-feedback-to-the-sc-on-making-cpython-free-threaded-and-pep-703/28540 https://discuss.python.org/t/poll-feedback-to-the-sc-on-maki...
- miraculixx 3y ago
  > That still relies on IPC, and the associated kernel overhead to set, ctl and access the shared memory, IN ADDITION to the kernel overhead of having to do a context switch between processes. In multithreading there is no need for a context switch. Also memory is automatically shared by all threads. The arena model can be completely managed by Python. I don't get why the SC has not chosen this. Well, I do get it, but it don't like what it implies.

Galanwe 3y ago

> No, no the same thing. Well to me it is the same thing. Two threads sharing state through a common heap, or two processes sharing state through a shared memory is pretty much indistinguishable, at least on Linux. The question is not multithreading or multiprocessing anymore, the difference to me is more semantic than real. The question is then just how these threads/processes communicate. I would argue that shared mutable state is rarely a good idea, and an equivalent message based system is often preferable. For the few use cases that remain where you would want shared mutable states, as mentioned in my original answer, Python has shared memory support, though with non-built-in types. Improving these shared types should be the only thing you advocate for.

Too 3y ago
Right now, you often get stuck on using multiprocessing because you need to share one object that is unpickleable and can't be sent over queues. Improving shared types would be a great addition. Problem is the reason for objects being unpicklable in the first place, is often a more underlying thing that can't be shared across processes, like open file handles.
miraculixx 3y ago
I agree, actually. It's just that (some) people really want multithreading. And with that Python had the rare chance to say, ok you'll get it, but we will make it safe by default. Now they said ok you get it, even though it's a big risk, we hope it'll work. If not, we'll take it back. In what world is "taking back" even an option?
thayne 3y ago
> two processes sharing state through a shared memory The point is that isn't very practical, especially in python. In practice, any data passed between processes must be serialized and deserialized over a pipe or in shared memory. With a thread per interpreter, native python objecta could potentially be shared across threads/interpreters as long as there was adequate synchronization. Technically, you could also do that with shared memory with multiprocessing, but it would be harder, and you would need to specify the object is intended to be shared from the beginning. > pretty much indistinguishable, at least on Linux. Linux isn't the only OS people use python on. > I would argue that shared mutable state is rarely a good idea, and an equivalent message based system is often preferable. I agree, but even for message passing, having to serialize and deserialize the object across process boundaries hurts performance.

usrbinbash 3y ago

> It would be easier to get it in multithreading with an arena based allocation model where objects live in a shared or non-shared area of memory. That still relies on IPC, and the associated kernel overhead to set, ctl and access the shared memory, IN ADDITION to the kernel overhead of having to do a context switch between processes. Threads only rely on the context switch. > I am just advocating it as the better model for Python to advance to https://discuss.python.org/t/poll-feedback-to-the-sc-on-making-cpython-free-threaded-and-pep-703/28540 https://discuss.python.org/t/poll-feedback-to-the-sc-on-maki...

miraculixx 3y ago
> That still relies on IPC, and the associated kernel overhead to set, ctl and access the shared memory, IN ADDITION to the kernel overhead of having to do a context switch between processes. In multithreading there is no need for a context switch. Also memory is automatically shared by all threads. The arena model can be completely managed by Python. I don't get why the SC has not chosen this. Well, I do get it, but it don't like what it implies.