3 ms·

All of these problems can be addressed using inter-process shared memory. Shared memory support is built into multiprocessing [1]. Now I agree it might not be

All of these problems can be addressed using inter-process shared memory. Shared memory support is built into multiprocessing [1]. Now I agree it might not be convenient, but that's a matter of libraries. This post would be much more constructive if it was speculation on what such a library could look like, instead of pretending that concurrency "doesn't work" in single-threaded Unix processes. 1. http://docs.python.org/2/library/multiprocessing.html#module-multiprocessing.sharedctypes http://docs.python.org/2/library/multiprocessing.html#module...

yummyfajitas 13y ago
Ok - using shared ctypes, how would you build the event processing system or the cache described in the blogpost? I'm sure it's possible to build an IPC garbage collector and enable everything described. But to describe it as "not convenient" is perhaps a small understatement.
- AnIrishDuck 13y ago
  Well, it sounds like a good start - at least for the cache example - would be a shared memory hash table. Building one in pure python with the existing multiprocessing primitives is doable, though obviously not trivial. Building one in C is also possible, though obviously there's the development time / performance tradeoff there. A shared memory hash table actually seems like a really useful thing to eventually make its way into the multiprocessing module. For the event example, parsing the data structures into shared memory and then accessing from there would remove the parsing and memory overhead he complains about. I don't really want to dig in to the standards he cites and see how difficult that solution would be, but again, I can't see a reason why it isn't possible. At any rate, in both cases, the solution is definitely doable; there's nothing inherent in python's concurrency model that "prevents you from properly making use of modern multicore hardware" Side note: shared memory access in modern multicore hardware may not get you the performance gains you would initially think. Especially for write-heavy workloads, cache invalidation is a huge problem.
  - yummyfajitas 13y ago
    For the event example, garbage collection is the tricky bit, not sharing. Sharing could be handled merely by writing C structs linearly to an array - it's cleaning them up when the message is done being processed that's the problem. A shared memory hash table is a) tricky to do and b) would require repeated polling of the hash table by client processes. Or perhaps you'd need to write to the hash table all the clients who are listening to that future.
    - AnIrishDuck 13y ago
      > For the event example, garbage collection is the tricky bit, not sharing. Sharing could be handled merely by writing C structs linearly to an array - it's cleaning them up when the message is done being processed that's the problem. I agree; however, this problem could still be bootstrapped to each individual process's GC. The parser sends the shared memory id to all worker processes, and the shared memory representation contains a reference counter. Each object decrements the reference counter in __del__; when the reference counter reaches zero, the memory is unmapped and reclaimed by the kernel. This would probably cause major fragmentation issues with small objects, so pooling might be necessary. This sounds like a job that could be abstracted by a library. > A shared memory hash table is a) tricky to do and b) would require repeated polling of the hash table by client processes. Or perhaps you'd need to write to the hash table all the clients who are listening to that future. I won't argue that shared memory hash tables (and concurrent data structures in general) are easy to implement. But that leads back to my original point; that kind of code should be in a library. Admittedly, Python today lacks libraries providing concurrent data structures. And it lacks modules that handle concurrent memory management. That doesn't mean there's something inherent in its design that prevents such libraries from existing.
anonymous 13y ago
Let me first say that I agree with you. However, I think that the threaded model is in general easier to work with, rather than shared memory. That said, this might be offset by python itself being easier to work with than another language. Also, separating work into multiple processes gives in general better security - one crashing worker won't impact the others. Then again, you won't take advantage of hyper-threading features of the CPU. In the end, everybody needs to consider their personal needs and make a tradeoff.
- AnIrishDuck 13y ago
  > I think that the threaded model is in general easier to work with, rather than shared memory. The thing is, threaded models of programming in other languages already use the "shared memory" model, they just make it more implicit. Which leads to tons of problems for inexperienced developers who don't really understand concurrency, or the performance problems that shared memory can bring in modern processors. > In the end, everybody needs to consider their personal needs and make a tradeoff. Agreed.
- mwcampbell 13y ago
  Can you please explain why using processes instead of threads means you can't take advantage of hyper-threading? It may seem obvious from the name "hyper-threading", but I want to make sure that this inference is accurate.
  - AnIrishDuck 13y ago
    It shouldn't. Processors with hyper-threading enabled expose two virtual processors to the operating system per physical core.
KirinDave 13y ago
> All of these problems can be addressed using inter-process shared memory. Shared memory support is built into multiprocessing [1]. Well, sorta. You're still incurring the mighty overhead of multiple processes and all that it entails if you want to dynamically resize the set of workers. One of the reasons that Scala, Erlang and Clojure have it so good on the concurrency AND parallelism front is that from their perspectively, these things are both cheap in terms of the overhead and the syntax. And of course there's locality to consider.

yummyfajitas 13y ago

Ok - using shared ctypes, how would you build the event processing system or the cache described in the blogpost? I'm sure it's possible to build an IPC garbage collector and enable everything described. But to describe it as "not convenient" is perhaps a small understatement.

AnIrishDuck 13y ago
Well, it sounds like a good start - at least for the cache example - would be a shared memory hash table. Building one in pure python with the existing multiprocessing primitives is doable, though obviously not trivial. Building one in C is also possible, though obviously there's the development time / performance tradeoff there. A shared memory hash table actually seems like a really useful thing to eventually make its way into the multiprocessing module. For the event example, parsing the data structures into shared memory and then accessing from there would remove the parsing and memory overhead he complains about. I don't really want to dig in to the standards he cites and see how difficult that solution would be, but again, I can't see a reason why it isn't possible. At any rate, in both cases, the solution is definitely doable; there's nothing inherent in python's concurrency model that "prevents you from properly making use of modern multicore hardware" Side note: shared memory access in modern multicore hardware may not get you the performance gains you would initially think. Especially for write-heavy workloads, cache invalidation is a huge problem.
- yummyfajitas 13y ago
  For the event example, garbage collection is the tricky bit, not sharing. Sharing could be handled merely by writing C structs linearly to an array - it's cleaning them up when the message is done being processed that's the problem. A shared memory hash table is a) tricky to do and b) would require repeated polling of the hash table by client processes. Or perhaps you'd need to write to the hash table all the clients who are listening to that future.
  - AnIrishDuck 13y ago
    > For the event example, garbage collection is the tricky bit, not sharing. Sharing could be handled merely by writing C structs linearly to an array - it's cleaning them up when the message is done being processed that's the problem. I agree; however, this problem could still be bootstrapped to each individual process's GC. The parser sends the shared memory id to all worker processes, and the shared memory representation contains a reference counter. Each object decrements the reference counter in __del__; when the reference counter reaches zero, the memory is unmapped and reclaimed by the kernel. This would probably cause major fragmentation issues with small objects, so pooling might be necessary. This sounds like a job that could be abstracted by a library. > A shared memory hash table is a) tricky to do and b) would require repeated polling of the hash table by client processes. Or perhaps you'd need to write to the hash table all the clients who are listening to that future. I won't argue that shared memory hash tables (and concurrent data structures in general) are easy to implement. But that leads back to my original point; that kind of code should be in a library. Admittedly, Python today lacks libraries providing concurrent data structures. And it lacks modules that handle concurrent memory management. That doesn't mean there's something inherent in its design that prevents such libraries from existing.

anonymous 13y ago

Let me first say that I agree with you. However, I think that the threaded model is in general easier to work with, rather than shared memory. That said, this might be offset by python itself being easier to work with than another language. Also, separating work into multiple processes gives in general better security - one crashing worker won't impact the others. Then again, you won't take advantage of hyper-threading features of the CPU. In the end, everybody needs to consider their personal needs and make a tradeoff.

AnIrishDuck 13y ago
> I think that the threaded model is in general easier to work with, rather than shared memory. The thing is, threaded models of programming in other languages already use the "shared memory" model, they just make it more implicit. Which leads to tons of problems for inexperienced developers who don't really understand concurrency, or the performance problems that shared memory can bring in modern processors. > In the end, everybody needs to consider their personal needs and make a tradeoff. Agreed.
mwcampbell 13y ago
Can you please explain why using processes instead of threads means you can't take advantage of hyper-threading? It may seem obvious from the name "hyper-threading", but I want to make sure that this inference is accurate.
- AnIrishDuck 13y ago
  It shouldn't. Processors with hyper-threading enabled expose two virtual processors to the operating system per physical core.

KirinDave 13y ago

> All of these problems can be addressed using inter-process shared memory. Shared memory support is built into multiprocessing [1]. Well, sorta. You're still incurring the mighty overhead of multiple processes and all that it entails if you want to dynamically resize the set of workers. One of the reasons that Scala, Erlang and Clojure have it so good on the concurrency AND parallelism front is that from their perspectively, these things are both cheap in terms of the overhead and the syntax. And of course there's locality to consider.